Skip to content

Conversation

arvi3411301
Copy link
Member

@arvi3411301 arvi3411301 commented Oct 9, 2019

Description

This PR introduces three new features:

  • Support for a new migrations folder structure.
  • Add squash command in preview.
  • List of migrations on the Console and ability to squash them from console.

New migrations folder structure

Starting with this commit, Hasura CLI supports a new directory structure for migrations folder and defaults to that for all new migrations created.

Each migration will get a new directory with the name format timestamp_name and inside the directory, there will be four files:

└── migrations
    ├── 1572237730898_squashed
    │   ├── up.sql
    │   ├── up.yaml
    │   ├── down.yaml
    │   └── down.sql

Existing files old migration format timestamp_name.up|down.yaml|sql will continue to work alongside new migration files.

Squash command

Lots of users have expressed their interest in squashing migrations (see #2724 and #2254) and some even built their own tools to do squash. In this PR, we take a systematic approach to squash migrations.

A new command called migrate squash is introduced. Note that this command is in PREVIEW and the correctness of squashed migration is not guaranteed (especially for down migrations). From our tests, it works for most use cases, but we have found some issues with squashing all the down migrations, partly because the console doesn't generate down migrations for all actions.

Hence, until we add an extensive test suite for squashing, we'll keep the command in preview. We recommend you to confirm the correctness yourself by diffing the SQL and Metadata before and after applying the squashed migrations (we're also thinking about embedding some checks into the command itself).

$ hasura migrate squash --help
(PREVIEW) Squash multiple migrations leading upto the latest one into a single migration file

Usage:
  hasura migrate squash [flags]

Examples:
  # NOTE: This command is in PREVIEW, correctness is not guaranteed and the usage may change.

  # squash all migrations from version 1572238297262 to the latest one:
  hasura migrate squash --from 1572238297262

Flags:
      --from uint             start squashing form this version
      --name string           name for the new squashed migration (default "squashed")
      --delete-source         delete the source files after squashing without any confirmation

Affected components

  • CLI

Related Issues

Close #2724, Close #2254,

Solution and Design

For the squash command, a state machine is implemented to track changes to Hasura metadata. After applying each action on the metadata state, a list of incremental changes is created.

Steps to test and verify

  1. Open console via cli and create some migrations.
  2. Run hasura migrate squash --from <version>

Limitations, known bugs & workarounds

  • The squash command is in preview
  • Support for squashing from the console is WIP
  • Support for squashing migrations that are not committed yet is planned.
  • Un-tracking or dropping a table will cause inconsistent squashed down migration since console doesn't generate correct down migration.
  • If cascade setting is set to true on any of the metadata action, generated migration may be wrong

@netlify
Copy link

netlify bot commented Oct 9, 2019

Deploy preview for hasura-docs ready!

Built with commit 6ef7209

https://deploy-preview-3072--hasura-docs.netlify.com

@hasura-bot
Copy link
Contributor

Review app for commit ea1b4cc deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-ea1b4cc9

@hasura-bot
Copy link
Contributor

Review app for commit bf1afff deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-bf1afffc

@hasura-bot
Copy link
Contributor

Review app for commit f9d623d deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-f9d623d4

@arvi3411301 arvi3411301 added the c/cli Related to CLI label Oct 23, 2019
@hasura-bot
Copy link
Contributor

Review app for commit 1ea1d07 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-1ea1d072

@hasura-bot
Copy link
Contributor

Review app for commit 826ee57 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-826ee574

@hasura-bot
Copy link
Contributor

Review app for commit b8a4315 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-b8a4315a

@hasura-bot
Copy link
Contributor

Review app for commit 3ce5c26 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-3ce5c268

@hasura-bot
Copy link
Contributor

Review app for commit 1dda07e deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-1dda07ea

@shahidhk shahidhk added the c/console Related to console label Oct 28, 2019
@shahidhk shahidhk changed the title cli(migrations): new folder structure and squash command cli(migrations): new folder structure and squash Oct 28, 2019
@arvi3411301 arvi3411301 marked this pull request as ready for review October 30, 2019 10:46
@hasura-bot
Copy link
Contributor

Review app for commit 620b7eb deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-620b7eb9

@hasura-bot
Copy link
Contributor

Review app for commit 9515941 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-9515941c

shahidhk pushed a commit that referenced this pull request Oct 30, 2019
### Description
Adds a `metadata diff` command to show comparisons between two different sets of Hasura metadata.
```
# Show changes between server metadata and the exported metadata file:
hasura metadata diff

# Show changes between server metadata and that in local_metadata.yaml:
hasura metadata diff local_metadata.yaml

# Show changes between metadata from metadata.yaml and metadata_old.yaml:
hasura metadata diff metadata.yaml metadata_old.yaml
```

Also adds a `--dry-run` flag to `metadata apply` command which will print the diff and exit rather than actually applying the metadata.

### Affected components 
- CLI
- Docs

### Related Issues
Close #3126, Close #3127

### Solution and Design
- Added `metadata_diff.go` and `metadata_diff_test.go`


### Steps to test and verify
```
hasura metadata export
# Make changes to migrations/metadata.yaml
hasura metadata diff
```

### Limitations, known bugs & workarounds
This is just a general-purpose diff. 

A more contextual diff with the understanding of metadata can be added once #3072  is merged.
@shahidhk shahidhk removed the c/console Related to console label Oct 31, 2019
@hasura-bot
Copy link
Contributor

Review app for commit 6ef7209 deployed to Heroku: https://hge-ci-pull-3072.herokuapp.com
Docker image for server: hasura/graphql-engine:pull3072-6ef72094

Copy link
Member

@shahidhk shahidhk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shahidhk shahidhk merged commit 980c65d into hasura:master Oct 31, 2019
@hasura-bot
Copy link
Contributor

Review app https://hge-ci-pull-3072.herokuapp.com is deleted

@shahidhk shahidhk deleted the v2-migrate branch October 31, 2019 02:21
polRk pushed a commit to polRk/graphql-engine that referenced this pull request Feb 12, 2020
### Description
Adds a `metadata diff` command to show comparisons between two different sets of Hasura metadata.
```
# Show changes between server metadata and the exported metadata file:
hasura metadata diff

# Show changes between server metadata and that in local_metadata.yaml:
hasura metadata diff local_metadata.yaml

# Show changes between metadata from metadata.yaml and metadata_old.yaml:
hasura metadata diff metadata.yaml metadata_old.yaml
```

Also adds a `--dry-run` flag to `metadata apply` command which will print the diff and exit rather than actually applying the metadata.

### Affected components 
- CLI
- Docs

### Related Issues
Close hasura#3126, Close hasura#3127

### Solution and Design
- Added `metadata_diff.go` and `metadata_diff_test.go`


### Steps to test and verify
```
hasura metadata export
# Make changes to migrations/metadata.yaml
hasura metadata diff
```

### Limitations, known bugs & workarounds
This is just a general-purpose diff. 

A more contextual diff with the understanding of metadata can be added once hasura#3072  is merged.
polRk pushed a commit to polRk/graphql-engine that referenced this pull request Feb 12, 2020
### Description
This PR introduces three new features:

- Support for a new migrations folder structure.
- Add `squash` command in preview.
- ~List of migrations on the Console and ability to squash them from console.~

#### New migrations folder structure

Starting with this commit, Hasura CLI supports a new directory structure for migrations folder and defaults to that for all new migrations created. 

Each migration will get a new directory with the name format `timestamp_name` and inside the directory, there will be four files:

```bash
└── migrations
    ├── 1572237730898_squashed
    │   ├── up.sql
    │   ├── up.yaml
    │   ├── down.yaml
    │   └── down.sql
```

Existing files old migration format `timestamp_name.up|down.yaml|sql` will continue to work alongside new migration files.

#### Squash command

Lots of users have expressed their interest in squashing migrations (see hasura#2724 and hasura#2254) and some even built [their own tools](https://github.com/domasx2/hasura-squasher) to do squash. In this PR, we take a systematic approach to squash migrations.

A new command called `migrate squash` is introduced. Note that this command is in **PREVIEW** and the correctness of squashed migration is not guaranteed (especially for down migrations). From our tests, **it works for most use cases**, but we have found some issues with squashing all the down migrations, partly because the console doesn't generate down migrations for all actions.

Hence, until we add an extensive test suite for squashing, we'll keep the command in preview. We recommend you to confirm the correctness yourself by diffing the SQL and Metadata before and after applying the squashed migrations (we're also thinking about embedding some checks into the command itself).

```bash
$ hasura migrate squash --help
(PREVIEW) Squash multiple migrations leading upto the latest one into a single migration file

Usage:
  hasura migrate squash [flags]

Examples:
  # NOTE: This command is in PREVIEW, correctness is not guaranteed and the usage may change.

  # squash all migrations from version 1572238297262 to the latest one:
  hasura migrate squash --from 1572238297262

Flags:
      --from uint             start squashing form this version
      --name string           name for the new squashed migration (default "squashed")
      --delete-source         delete the source files after squashing without any confirmation
```

### Affected components 
<!-- Remove non-affected components from the list -->

- CLI

### Related Issues
<!-- Please make sure you have an issue associated with this Pull Request -->
<!-- And then add `(close #<issue-no>)` to the pull request title -->
<!-- Add the issue number below (e.g. hasura#234) -->
Close hasura#2724, Close hasura#2254, 

### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->

For the squash command, a state machine is implemented to track changes to Hasura metadata. After applying each action on the metadata state, a list of incremental changes is created.

### Steps to test and verify
1. Open console via cli and create some migrations.
2. Run `hasura migrate squash --from <version>`

### Limitations, known bugs & workarounds
<!-- Limitations of the PR, known bugs and suggested workarounds -->

<!-- Feel free to delete these comment lines -->
- The `squash` command is in preview
- Support for squashing from the console is WIP
- Support for squashing migrations that are not committed yet is planned.
- Un-tracking or dropping a table will cause inconsistent squashed down migration since console doesn't generate correct down migration.
- If cascade setting is set to `true` on any of the metadata action, generated migration may be wrong
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/cli Related to CLI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Let's make resetting(squash) migrations EASY! Squash migrations
3 participants