Make pgroll pull and pgroll migrate command able to handle incompatible migration file formats
#812
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add `RawMigration` alongside `Migration`. `RawMigration` contains the raw operations as a `json.RawMessage` instead of a fully parsed `Operations` type. This allows us to read the migration file without parsing the operations, which is useful for migrations that are not compatible with the current version of pgroll.
Make the `State.SchemaHistory` method return raw migrations instead of migrations. This allows us to retrieve old migration histories that contain migrations that are not compatible with the current version of pgroll.
Change the return type of `MissingMigrations` to return raw migrations instead of `migrations.Migrations`. This means that migrations in the schema history that use older formats that can no longer be serialized will not cause `MissingMigrations` to fail.
Make `UnappliedMigrations` read migrations from the local filesystem as raw migrations instead of deserializing them. This allows unapplied migrations in directories containing migrations in older file formats to be found, without causing an error. Change the return type to `RawMigration` to avoid having to parse the migrations before returning them. It is the caller's responsibility to parse the migrations, if required.
Make `pgroll pull` write raw migrations rather than `migrations.Migration`s. This means the command can write migrations in older file formats that can't be deserialized into `migration.Migration` objects.
* Ensure that `UnappliedMigrations` can handle local migration files that are not deserializable. * Ensure that `UnappliedMigrations` can handle remote migrations that are not deserializable.
* Ensure that `MissingMigrations` can handle local migration files that are not deserializable. * Ensure that `MissingMigrations` can handle remote migrations that are not deserializable.
kvch
approved these changes
May 6, 2025
andrew-farries
added a commit
that referenced
this pull request
May 21, 2025
Remove support for specifying a `name` field in migration files:
```yaml
name: 01_some_migration # <-- no longer supported
operations:
- create_table:
name: products
columns:
- name: id
type: serial
pk: true
- name: name
type: varchar(255)
```
The migration filename is the source of truth for the migration name.
The `name` field was made optional and its use discouraged in the
documentation in #744. This PR removes support for it entirely.
⚠️ **Warning**
**This is a breaking change:** migration files that set the `name` field
will not be runnable by `pgroll` until the `name` field is removed.
Because of #812 `pgroll pull` and
`pgroll migrate` will still work with migration histories that contain
migrations with `name` fields.
The easiest way to remove the `name` fields from all migrations in the
history is to `pgroll pull` the migration history; this will serialize
all migrations in the remote history, removing the any `name` fields.
---
Closes #715 and #850
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ensure that the
pgroll pullandpgroll migratecommands are able to operate safely when either:pgroll.pgroll.This ensures that these commands work in the presence of older migration formats, either locally or in the remote schema history.
This is done by adding a new
RawMigrationtype to themigrationspackage, which does not deserialize theOperationsin the migration but leaves them as ajson.RawMessage.The following methods in the
rollandstatepackages are then updated to work with and returnRawMigrations:SchemaHistory- used to return the schema history from the target database.MissingMigrations- used to find remote migrations that don't exist in the local historyUnappliedMigrations- used to find local migrations that don't exist in the remote.By updating these methods and the
pullandmigratecommands that use them, these commands can work safely even with local or remote migrations that can't be deserialized by the current version ofpgroll.Tests are added for both the
MissingMigrationsandUnappliedMigrationsmethods to ensure that they work with un-deserializable migrations.Part of #770