Pull Request

Related issue

Fixes #5529

What does this PR do?

This PR adds the ability to rollback the migration to a newer version.
To use this new ability, one must cancel a processing or failed (or enqueued) DatabaseUpgrade task.

Doing so will also cancel all enqueued, processing or failed DatabaseUpgrade tasks.

The rollback takes place during the processing of the taskCancelation task itself.

The rollback executes the following algorithm:

1. For each index, attempt to rollback the index
   1. Make sure the index is closed so that there aren't any remaining reference to the index
   2. Reopen the index normally, check if its version is the target version (index was never upgraded in the first place). If it is, move to the next index.
   3. If the version is different from the target version, attempt to open the index in a special mode where its data is the data from before the last write to the index.
   4. If the version of the index in the special mode is the target version, then write something to the index, and commit, discarding the last write to the index.
   5. Otherwise, the rollback cannot be performed. Restart Meilisearch to attempt the upgrade again.
2. After all the indexes have been rollbacked, the index scheduler itself is downgraded to the target version: note that currently only the version of the index scheduler is rewritten, in case of non-backward compatible changes in the task queue one should incorporate a proper downgrade procedure
3. Lastly, the VERSION file is rewritten to the target version.

As the index rollback uses the PREV_SNAPSHOT feature from LMDB, it has good properties:

• rollback should not require free disk space
• rollback should be extremely fast even if the upgrade was long
• rollback has not a lot of "points of failure"

Implementation

• Add an Index::rollback function that attempts to rollback an index to a previous version using the described rollback algorithm
• Add an IndexMapper::rollback_index function that puts the index in safe conditions to be rollbacked, regarding the cache of indexes.
• Modify the task batching algorithm (tick function):
  1. Check taskCancelation tasks before databaseUpgrade tasks. When starting a Meilisearch that has an upgrade task and a task cancelation, this allows executing the canceling task before the upgrade task.
  2. Check that the version of the index scheduler is the expected version, refuse to batch any task that isn't a task cancelation or a database upgrade if it isn't the case. This is because a successful downgrade via a taskCancelation should disable the execution of all other tasks (should deletion tasks be allowed as well?)
• Add cancelling support to upgrade, so that calling the cancel route can abort a databaseUpgrade task that is currently processing. For now the support is rather superficial (one check between each index), it could be improved to add cancel points while an index is upgrading.
• Modify the processing of taskCancelation tasks to call the rollback routine when at least one database upgrade task is canceled.
• Move test module out of the giant index.rs