-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Add STORAGE_VERSION
option that allows you to specify the target storage version when serializing a database
#15794
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add STORAGE_VERSION
option that allows you to specify the target storage version when serializing a database
#15794
Conversation
…age version when serializing a database
Thanks! It was indeed ready to go in. |
Working on this related to #15637 I am extremely confused by some of the decisions made here. I think of it kind of like the "patch" part of the semantic versioning, Can we discontinue the compatibility versioning entirely and just bump the next serialized storage version (the 'serialization' map in Relevant PR for context on the |
…torage_version` (#16533) This PR relates to #15794 With that PR, we introduced an upgrade to the previously established `serialize_compatibility` flag. The `storage_version` now lives in the db, and dictates which compression algorithms are used. This storage_version defaults to 0.10.2 for the sake of backwards compatibility. The ZSTD and Roaring compression algorithms were added as part of 1.2.0, which is not covered by this default. Because of this, the benchmarks for these compression algorithms were silently using `Uncompressed`, as the forced compression algorithm is not available. This PR adds support for `assert` blocks in the benchmark runner, which are similar to `result_query`, any number of `assert` blocks can be added to verify the state of the db before running the benchmark. `storage persistent` can currently be used to make the db used in the benchmark a persistent one that does not live in-memory, so that it can be checkpointed. This PR introduces a second optional option to `storage`: `storage persistent <version>` which is similar to the SQL version: `attach 'db' (STORAGE VERSION <version>)`
Follow-up from #15702
Supersedes/builds on top of #14981
This PR change the
storage_compatibility_version
from being a setting set on every session to be written in the database file.Previously we would set this setting at run-time, and it would be shared across all database instances:
This has a number of issues:
storage_compatibility_version
would revert back towards the default setting (currentlyv0.10.0
)STORAGE_VERSION parameter
This PR reworks this so that the storage version is instead specified on
ATTACH
. When none is specified:storage_compatibility_version
is used when creating a new databaseAs a result, we can target the storage version towards the desired supported version when creating a new database. When opening an existing database, we will keep on writing targeting the same DuckDB version (i.e. we never automatically "upgrade" the file to a newer DuckDB version). The user can manually upgrade a file by opening an older file while targeting a later storage version.
For example:
Note that we cannot downgrade a file. If we try to open a file that targets e.g. version v1.2.0 with an explicit storage version of v1.0.0, we get an error:
Opening with DuckDB < v1.1.3
When opening a file that targets
v1.2.0
in an older DuckDB version, we now get a storage incompatibility error:The description in the error is not entirely correct - but the error is a lot more descriptive than the previous error that would be thrown in this scenario (which was
INTERNAL Error: Unsupported compression function type
).The error message has also been improved in #15702 already.