This is an involved ART PR that mainly covers four topics. All these topics depend on each other, so I decided to bundle them in on PR—general related issues: https://github.com/duckdblabs/duckdb-internal/issues/199, https://github.com/duckdblabs/duckdb-internal/issues/102.

1. Serializing small indexes directly to the WAL. Before this PR, we would rebuild the ART when replaying the WAL. This led to hard-to-fix index binder issues and specialized, non-performant index creation code.
2. Index changes are now accounted for when estimating the automatic checkpoint threshold.
3. Decoupling index catalog entries and their physical storage by removing the index field from the IndexCatalogEntry. This is a step towards pluggable indexes and also towards custom catalogs.
4. Primary keys and foreign keys now have names.

WAL Serialization

We reuse most of our existing serialization code for serializing an ART index to the WAL. However, we do not need to write the allocator data onto partial blocks, as we are directly writing the bytes from their respective buffers. Therefore, we pass a flag to the existing index serialization functions. Then, we only return all data_ptr_ts of relevant buffers and their allocation_size. Related fixed issues: https://github.com/duckdblabs/duckdb-internal/issues/345, https://github.com/duckdblabs/duckdb-internal/issues/374

NOTE: I changed the index serialization to work with the WAL and checkpoint managers. Before this PR, we used the metadata manager for serializing non-block index data, which is not accessible from the WAL. Now, we use the Binary(De)Serializer directly. This also means adding custom serialization code to the respective JSON files for new storage information structs. To guarantee backward compatibility, I kept some of the old deserialization code.

NOTE: THIS BREAKS FORWARD STORAGE COMPATIBILITY. Unifying the index serialization code to use the Binary(De)Serializer avoids two separate implementations (WAL and checkpoint manager). If we do not want this, the implementation becomes more cumbersome, and I will implement two different code paths (@Maxxen?).

Account Indexes for Automatic Checkpoints

When estimating the size of all changes in a transaction, we now consider an estimated index size. This enables indexes to trigger automatic checkpoints (#6273).

Decoupling the Index Catalog Entry from the Physical Index Storage

I removed the physical index from the index catalog entry. Also, I added names to primary and foreign keys. Physical index objects now also contain their names. These names are the connection between a catalog entry and an index object. We need this connection in various places where we directly used the linked physical object in the catalog entry previously.

Additionally, I removed the IndexType enum, which is now simply a string. This allows adding arbitrary new index types through extensions without changing a duckdb enum. Related issue: https://github.com/duckdblabs/duckdb-internal/issues/321

Questions

src/include/duckdb/storage/serialization/create_info.json: I changed a type here and removed a field. Is this allowed?

Follow-Up PRs

We now use an initial_art_size variable in the DuckIndexEntry. I plan to remove this variable again in a follow-up PR that dives more into the unique names of indexes. In this follow-up PR, I plan to add a function to traverse the catalog and execute a callback function on all existing indexes (via DuckTableEntry). Two usages for this callback function are (1) checking for unique name conflicts with primary and possibly foreign keys during index creation and (2) retrieving a matching index from the undo buffer during WAL size estimation.

At the moment, conflicting names between CREATE INDEX and primary/foreign keys are not problematic, as we always deserialize primary/unique/foreign indexes before separately created indexes, and because it is not yet possible to set custom names for these constraints.