Description

We need to implement a database upgrade (and downgrade) procedure for turning a regular Lighthouse node into a tree-states node. I had a previous version of this working but abandoned it because the on-disk format was still changing. The on-disk format is still expecting a few changes (e.g. #4475), but should be closer to stability now.

Steps to resolve

I think the upgrade should proceed as follows:

• Delete all freezer states (Add logic to prune all historic states #4481).
• Rewrite hot states into new tree-states format (complete with pubkeys, see Put validator pubkey cache back in beacon_chain #4413).
• Update OnDiskStoreConfig with hierarchy exponents (see Tree states to support per-slot state diffs  #4475) and linear_blocks=false.
• Rewrite DBColumn::BeaconBlockRoots to remove chunked keys.

Users can then recover historic states if they want them by running with --reconstruct-historic-states, which will reconstruct from genesis to the split slot.