Supersedes tendermint/tendermint#6032, breaking that issue down into more concrete, clearly separated deliverables/sub-tasks.

It's ultimately expensive and painful for the team to have to cater to many different use cases that require different underlying storage engines, and we would rather converge on a single database that meets our core requirements well. We do, however, want to simultaneously provide ways for integrators to access and transform core data into whatever storage system suits their use case.

Problems

1. Through cometbft-db, CometBFT currently supports multiple database backends. As such:
   1. CometBFT does not make extensive use of database-specific optimizations.
   2. Storage behavior is not consistent across different databases, potentially resulting in more troubleshooting and bug fixing work for the team (e.g. genesis file may go over the recommended maximum value size tendermint/tendermint#8416, Remove genesis persistence in state db #1017 ).
2. While we could just decide to only support GoLevelDB (as per use only goleveldb tendermint/tendermint#9741), one of the most commonly used underlying databases for CometBFT, it seems to struggle with pruning (see Add in-process compaction support to LevelDB tendermint/tendermint#9743, Investigate why Tendermint disk storage keeps growing over time informalsystems/interchain#1). It's also not clear, given our typical storage workloads for the most common use cases, whether the underlying data structure that LevelDB provides is even suitable for CometBFT. Current problems experienced by operators seem to suggest otherwise.
3. We currently do not have a very clear set of requirements for an underlying database for CometBFT.

RFC-001 provides some more detail around the problem space.

Work Breakdown

In order to achieve our overall goal of storage simplification, we need to complete the following work.

• Define external users and use cases for the CometBFT storage layer #68
• Establish the baseline and future requirements for the storage backend #63
• Evaluate database engines according to requirements and decide which one to optimize #64
• Refactor CometBFT to use a single underlying database #1039
• Add support for users to migrate to the chosen database

Original issue: tendermint/tendermint#9749