Supersedes #6032, breaking that issue down into more concrete, clearly separated deliverables/sub-tasks.

It's ultimately expensive and painful for the team to have to cater to many different use cases that require different underlying storage engines, and we would rather converge on a single database that meets our core requirements well. We do, however, want to simultaneously provide ways for integrators to access and transform core data into whatever storage system suits their use case.

Problems

1. Through tm-db, Tendermint currently supports multiple database backends. As such:
   1. Tendermint does not make extensive use of database-specific optimizations.
   2. Storage behavior is not consistent across different databases, potentially resulting in more troubleshooting and bug fixing work for the team (e.g. genesis file may go over the recommended maximum value size #8416).
2. While we could just decide to only support GoLevelDB (as per use only goleveldb #9741), one of the most commonly used underlying databases for Tendermint, it seems to struggle with pruning (see Add in-process compaction support to LevelDB #9743). It's also not clear, given our typical storage workloads for the most common use cases, whether the underlying data structure that LevelDB provides is even suitable for Tendermint. Current problems experienced by operators seem to suggest otherwise.
3. It's also not clear whether it makes sense for Tendermint to support event indexing - another source of storage space consumption, and something that is actually more of an application-level concern. ADR-082: Data Companion Push API #9437 is work towards designing one possible solution that will allow us to offload that responsibility to an external process in a consistent way across Tendermint-based networks.
4. We currently do not have a very clear set of requirements for an underlying database for Tendermint.
5. Tendermint currently maintains its own WAL - is this even necessary, given that the underlying database should actually be taking care of this? It is another source of complexity and potential point of failure in the system that the team has to maintain.

RFC-001 provides some more detail around the problem space.

Work Breakdown

In order to achieve our overall goal of storage simplification, we need to complete the following work.

• Establish requirements for a single underlying storage engine #9750
• Evaluate available database engines according to our requirements #9944
• Provide a mechanism to offload event indexing
• Refactor Tendermint to use a single underlying database