We need to create a testing infrastructure focused on storage in order to test and validate the storage optimizations and changes we plan to implement based on the outcome of the issues where we define metrics and understand workloads (#46), create a baseline (#67), and evaluate database engines (#64).

In order to create this testing environment, the following tasks should be performed:

• Identify a workload that generates enough data on storage for meaningful testing (at least 10s of GB) on a single machine.
• Containerized (Docker) environment for testing that can be run locally and in a CI environment
• Support for multiple ComeBFT binaries compiled with all the supported databases (e.g. LevelDB, RocksDB)
• Allow testing for multiple database configuration options
• Generate testing output data that can be used to generate reports that can display improvements (or not) against a baseline or any other particular branch/commit.

17.01.2024.

The scope of the issue has been changed and here is the plan on what will be completed during Q1 2024.

• Identify a workload that generates enough data for meaningful storage.
• Record the size of the databases during the experiment

Definition of Done (DoD):

• Decided and fixed workload that will be used to benchmark any storage improvements.
• Running the workload with metrics done in Establish and implement the relevant metrics to understand storage workloads #46 on baseline CometBFT without the optimizations
• Running the workload with metrics done in Establish and implement the relevant metrics to understand storage workloads #46 on CometBFT with optimizations from Reconsider representation of keys for state and block store #1041 and Add in-process compaction support to databases #49.
• Produce a report comparing the results obtained by all the experiments. docs: report on storage changes and decisions.  #2596

Deprioritized - left for later (open issue to continue tracking if needed):

• Any end user (e.g. developer) can run this testing infrastructure locally without too much local configuration and can generate testing data for validating the database optimizations during the development process
• The storage testing e2e runs in a CI infrastructure (Github actions) (optional, it is not yet well defined and not clear what it would contain).
• Generated reports that can be shared with the community that outline the improvements accomplished as a result of optimizations and new implementations