Was tendermint/tendermint#9773

We need to identify the set of metrics to understand the storage workloads of CometBFT. The metrics should help us identify:

• The access patterns (sequential / random access)
• How often the data is read/written
• Who reads/writes the data
• Is the data accessed by multiple components or just one.
• How much of the total height time is spent in storage - on a small network, on a big network? (Is storage a bottleneck? )

Open questions: Do we want information on which CometBFT Blockstore / StateStore method call triggered the access or is read/write/delete count and timing enough?

Draft implementation: tendermint/tendermint#9774

DoD

• Set of candidate metrics is established, discussed and well understood.
• The identified metrics are added into the Tendermint codebase.
• Created a custom grafana dashboard for easy monitoring. fix(localnet): Grafana dashboards for storage tests - removed experimental metrics #2448 Grafana dashboards for local use #2107 fix(monitoring): Do not convert ms to ms #2615 monitoring: Add block time to storage metrics #2616