Was: tendermint/tendermint#5752

Proposal

Decide if we want to forward-port tendermint/tendermint#6549 to main, and do it if we decide so.

Original description (tendermint/tendermint#5752)

Summary

If the metric consensus_block_size_bytes is read with Prometheus in pull mode, data points may get skipped. This is because it is a gauge value that gets overwritten each time a new block is being processed. Therefore consensus_block_size_bytes is currently a useless/unreliable metric when used in pull mode.

Problem Definition

In order to monitor a network, it would be beneficial to have various metrics (such as block sizes) that could be monitored suspicious activity.

Proposal

Add a metric related to consensus_block_size_bytes that does not skip data points. It could either provide an aggregated value or provide historic values for recent blocks. Other metrics useful for monitoring for suspicious activity could also be added in a similar manner.