coinstatsindex way more fragile than any other data w.r.t. power loss #24076
Description
Two times since a fresh 22.0 installation I've had a power loss event (artifically produced to test system behavior), and each time I've started getting incorrect muhash values for block heights since approximately that moment
"Approximately", as in I've not taken care to record the exact time of events because I've expecting nothing of interest from bitcoind in particular; "incorrect", as in compared to my another long-running nodes running with coinstatsindex without power outages, and to its own muhash for the current tip calculated without using the index, gettxoutsetinfo muhash null false.
Expected behavior
As usual when testing power outage behavior, I've expected pretty much anything except silently returning incorrect data. Most desirable would be checking several recent block muhashes on restart and auto-correcting the problem.
To reproduce
Unsure if it can be reproduced reliably, but 2/2 times are rather good indicator that it may not be uncommon
System information
Ubuntu Linux 20.04, ~/.bitcoin on /dev/md raid1 over two HDDs, ext4 fs with default mount options
Running bitcoind 22.0 prebuilt binaries from bitcoincore (amd64).
Here's debug.log from a typical restart (I see nothing interesting there, just in case if some configured or autodetected parameters are relevant):
2022-01-14T17:10:34Z Bitcoin Core version v22.0.0 (release build)
2022-01-14T17:10:34Z Assuming ancestors of block 00000000000000000008a89e854d57e5667df88f1cdef6fde2fbca1de5b639ad have valid signatures.
2022-01-14T17:10:34Z Setting nMinimumChainWork=00000000000000000000000000000000000000001fa4663bbbe19f82de910280
2022-01-14T17:10:34Z Using the 'shani(1way,2way)' SHA256 implementation
2022-01-14T17:10:34Z Using RdSeed as additional entropy source
2022-01-14T17:10:34Z Using RdRand as an additional entropy source
2022-01-14T17:10:34Z Default data directory /home/bitnode/.bitcoin
2022-01-14T17:10:34Z Using data directory /srv/slow/dot-bitcoin
2022-01-14T17:10:34Z Config file: /etc/bitcoin/bitcoin.conf
2022-01-14T17:10:34Z Config file arg: blockfilterindex="1"
2022-01-14T17:10:34Z Config file arg: coinstatsindex="1"
2022-01-14T17:10:34Z Config file arg: dbcache="2000"
2022-01-14T17:10:34Z Config file arg: listen="1"
2022-01-14T17:10:34Z Config file arg: maxuploadtarget="450"
2022-01-14T17:10:34Z Config file arg: peerblockfilters="1"
2022-01-14T17:10:34Z Config file arg: server="1"
2022-01-14T17:10:34Z Config file arg: txindex="1"
2022-01-14T17:10:34Z Config file arg: upnp="0"
2022-01-14T17:10:34Z Config file arg: whitelist="download@127.0.0.1"
2022-01-14T17:10:34Z Config file arg: whitelist="download@192.168.1.0/24"
2022-01-14T17:10:34Z Config file arg: zmqpubrawblock="tcp://127.0.0.1:28332"
2022-01-14T17:10:34Z Config file arg: zmqpubrawtx="tcp://127.0.0.1:28333"
2022-01-14T17:10:34Z Command-line arg: conf="/etc/bitcoin/bitcoin.conf"
2022-01-14T17:10:34Z Command-line arg: daemonwait=""
2022-01-14T17:10:34Z Command-line arg: datadir="/srv/slow/dot-bitcoin/"
2022-01-14T17:10:34Z Command-line arg: pid="/run/bitcoind/bitcoind.pid"
2022-01-14T17:10:34Z Using at most 125 automatic connections (1024 file descriptors available)
2022-01-14T17:10:34Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2022-01-14T17:10:34Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2022-01-14T17:10:34Z Script verification uses 15 additional threads
2022-01-14T17:10:34Z scheduler thread start
2022-01-14T17:10:34Z HTTP: creating work queue of depth 16
2022-01-14T17:10:34Z Using random cookie authentication.
2022-01-14T17:10:34Z Generated RPC authentication cookie /srv/slow/dot-bitcoin/.cookie
2022-01-14T17:10:34Z HTTP: starting 4 worker threads
2022-01-14T17:10:34Z Using wallet directory /srv/slow/dot-bitcoin/wallets
2022-01-14T17:10:34Z init message: Verifying wallet(s)…
2022-01-14T17:10:34Z init message: Loading banlist…
2022-01-14T17:10:34Z SetNetworkActive: true
2022-01-14T17:10:34Z Using /16 prefix for IP bucketing
2022-01-14T17:10:34Z Cache configuration:
2022-01-14T17:10:34Z * Using 2.0 MiB for block index database
2022-01-14T17:10:34Z * Using 249.8 MiB for transaction index database
2022-01-14T17:10:34Z * Using 218.5 MiB for basic block filter index database
2022-01-14T17:10:34Z * Using 8.0 MiB for chain state database
2022-01-14T17:10:34Z * Using 1521.7 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2022-01-14T17:10:34Z init message: Loading block index…
2022-01-14T17:10:34Z Switching active chainstate to Chainstate [ibd] @ height -1 (null)
2022-01-14T17:10:34Z Opening LevelDB in /srv/slow/dot-bitcoin/blocks/index
2022-01-14T17:10:34Z Opened LevelDB successfully
2022-01-14T17:10:34Z Using obfuscation key for /srv/slow/dot-bitcoin/blocks/index: 0000000000000000
2022-01-14T17:10:38Z LoadBlockIndexDB: last block file = 2883
2022-01-14T17:10:38Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=17, size=20113255, heights=718645...718661, time=2022-01-14...2022-01-14)
2022-01-14T17:10:38Z Checking all blk files are present...
2022-01-14T17:10:38Z Opening LevelDB in /srv/slow/dot-bitcoin/chainstate
2022-01-14T17:10:39Z Opened LevelDB successfully
2022-01-14T17:10:39Z Using obfuscation key for /srv/slow/dot-bitcoin/chainstate: d57f45de91bd35fa
2022-01-14T17:10:39Z Loaded best chain: hashBestChain=000000000000000000001cab1821f8b5ce71faccfde4d9ae11ab444c0dff979f height=718643 date=2022-01-14T14:19:04Z progress=0.999964
2022-01-14T17:10:39Z init message: Verifying blocks…
2022-01-14T17:10:39Z Verifying last 6 blocks at level 3
2022-01-14T17:10:39Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2022-01-14T17:12:34Z No coin database inconsistencies in last 6 blocks (9259 transactions)
2022-01-14T17:12:34Z block index 120309ms
2022-01-14T17:12:34Z Opening LevelDB in /srv/slow/dot-bitcoin/indexes/txindex
2022-01-14T17:12:35Z Opened LevelDB successfully
2022-01-14T17:12:35Z Using obfuscation key for /srv/slow/dot-bitcoin/indexes/txindex: 0000000000000000
2022-01-14T17:12:35Z txindex thread start
2022-01-14T17:12:35Z txindex is enabled at height 718643
2022-01-14T17:12:35Z txindex thread exit
2022-01-14T17:12:35Z Opening LevelDB in /srv/slow/dot-bitcoin/indexes/blockfilter/basic/db
2022-01-14T17:12:35Z Opened LevelDB successfully
2022-01-14T17:12:35Z Using obfuscation key for /srv/slow/dot-bitcoin/indexes/blockfilter/basic/db: 0000000000000000
2022-01-14T17:12:35Z basic block filter index thread start
2022-01-14T17:12:35Z basic block filter index is enabled at height 718643
2022-01-14T17:12:35Z basic block filter index thread exit
2022-01-14T17:12:35Z Opening LevelDB in /srv/slow/dot-bitcoin/indexes/coinstats/db
2022-01-14T17:12:36Z Opened LevelDB successfully
2022-01-14T17:12:36Z Using obfuscation key for /srv/slow/dot-bitcoin/indexes/coinstats/db: 0000000000000000
2022-01-14T17:12:36Z coinstatsindex thread start
2022-01-14T17:12:36Z coinstatsindex is enabled at height 718643
2022-01-14T17:12:36Z coinstatsindex thread exit
2022-01-14T17:12:36Z block tree size = 718667
2022-01-14T17:12:36Z nBestHeight = 718643
2022-01-14T17:12:36Z loadblk thread start
....