Description

There's a bug lurking in the database code that can cause occasional database corruption. It doesn't happen consistently, but when it does it seems to result in a zero hash (0x00) appearing in the randao_mixes array.

The case I'm investigating presented as corruption at slot 135168 on Prater:

Feb 08 07:31:26.461 ERRO State reconstruction failed             error: HotColdDBError(BlockReplayBlockError(HeaderInvalid { reason: ParentBlockRootMismatch { state: 0x373eb699eae0110e474671cab72d5c6ca666d4a6f5a5a356f2af89039ad98382, block: 0xabf45deec98af2873a04d352ebbc54eac35d00c8157fea27b21f9adc2446233b } })), service: beacon

Oddly the first corrupt state actually occurs much earlier. I found that the state at slot 12288 was corrupt using this (fish) script:

for i in (seq 0 2048 135168)
    set checksum (curl -s -H "Accept: application/octet-stream" "http://localhost:5052/eth/v2/debug/beacon/states/$i" | sha256sum)
    echo "$i: $checksum"
end

Diffing the corrupt state at slot 12288 against the real state reveals a 0x00 value in the randao_mixes at index 320. This is interesting because that corresponds to epoch 320, i.e. 64 epochs prior to slot 12288 (epoch 384).

I think the bug must be in store_updated_vector, which is responsible for writing the randao mixes in the flat format used by the database:

It's possible that we're somehow re-writing the old state at 12288 which inapproriately zeroes some entries and corrupts all subsequent states. I don't think the corruption can occur the first time state 12288 is written else it would have failed the block root check at that point or shortly after.

Will update this issue with more info soon.