Failed log processing in specific block due to database lock causing incorrect checksums for future blocks #6359
Description
Expected Behavior
During syncing process it should process all the blocks and it should have correct checksum for each block.
Current Behavior
During syncing process, node failed to process logs in block 31203880 because of error: Execution Error: error returned from database: (code: 5) database is locked
Output from the logs where this issue happened:
2024-06-06T14:57:50.929890747Z 2024-06-06T14:57:50.929801Z INFO ThreadId(09) chain_indexer::block: Processed block number: 31203843
2024-06-06T14:57:51.003544195Z 2024-06-06T14:57:51.003293Z INFO ThreadId(09) chain_indexer::block: Sync progress 32.40% @ block 31203843
2024-06-06T14:57:51.004447891Z 2024-06-06T14:57:51.004318Z INFO ThreadId(07) chain_indexer::block: retrieved 1 significant chain events from block #31203843 with 1 logs
2024-06-06T14:57:51.004873533Z 2024-06-06T14:57:51.004773Z INFO ThreadId(07) chain_indexer::block: Current indexer state at block #block #31203843 with 1 logs with checksum: 0xaa89077fe1a83b55e92967e0d0272b1912f8f5673a29b196c5b11a3b6b0655c7
2024-06-06T14:57:51.004899934Z 2024-06-06T14:57:51.004797Z INFO ThreadId(07) chain_indexer::block: Processed block number: 31203880
2024-06-06T14:57:51.078548713Z 2024-06-06T14:57:51.078402Z INFO ThreadId(08) chain_indexer::block: Sync progress 32.40% @ block 31203880
2024-06-06T14:57:51.079081838Z 2024-06-06T14:57:51.078928Z WARN ThreadId(08) Request{http.method=GET http.target=/api/v3/account/balances request_id="9728d8e3-92bd-471c-ab88-28f40291b9ee"}: hopr_lib: Safe balance in the DB 76036563530661000000000 HOPR mismatches on chain balance: 75378678175361000000000 HOPR
2024-06-06T14:57:51.079313005Z 2024-06-06T14:57:51.079234Z ERROR ThreadId(08) chain_indexer::block: failed to process logs in block #31203880 with 1 logs into events: Execution Error: error returned from database: (code: 5) database is locked
2024-06-06T14:57:51.079446109Z 2024-06-06T14:57:51.079367Z INFO ThreadId(08) chain_indexer::block: Current indexer state at block #block #31203880 with 1 logs with checksum: 0xaa89077fe1a83b55e92967e0d0272b1912f8f5673a29b196c5b11a3b6b0655c7
Issue summary
The issue occurs across all versions: 2.1.0, 2.1.1, and 2.1.2.
- It doesn't happen with every re-sync, but it has become more frequent.
- On my Avado setup, I tried syncing with different providers (
Nethermind + Lodestar,Nethermind + Nimbus) without success in achieving a proper checksum. - On my VPS, I attempted to sync using the public node RPC provider and my local
Nethermind + LodestarRPC provider. After several attempts, I successfully synced my node on my VPS. - Others have also encountered this issue on their VPS & dappnode setups.
Steps to Reproduce
- You can try to re-sync your node from scratch and after full sync check if latest block has a correct hash, if not you should search for DB lock, Use debug log type to get more details.
Were you able to reproduce the bug using the above steps consistently each time? NO
Detailed Description
HOPRd version: 2.1.0
Device: VPS
Setup mode: https://github.com/hoprnet/hoprnet/tree/master/deploy/compose
Logs
Full logs: https://drive.google.com/file/d/1ZPBr7hvhC6HXYh_bCFcWLojtL04nhXvD/view?usp=sharing