thread 'tokio-runtime-worker' has overflowed its stack and then restart #6656
Description
Expected Behavior
- doesn't crash
- mark the overflow with word ERROR in the logs to easily find it next time.
Current Behavior
Crashes
Steps to Reproduce
Just sync from scratch.
Did not try to reproduce. DM me if I should.
Context (Environment)
2.2.0-rc.1+commit.faf67d15 and 2.2.0-rc.1+commit.ced1f5f2
Local RPC, nethermind 1.29.1
docker run --pull always -d --restart always --name hopr-r -m 3g -e OPR_INTERNAL_LIBP2P_YAMUX_MAX_NUM_STREAMS=1024 --platform linux/x86_64 --log-driver json-file --log-opt max-size=100M --log-opt max-file=5 -ti -v <REDACTED> -p <REDACTED>:<REDACTED>/tcp -p <REDACTED>:<REDACTED>/udp -p <REDACTED>:<REDACTED> europe-west3-docker.pkg.dev/hoprassociation/docker-images/hoprd:latest --network rotsee --init --api --identity /app/hoprd-db/.hopr-id --data /app/hoprd-db --password <REDACTED> --apiHost "0.0.0.0" --apiToken <REDACTED> --announce --safeAddress <REDACTED> --moduleAddress <REDACTED> --host <REDACTED> --provider http://<local-nethermind>
Logs
2024-11-20T17:54:18.337902Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30905831 log_count=1 last_log_checksum="0x2fe21a4e1a356573c84fa04218087b78963d2040fb4636d0be9df062c30a7492"
2024-11-20T17:54:18.338314Z INFO ThreadId(13) chain_indexer::block: Sync progress indexer="rpc" progress=16.354287348788986 block=30905906
2024-11-20T17:54:18.342049Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30905906 log_count=5 last_log_checksum="0x79e51eb180d7a3557a1df13fb5629d93d661503138b2ee91da9226899de097da"
2024-11-20T17:54:18.342495Z INFO ThreadId(13) chain_indexer::block: Sync progress indexer="rpc" progress=16.3543142573565 block=30905908
2024-11-20T17:54:18.344526Z INFO ThreadId(13) chain_indexer::block: Indexer state update block_number=30905908 log_count=1 last_log_checksum="0x40e2ff41bb01f41ab7cd26fa77e379041b6fc85fe262027ea1575b0ea6143f3d"
2024-11-20T17:54:18.345108Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.35435462020777 block=30905911
2024-11-20T17:54:18.372408Z INFO ThreadId(13) chain_indexer::block: Indexer state update block_number=30905911 log_count=1 last_log_checksum="0x0c6ff1797f91674a31cecb759337be44eb88988a5a03b5f659fe640430127444"
2024-11-20T17:54:18.372934Z INFO ThreadId(10) chain_indexer::block: Sync progress indexer="rpc" progress=16.354489163045336 block=30905921
2024-11-20T17:54:18.375521Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30905921 log_count=1 last_log_checksum="0x386ccc63536612e86ed8d701144cca03fca063ac7d70cbea4b246e428ca357e5"
2024-11-20T17:54:18.375561Z WARN ThreadId(11) hopr_chain_rpc::client: RPC call will retry method="eth_blockNumber" backoff_in_ms=1000
2024-11-20T17:54:19.377865Z INFO ThreadId(13) chain_indexer::block: Sync progress indexer="rpc" progress=16.356022951393577 block=30906035
2024-11-20T17:54:19.380010Z INFO ThreadId(13) chain_indexer::block: Indexer state update block_number=30906035 log_count=1 last_log_checksum="0xd2b279198aa19036dd3046dd130bd5bdaa5cad09bf6e132fceea4348cb454176"
2024-11-20T17:54:19.380498Z INFO ThreadId(10) chain_indexer::block: Sync progress indexer="rpc" progress=16.356157494231137 block=30906045
2024-11-20T17:54:19.382502Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30906045 log_count=1 last_log_checksum="0x2f79786bd2590b3c1c9759455e8ee5f27eda52349c194c4d6c126db800df1536"
2024-11-20T17:54:19.383009Z INFO ThreadId(13) chain_indexer::block: Sync progress indexer="rpc" progress=16.35627858278495 block=30906054
2024-11-20T17:54:19.386142Z INFO ThreadId(10) chain_indexer::block: Indexer state update block_number=30906054 log_count=1 last_log_checksum="0x8c925e6736520e14d6832a28cf88813bd35f0a12217d865856e608b2ffa3bc0a"
2024-11-20T17:54:19.386843Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.356413125622513 block=30906064
2024-11-20T17:54:19.390023Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30906064 log_count=2 last_log_checksum="0x4feb2aa847e60808fd3e4af854ded6c3007753cdee118fac8413714c86f2e845"
2024-11-20T17:54:19.390588Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.35653421417632 block=30906073
2024-11-20T17:54:19.393929Z INFO ThreadId(11) chain_indexer::block: Indexer state update block_number=30906073 log_count=1 last_log_checksum="0x4b02f9894334f4a1043606137b14f7ec3050def437eab62fd9704afc6554201c"
2024-11-20T17:54:19.394526Z INFO ThreadId(02) chain_indexer::block: Sync progress indexer="rpc" progress=16.356547668460077 block=30906074
2024-11-20T17:54:19.396819Z INFO ThreadId(02) chain_indexer::block: Indexer state update block_number=30906074 log_count=1 last_log_checksum="0x510eae90e4cb3ae3b20eb80206d2b2211fbf043b654e0098d73b658e15230beb"
2024-11-20T17:54:19.397402Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.35665530273013 block=30906082
2024-11-20T17:54:19.400494Z INFO ThreadId(10) chain_indexer::block: Indexer state update block_number=30906082 log_count=1 last_log_checksum="0x5945a52f613f40e55b1952c4d40edafea2c8cb2e0e13440cbe1938eedae75096"
2024-11-20T17:54:19.400960Z INFO ThreadId(02) chain_indexer::block: Sync progress indexer="rpc" progress=16.356762937000184 block=30906090
2024-11-20T17:54:19.404131Z INFO ThreadId(02) chain_indexer::block: Indexer state update block_number=30906090 log_count=2 last_log_checksum="0x2be3632926897733ae32a44ececff8b69919b4954bade270e477bf3ebc5868ba"
2024-11-20T17:54:19.404725Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.35677639128394 block=30906091
2024-11-20T17:54:19.407407Z INFO ThreadId(10) chain_indexer::block: Indexer state update block_number=30906091 log_count=1 last_log_checksum="0x018bc84a01ba5692816dee0c31ff025b58693dea3c817f726e1a7589c9a3103f"
2024-11-20T17:54:19.407852Z INFO ThreadId(10) chain_indexer::block: Sync progress indexer="rpc" progress=16.3569109341215 block=30906101
2024-11-20T17:54:19.411321Z INFO ThreadId(02) chain_indexer::block: Indexer state update block_number=30906101 log_count=3 last_log_checksum="0x01911b9d930aed2df5d8ac4387c2a94e92e3deebd12617f07d6c2564b199bea2"
2024-11-20T17:54:19.411791Z INFO ThreadId(11) chain_indexer::block: Sync progress indexer="rpc" progress=16.357018568391553 block=30906109
2024-11-20T17:54:19.414587Z INFO ThreadId(02) chain_indexer::block: Indexer state update block_number=30906109 log_count=2 last_log_checksum="0xdc95593db076d564ee2b450bd4f720d802634517a85ba2c58eefe201f990e627"
2024-11-20T17:54:19.414626Z WARN ThreadId(02) hopr_chain_rpc::client: RPC call will retry method="eth_blockNumber" backoff_in_ms=1000
thread 'tokio-runtime-worker' has overflowed its stack
fatal runtime error: stack overflow
2024-11-20T17:54:21.401426Z INFO ThreadId(01) hoprd: Starting HOPR daemon version="2.2.0-rc.1+commit.faf67d15" ...