HOPRd node has aggregation issues #6419
Description
Expected Behavior
Tickets aggregation should happen depending on aggregation_threshold value.
Current Behavior
It has aggregation issues, which leads to lockup unredeemed tickets:
2024-07-29T01:46:53.593902268Z 2024-07-29T01:46:53.593808Z INFO ThreadId(08) chain_indexer::block: Current indexer state at block #block #35204714 with 0 logs with checksum: 0x3dea04890e05d1927d5ad3ade73b14543593e042816496e554eb24f7276a5cfa
2024-07-29T01:46:53.990560652Z 2024-07-29T01:46:53.990351Z WARN ThreadId(09) core_protocol::ticket_aggregation::processor: Error occured on ticket aggregation for '0x151ae219c976ced3eee3b782669552f3df52b732bd71ba39b989f7075ccb2a5e', performing a rollback: underlying transport error while sending packet: Timed out on sending a packet
2024-07-29T01:46:53.990818810Z 2024-07-29T01:46:53.990595Z DEBUG ThreadId(09) sea_orm::database::transaction: SELECT "channel"."id", "channel"."channel_id", "channel"."source", "channel"."destination", "channel"."balance", "channel"."status", "channel"."epoch", "channel"."ticket_index", "channel"."closure_time" FROM "channel" WHERE "channel"."channel_id" = '0x151ae219c976ced3eee3b782669552f3df52b732bd71ba39b989f7075ccb2a5e' LIMIT 1
2024-07-29T01:46:53.991214931Z 2024-07-29T01:46:53.991062Z DEBUG ThreadId(497) sqlx::query: summary="SELECT \"channel\".\"id\", \"channel\".\"channel_id\", \"channel\".\"source\", …" db.statement="\n\nSELECT\n \"channel\".\"id\",\n \"channel\".\"channel_id\",\n \"channel\".\"source\",\n \"channel\".\"destination\",\n \"channel\".\"balance\",\n \"channel\".\"status\",\n \"channel\".\"epoch\",\n \"channel\".\"ticket_index\",\n \"channel\".\"closure_time\"\nFROM\n \"channel\"\nWHERE\n \"channel\".\"channel_id\" = ?\nLIMIT\n ?\n" rows_affected=1 rows_returned=1 elapsed=207.446µs
2024-07-29T01:46:53.991707066Z 2024-07-29T01:46:53.991426Z DEBUG ThreadId(09) sea_orm::database::transaction: UPDATE "ticket" SET "state" = 0 WHERE "ticket"."channel_id" = '0x151ae219c976ced3eee3b782669552f3df52b732bd71ba39b989f7075ccb2a5e' AND "ticket"."channel_epoch" = x'0000000000000000000000000000000000000000000000000000000000000005' AND "ticket"."state" = 2
2024-07-29T01:46:53.996612812Z 2024-07-29T01:46:53.995590Z DEBUG ThreadId(511) sqlx::query: summary="UPDATE \"ticket\" SET \"state\" …" db.statement="\n\nUPDATE\n \"ticket\"\nSET\n \"state\" = ?\nWHERE\n \"ticket\".\"channel_id\" = ?\n AND \"ticket\".\"channel_epoch\" = ?\n AND \"ticket\".\"state\" = ?\n" rows_affected=835 rows_returned=0 elapsed=3.963719ms
2024-07-29T01:46:53.996747616Z 2024-07-29T01:46:53.996612Z DEBUG ThreadId(09) hopr_db_api::tickets: rollback happened for ticket aggregation in '0x151ae219c976ced3eee3b782669552f3df52b732bd71ba39b989f7075ccb2a5e' with 835 tickets rolled back as a result
2024-07-29T01:46:53.996760577Z 2024-07-29T01:46:53.996657Z DEBUG ThreadId(09) core_strategy::aggregating: tried ticket aggregation in channel 0x151ae219c976ced3eee3b782669552f3df52b732bd71ba39b989f7075ccb2a5e without any issues
What I have checked:
- Checksum matches with my testing nodes. Output of one of my testing node, for the same block number: (
2024-07-29T01:46:52.591258438Z 2024-07-29T01:46:52.591149Z INFO ThreadId(07) chain_indexer::block: Current indexer state at block #block #35204714 with 0 logs with checksum: 0x3dea04890e05d1927d5ad3ade73b14543593e042816496e554eb24f7276a5cfa) - Node has enough xDai: 0.09
- Same issue which occurs on the other node as well, we checked checksum is correct, node has enough funds.
Strategy settings on both nodes:
{
"on_fail_continue": true,
"allow_recursive": true,
"strategies": {
"Aggregating": {
"aggregation_threshold": 100,
"unrealized_balance_ratio": 0.9,
"aggregate_on_channel_close": true
},
"AutoRedeeming": {
"redeem_only_aggregated": true,
"minimum_redeem_ticket_value": "30000000000000000000 HOPR",
"on_close_redeem_single_tickets_value_min": "90000000000000000 HOPR"
},
"ClosureFinalizer": {
"max_closure_overdue": 3600
}
}
}
Steps to Reproduce
- Can't reproduce
Were you able to reproduce the bug using the above steps consistently each time? NO
Detailed Description
HOPRd version: 2.1.3 (Public)
Logs
There are 2 logs, one from VPS and from the dappnode: https://drive.google.com/drive/folders/1jG8r0HL-4oKFGr2icxWeks7DIZ4ujbku?usp=sharing