ReplyOpeners missing for the decoding procedure #7288
Description
The rotsee nodes encounter issues with missing reply openers causing the decoding process to stall and fail on the msg protocol completely
Current Behavior
The nodes cannot decode neither a message, nor an ack in #7255 :
2025-07-04T08:14:26.149502Z DEBUG ThreadId(07) hopr_transport_protocol::stream: Opening outgoing peer-to-peer stream peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7
2025-07-04T08:19:00.956052Z DEBUG ThreadId(12) hopr_transport_probe::probe: probe failed peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 pseudonym=0xef124099c0e57fb7c4e1 probe=Ping(1ed4dccaee418e7cc8d762dd22a73630eaa7af995dd48cb4b83f554073484008) reason="timeout"
2025-07-04T08:19:10.969448Z DEBUG ThreadId(11) hopr_transport_probe::probe: probe failed peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 pseudonym=0xef124099c0e57fb7c4e1 probe=Ping(9357c05387826bddcc116c3b08818929f8364e5eff57fcbde0d44b942674e521) reason="timeout"
2025-07-04T08:19:55.089018Z ERROR ThreadId(13) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: logical error: failed to construct an incoming packet: failed to decode packet: couldn't find reply opener for pseudonym: 0x02be8be1d59807644ff4594b7f915ee8e52e
2025-07-04T08:19:55.183185Z ERROR ThreadId(13) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: logical error: failed to construct an incoming packet: failed to decode packet: couldn't find reply opener for pseudonym: 0x02be8be1d59807644ff46e333c8e1d44d4b7
2025-07-04T08:19:55.183767Z ERROR ThreadId(13) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: ack validation error: received unexpected acknowledgement for half key challenge 0xae95e5904e7fa4277f734f15ffbfa48403f5ca8bf453324946a8d7239b0dced38596c416b8bc54e28a53a9aaf8041662e6c8a3de6a6057b5510e9f0e30bd0073e14196ac51dcaa13946930a63cd232626b9d8e444897a817f5b9b73a5d68250c
2025-07-04T08:19:55.184447Z ERROR ThreadId(13) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: ack validation error: received unexpected acknowledgement for half key challenge 0xd20205a8e7bd1722f6c7ff2fa5c4eccca0577e4ccd36fa5c84a3df8a86c5412d63fac616ad76daf35082a2d4a22b8f7624c0068505af9cf84454fe3b07fbe775ccb27fafb7b81120b0e6c412f35226eeb5044447039325bab537d32cca42e90f
2025-07-04T08:24:02.341551Z ERROR ThreadId(09) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: ack validation error: received unexpected acknowledgement for half key challenge 0xc8d6dd7fa8f7b337df20dd62347659d89774e45c7c29d3a213350a1d9ed341bc792e19b12cdfb035ca0ea5682316a31a6189eabe5edb799ab71fbcbec783da342b36585cda821ad2d5fd640ea66346561d2ee7d89967cf067397e90877030a0e
2025-07-04T08:24:02.342082Z ERROR ThreadId(09) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: ack validation error: received unexpected acknowledgement for half key challenge 0xc85dca0e79b7eb821ac40ebdce756ac81da500cdd84d1fa802f0eb623da75b7fde4d3c9abe4092a620ad978c2e42c5861389194935620a6f4e6ef5f32549f8da79ea1c62bfcc0dd81cd9f63b32c5ba2431f5aa6c7879c9dda4171045596daf0b
2025-07-04T08:24:16.336426Z DEBUG ThreadId(07) hopr_transport_probe::probe: probe failed peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 pseudonym=0x2b8ef6b6f1761b652e78 probe=Ping(665bc36e8b15d3334719f6b665f8d526ffbc0234ee986d3a716c37e3fec0daab) reason="timeout"
2025-07-04T08:24:21.347170Z DEBUG ThreadId(07) hopr_transport_probe::probe: probe failed peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 pseudonym=0x2b8ef6b6f1761b652e78 probe=Ping(50268b0500e8a4697ff1516c7dfd6556d9817196d6fe0d727110a8d5e135f7e0) reason="timeout"
2025-07-04T08:25:14.455647Z ERROR ThreadId(07) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWNGfx1dbsGRVVs2P8fxr6jAFCpZhEhGUGb3QPxEdYqKV7 error=failed to construct packet: DB general error: logical error: failed to construct an incoming packet: failed to decode packet: couldn't find reply opener for pseudonym: 0x02be8be1d59807644ff4ede00ad77b5c002d
The same can be observed for different machines:
2025-07-04T08:35:14.466367Z ERROR ThreadId(12) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWMmJNvw7Fyd7PxPpeC5oRzU2ykRiuHAzMseLuZ6ySVgUq error=failed to construct packet: DB general error: logical error: failed to construct an incoming packet: failed to decode packet: couldn't find reply opener for pseudonym: 0x40a72c7b7be32a2f4f8f84e64b5b10e27a0a
2025-07-04T08:35:14.466857Z ERROR ThreadId(12) msg protocol processing - incoming: hopr_transport_protocol: Failed to process the received message peer=12D3KooWMmJNvw7Fyd7PxPpeC5oRzU2ykRiuHAzMseLuZ6ySVgUq error=failed to construct packet: DB general error: ack validation error: received unexpected acknowledgement for half key challenge 0x8e52565f69b6874d2fcbb58a020ceab3979ffe2d5b068f3fb8b230aa4f3111a5fdca58750696577141db3d85a93601838d46ea9dacee995ae5fb5b75e3b71b483f2d9478f52430dddd82563579b602cdd7c2a798831a1c1b7e82662464f2ed00
Expected Behavior
The decoding process does not fail.
(Optional) Possible Solution
Steps to Reproduce
- run a rotsee node with fix(hopr-lib): Connectability issues in the local cluster and production #7255
Were you able to reproduce the bug using the above steps consistently each time? YES
Context (Environment)
rotsee with #7255
Metadata
Metadata
Assignees
Labels
Type
Projects
Status