Ping in E2E tests failing consistently #6529
Description
Expected Behavior
The E2E tests in the CI should pass the same way as they pass locally in a Linux environment.
Current Behavior
When the CI uses an Anvil snapshot to run the tests, the
tests/test_integration.py::test_hoprd_ping_should_work_between_nodes_in_the_same_network
fails consistently.
See the logs for details.
(Optional) Possible Solution
The problem might be likely due to the use of the Anvil snapshot mechanism that triggers a race condition in
the on-annoucement ping.
Per @Teebor-Choka :
Tibor: We can discard the on announce multiaddress dial.
Tibor: But there will be this piece of code, which will have to deregister a peer and re-register with a new address on announce (pls add to the issue).
Steps to Reproduce
- run
test_integrationin the CI smoke tests
Were you able to reproduce the bug using the above steps consistently each time? YES/
Logs
The on-announcement part:
2024-10-02T04:54:42.016846Z DEBUG ThreadId(06) hopr_transport_p2p::behavior::discovery: p2p - discovery - Announcement peer=12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC addresses=[/ip4/127.0.0.1/tcp/19042]
2024-10-02T04:54:42.016977Z INFO ThreadId(06) hopr_transport_p2p::swarm: New record peer=12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC multiaddress=/ip4/127.0.0.1/tcp/19042
2024-10-02T04:54:42.017124Z ERROR ThreadId(06) hopr_transport_p2p::swarm: Failed to dial the peer: Dial error: dial condition was configured to only happen when both disconnected (`PeerCondition::Disconnected`) and there is currently no ongoing dialing attempt (`PeerCondition::NotDialing`), but node is already connected or dial is in progress, thus cancelling new dial. peer=12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC address=/ip4/127.0.0.1/tcp/19042
2024-10-02T04:54:42.017222Z DEBUG ThreadId(06) hopr_transport_p2p::behavior::discovery: p2p - discovery - Network registry allow peer=12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC
2024-10-02T04:54:42.023294Z DEBUG ThreadId(05) hopr_transport_p2p::behavior::discovery: p2p - discovery - Announcement peer=12D3KooWHc2LPyvYGLJbHoQeJUBXBjDRfMY7msobsPoWj8rCAGHr addresses=[/ip4/127.0.0.1/tcp/19062]
the manual ping part:
2024-10-02T04:54:52.175458Z DEBUG ThreadId(06) request{method=POST uri=/api/v3/peers/12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC/ping version=HTTP/1.1}:ping{peer=PeerId("12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC")}: core_network::ping: Ping to '12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC' failed
2024-10-02T04:54:52.181657Z ERROR ThreadId(04) request{method=POST uri=/api/v3/peers/12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC/ping version=HTTP/1.1}:ping{peer=PeerId("12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC")}: hopr_transport: Manual ping failed: Network monitoring error: error while pinging peer 12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC: ping error peer="12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC"
2024-10-02T04:54:52.182103Z DEBUG ThreadId(04) request{method=POST uri=/api/v3/peers/12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC/ping version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=15014 ms status=422
2024-10-02T04:54:52.182337Z DEBUG ThreadId(04) hopr_transport_p2p::behavior::discovery: p2p - discovery - Closing connection (reason: low ping connection quality peer=12D3KooWA494BRhXs2DpMm5e2DWkPZcot3WpYwB4KBj2udP9xvPC