Hash P2P messages as they are received instead of at process-time #9045
Conversation
|
utACK. Somehow I thought we were already doing this. |
|
ACK |
|
@TheBlueMatt Did you time how long the hashing was taking on a typical full block? |
|
@rebroad This is about P2P messages and their checksums... |
|
ACK fe1dc62 |
|
I did not Rome the hashing, though I expect it to be between 3 and 10 ms (we use a particularly slow SHA256 implementation, as we don't incorporate any handcoded asm), but the lower bound is about 3, assuming it's cache-hot. On October 31, 2016 4:50:20 AM EDT, R E Broadley notifications@github.com wrote:
|
|
mm, NACK unless there are numbers that show that this is preferable.
|
|
@theuni While hashing is incredibly fast, it can be cause a big hit to latency for time-from-block-receive-to-relay (lower bound on hashing 1MB is around 3ms, whereas total runtime for deserialize and AcceptBlock, not including storage, is something like 15ms), especially if you received the full "block" message. Doing it as packets come in off the wire should never be a bottleneck - hashing is, more often than not, going to be able to easily keep up with your wire speed. This means that you can hash the message as it comes off the wire and when you receive the last packet (ie are ready to start processing) you dont have to wait for hashing to complete. Indeed, it complicates CConnman somewhat but CNetMessage will likely have to change for protocol encryption anyway, so its not like the logic in the same place isnt gonna have to learn protocol encryption. |
|
@TheBlueMatt ok, I've come around on this. Thanks to @gmaxwell and @sipa for the discussion over the weekend. utACK fe1dc62. |
…cess-time fe1dc62 Hash P2P messages as they are received instead of at process-time (Matt Corallo)
… at process-time fe1dc62 Hash P2P messages as they are received instead of at process-time (Matt Corallo)
… at process-time fe1dc62 Hash P2P messages as they are received instead of at process-time (Matt Corallo)
… at process-time fe1dc62 Hash P2P messages as they are received instead of at process-time (Matt Corallo)
…t framework update) 7b04c0a [Test] Clean duplicate connections creation in mining_pos_coldStaking.py (furszy) 15a799e [Test] MAX_PROTOCOL_MESSAGE_LENGTH PIVXified. (furszy) 0aedf35 [tests] Don't import asyncio to test magic bytes (John Newbery) 9bb5309 Refactor resource exhaustion test (Troy Giorshev) 589a780 Fix "invalid message size" test (Troy Giorshev) 8a0c12b Move size limits to module-global (Troy Giorshev) b3391c5 Remove two unneeded tests (Troy Giorshev) 1404e68 test: Add various low-level p2p tests (furszy) 8aaf7e1 test: Remove fragile assert_memory_usage_stable (MarcoFalke) f68e22c [Test] p2p_invalid_messages.py do not change msg.command in test_command and raise sync_with_ping timeout (furszy) c851c0b Test framework: Wait for verack by default on every new connection (furszy) c02e9a0 [Test] move framework subversion to string (furszy) 3472a39 Replace coroutine with async def in p2p_invalid_messages.py (Elichai Turkel) 33c7b19 net_processing: align msg checksum check properly. (furszy) 7f71b1b Hash P2P messages as they are received instead of at process-time (furszy) 215a638 test: Skip flaky p2p_invalid_messages test on macOS (Fabian Jahr) c11f565 qa: Make swap_magic_bytes in p2p_invalid_messages atomic (MarcoFalke) be979ad test: Fix race in p2p_invalid_messages (MarcoFalke) 6a72f0c qa: Add tests for invalid message headers (MarcoFalke) 51ddd3d Introduce p2p_invalid_messages.py test (furszy) 55a37b5 net: fix missing jump line in "Oversized message from peer.." error. (furszy) 0edfce5 test_node: get_mem_rss fixups (MarcoFalke) 6f21213 tests: add P2PConnection.send_raw_message (James O'Beirne) ae68c6e tests: add utility to assert node memory usage hasn't increased (James O'Beirne) 8469afe test: forward timeouts properly in send_blocks_and_test (James O'Beirne) db28a53 Skip is_closing() check when not available. (Daniel Kraft) be9dacb tests: fixes mininode's P2PConnection sending messages on closing transport (marcoagner) 53599c8 qa: Avoid start/stop of the network thread mid-test (furszy) 688190c [qa] mininode: Expose connection state through is_connected (MarcoFalke) Pull request description: Part of the deep and long net and ser work that I'm doing (and Tor v3 onion addresses support). Friend of #2359. Focused on the end goal of implementing the `p2p_invalid_messages` functional test which validates that invalid msg headers, msg types, oversized payloads and inventory msgs aren't accepted nor can cause a resource exhaustion. And an extra covered scenario, in `p2p_invalid_tx.py`, for the orphan pool overflow. Plus, to get up to the goal, had to work over the functional test framework as well. So.. adapted list: * bitcoin#9045. * bitcoin#13512. * bitcoin#13517. * bitcoin#13715. * bitcoin#13747. * bitcoin#14456. * bitcoin#14522. * bitcoin#14672. * bitcoin#14693. * bitcoin#14812. * bitcoin#15246. * bitcoin#15330. * bitcoin#15697. * bitcoin#16445. * bitcoin#17931. * bitcoin#17469. * bitcoin#18628 (only `p2p_invalid_tx.py` and `p2p_invalid_messages.py`. We don't support the other tests yet). * bitcoin#19177. * bitcoin#19264. ACKs for top commit: random-zebra: utACK 7b04c0a and merging... Tree-SHA512: 48d1f1a6acc24a9abce033fbf1f281ba4392147da39d118a694a09d63c3e0610cc1a29d711b434b16cc0d0e030dd9f1a89843870091b6999b862b9ab18a20679
Better to hash as you receive bytes from the wire so that you reduce latency when you have the full message, arguably. If anything else probably best to not do the hashing in the message processing thread.