Skip to content

Conversation

maflcko
Copy link
Member

@maflcko maflcko commented Jun 21, 2018

This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

@maflcko maflcko added the Tests label Jun 21, 2018
@maflcko
Copy link
Member Author

maflcko commented Jun 21, 2018

Please review the refactoring in #13512 first.

@jnewbery
Copy link
Contributor

Big concept ACK!

feature_block.py currently failing on Travis.

Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursory review. A few nits inline.

Will do full review once #13512 is merged.

@@ -71,23 +71,23 @@ P2P messages. These can be found in the following source files:

#### Using the P2P interface

- Set `setup_mininode_event_loop = True` in `set_test_params()` to spawn a thread
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to start the network thread for all tests, so it's not necessary for the test writer to even perform this step?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I guess there is no overhead in running an event loop with no events.

- `mininode.py` contains all the definitions for objects that pass
over the network (`CBlock`, `CTransaction`, etc, along with the network-level
wrappers for them, `msg_block`, `msg_tx`, etc).

- P2P tests have two threads. One thread handles all network communication
with the bitcoind(s) being tested (using python's asyncore package); the other
with the bitcoind(s) being tested; the other
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Join lines

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

md renderer will take care of this ;)

@@ -15,7 +15,7 @@

from test_framework.test_framework import BitcoinTestFramework
from test_framework.util import *
from test_framework.mininode import CTransaction, network_thread_start
from test_framework.mininode import CTransaction
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be changed to from test_framework.messages import CTransaction :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

;)

@@ -13,11 +13,10 @@
P2PInterface: A high-level interface object for communicating to a node over P2P
P2PDataStore: A p2p interface class that keeps a store of transactions and blocks
and can respond correctly to getdata and getheaders messages"""
import asyncore
import asyncio
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@@ -41,14 +41,14 @@
)
from test_framework.mininode import (
P2PInterface,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can join to a single line

@maflcko maflcko force-pushed the Mf1806-qaAsyncIo branch 8 times, most recently from c0077ce to aaaa4a4 Compare June 24, 2018 18:14
@maflcko
Copy link
Member Author

maflcko commented Jun 24, 2018

aaaaddressed @jnewbery feedback in aaaa4a4b17506f526bebee65b0ea930289f79016

Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK aaaa4a4b17506f526bebee65b0ea930289f79016.

Looks great Marco. Thanks for doing this. A few nits inline - feel free to ignore.

@@ -160,9 +158,8 @@ def run_test(self):
self.block_time += 1
height += 1

# We're adding new connections so terminate the network thread
# We're adding new connections and terminate the old ones
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: just remove this comment entirely. It adds no information to the code.


@property
def is_connected(self):
return self._conn_open
return bool(self._transport)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think return self._transport is not None is preferable

# There is only one event loop and no more than one thread must be created
assert not self._network_event_loop

NetworkThread._network_event_loop = asyncio.new_event_loop()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is accessed outside the class, so it shouldn't be marked as private. Prefer network_event_loop


def handle_close(self):
"""asyncore callback when a connection is closed."""
def connection_made(self, _transport):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: prefer transport as the argument name

def connection_lost(self, exc):
"""asyncio callback when a connection is closed."""
if exc:
logger.warning("Connection lost to {}:{} due to {}".format(self.dstaddr, self.dstport, exc))
logger.debug("Closing connection to: %s:%d" % (self.dstaddr, self.dstport))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this should be in an else branch to avoid log repetition

@@ -32,6 +32,8 @@
sync_blocks,
sync_mempools,
)
from .mininode import NetworkThread
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: sort imports

@maflcko maflcko force-pushed the Mf1806-qaAsyncIo branch from aaaa4a4 to fa87da2 Compare June 25, 2018 18:08
@maflcko
Copy link
Member Author

maflcko commented Jun 25, 2018

Force pushed to address feedback

@jnewbery
Copy link
Contributor

Tested ACK fa87da2

@DrahtBot
Copy link
Contributor

DrahtBot commented Jun 26, 2018

Note to reviewers: This pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@laanwj
Copy link
Member

laanwj commented Jun 29, 2018

utACK fa87da2

@laanwj laanwj merged commit fa87da2 into bitcoin:master Jun 29, 2018
laanwj added a commit that referenced this pull request Jun 29, 2018
fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
@maflcko maflcko deleted the Mf1806-qaAsyncIo branch June 30, 2018 10:52
random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request Jun 28, 2021
…t framework update)

7b04c0a [Test] Clean duplicate connections creation in mining_pos_coldStaking.py (furszy)
15a799e [Test] MAX_PROTOCOL_MESSAGE_LENGTH PIVXified. (furszy)
0aedf35 [tests] Don't import asyncio to test magic bytes (John Newbery)
9bb5309 Refactor resource exhaustion test (Troy Giorshev)
589a780 Fix "invalid message size" test (Troy Giorshev)
8a0c12b Move size limits to module-global (Troy Giorshev)
b3391c5 Remove two unneeded tests (Troy Giorshev)
1404e68 test: Add various low-level p2p tests (furszy)
8aaf7e1 test: Remove fragile assert_memory_usage_stable (MarcoFalke)
f68e22c [Test] p2p_invalid_messages.py do not change msg.command in test_command and raise sync_with_ping timeout (furszy)
c851c0b Test framework: Wait for verack by default on every new connection (furszy)
c02e9a0 [Test] move framework subversion to string (furszy)
3472a39 Replace coroutine with async def in p2p_invalid_messages.py (Elichai Turkel)
33c7b19 net_processing: align msg checksum check properly. (furszy)
7f71b1b Hash P2P messages as they are received instead of at process-time (furszy)
215a638 test: Skip flaky p2p_invalid_messages test on macOS (Fabian Jahr)
c11f565 qa: Make swap_magic_bytes in p2p_invalid_messages atomic (MarcoFalke)
be979ad test: Fix race in p2p_invalid_messages (MarcoFalke)
6a72f0c qa: Add tests for invalid message headers (MarcoFalke)
51ddd3d Introduce p2p_invalid_messages.py test (furszy)
55a37b5 net: fix missing jump line in "Oversized message from peer.." error. (furszy)
0edfce5 test_node: get_mem_rss fixups (MarcoFalke)
6f21213 tests: add P2PConnection.send_raw_message (James O'Beirne)
ae68c6e tests: add utility to assert node memory usage hasn't increased (James O'Beirne)
8469afe test: forward timeouts properly in send_blocks_and_test (James O'Beirne)
db28a53 Skip is_closing() check when not available. (Daniel Kraft)
be9dacb tests: fixes mininode's P2PConnection sending messages on closing transport (marcoagner)
53599c8 qa: Avoid start/stop of the network thread mid-test (furszy)
688190c [qa] mininode: Expose connection state through is_connected (MarcoFalke)

Pull request description:

  Part of the deep and long net and ser work that I'm doing (and Tor v3 onion addresses support). Friend of #2359.

  Focused on the end goal of implementing the `p2p_invalid_messages` functional test which validates that invalid msg headers, msg types, oversized payloads and inventory msgs aren't accepted nor can cause a resource exhaustion. And an extra covered scenario, in `p2p_invalid_tx.py`, for the orphan pool overflow.

  Plus, to get up to the goal, had to work over the functional test framework as well.

  So.. adapted list:

  * bitcoin#9045.
  * bitcoin#13512.
  * bitcoin#13517.
  * bitcoin#13715.
  * bitcoin#13747.
  * bitcoin#14456.
  * bitcoin#14522.
  * bitcoin#14672.
  * bitcoin#14693.
  * bitcoin#14812.
  * bitcoin#15246.
  * bitcoin#15330.
  * bitcoin#15697.
  * bitcoin#16445.
  * bitcoin#17931.
  * bitcoin#17469.
  * bitcoin#18628 (only `p2p_invalid_tx.py` and `p2p_invalid_messages.py`. We don't support the other tests yet).
  * bitcoin#19177.
  * bitcoin#19264.

ACKs for top commit:
  random-zebra:
    utACK 7b04c0a and merging...

Tree-SHA512: 48d1f1a6acc24a9abce033fbf1f281ba4392147da39d118a694a09d63c3e0610cc1a29d711b434b16cc0d0e030dd9f1a89843870091b6999b862b9ab18a20679
UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Jun 29, 2021
…tests

fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Jun 29, 2021
…tests

fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Jul 1, 2021
…tests

fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Jul 2, 2021
…tests

fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Jul 2, 2021
…tests

fa87da2 qa: Avoid start/stop of the network thread mid-test (MarcoFalke)

Pull request description:

  This simplifies test writing by removing the need to handle the network thread in tests. E.g. start thread, join thread, restart thread mid-test, adding p2p connections at the "right" time, ...

Tree-SHA512: 533642f12fef5496f1933855edcdab1a7ed901d088d34911749cd0f9e044c8a6cb1f89985ac3a7f41a512943663e4e270a61978f6f072143ae050cd102d4eab8
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants