-
Notifications
You must be signed in to change notification settings - Fork 37.8k
net: Decouple CConnman and message serialization #9128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -6502,9 +6509,13 @@ bool SendMessages(CNode* pto, CConnman& connman) | |||
const Consensus::Params& consensusParams = Params().GetConsensus(); | |||
{ | |||
// Don't send anything until we get its version message | |||
if (pto->nVersion == 0) | |||
if (!pto->fSuccessfullyConnected || pto->fDisconnect) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you!!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Don't send anything until we get its version message
Comment is out-of-date?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dcousens the version message sets fSuccessfullyConnected, but I guess that's not as clear after this change.
How about "don't send anything until we're successfully connected and ok with the peer's version" ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think if we are still referencing version here to explain why, then fSuccessfullyConnected
isn't appropriate.
Otherwise, just say "don't send anything until we're successfully connected".
Grr, will fix up the Windows build problem in the morning. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
netmessagemaker and handling of initial Version message in main.cpp are good improvements.
Concept ACK, though I'm not convinced there arent too many copies when making the message and passing it into CConnman, but will audit when you fix up build, etc. |
@TheBlueMatt there's only 1 copy (other than the serialization itself), and it comes from combining the header and payload. See the note here: https://github.com/bitcoin/bitcoin/pull/9128/files#diff-9a82240fe7dfe86564178691cc57f2f1R2590 Either way, it for sure cuts down on copies compared to before. |
@@ -5723,8 +5730,8 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv, | |||
pfrom->id, | |||
FormatStateMessage(state)); | |||
if (state.GetRejectCode() < REJECT_INTERNAL) // Never send AcceptToMemoryPool's internal codes over P2P | |||
connman.PushMessage(pfrom, NetMsgType::REJECT, strCommand, (unsigned char)state.GetRejectCode(), | |||
state.GetRejectReason().substr(0, MAX_REJECT_MESSAGE_LENGTH), inv.hash); | |||
connman.PushMessage(pfrom, msgMaker.Make(NetMsgType::REJECT, strCommand, (unsigned char)state.GetRejectCode(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: indent
Concept ACK. |
winsok needs an extra cast:
|
9d4bf93
to
6d331d6
Compare
Fixed several things:
|
I'm testing this + #9125 |
Needs rebase after #9075. |
{ | ||
return nType; | ||
} | ||
void skip(size_t nSize) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CSizeComputer uses 'seek' for this function (inspired by the dd command, where skip is something you do on input, and seek is something you do on output).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Will change.
* | ||
* Note that the referenced vector must already be large enough to hold all data to be serialized. | ||
*/ | ||
class CVectorOverWriter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not convinced there is a need to separate CVectorOverWriter and and CVectorInserter. Couldn't you have a single one, which a constructor that initialized the write pointer at the beginning, and one that initializes it at the end?
Writes would involve first checking whether we'd not go out of bounds, and resizing if necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I think that should work fine. Will change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again didnt do a full review, just looked over the actual code changes and came up with a few comments...will do a full review when they're addressed.
|
||
unsigned int nSize = strm.size() - CMessageHeader::HEADER_SIZE; | ||
LogPrint("net", "sending %s (%d bytes) peer=%d\n", SanitizeString(sCommand.c_str()), nSize, pnode->id); | ||
uint256 hash = Hash(msg.data.data() + msg.header_pad_size, msg.data.data() + msg.data.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldnt this happen inside CNetMsgMaker::Make? That way we can just have a BIP151MsgMaker?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was left out of CNetMsgMaker because I assumed that we'd want CNode making the decision about how to wrap the data for sending. If it's to be encrypted, we'll need some per-node state vars.
So I guess it depends whether encryption status and state are to end up somewhere in main like CNodeState, or in CNode. Since (for once) this seems like something CConnman should know about but nobody else should have to care, I figured CNode was the likely place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I should mention: I've gone back and forth a few times on this (probably in this PR even). If anyone has a strong preference for a different approach that makes sense, I'm fine with changing it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isnt in CNode, though? If it were in CNode I'd be OK with that, but putting it in CConnman seems like /the/ layer violation this PR intends to fix?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The result of this PR is that CConnman now receives only dumb bytes with no knowledge of what they are. All internal structures could change without affecting the net side.
Dealing with wrapping the raw bytes up for transmission is something different, and arguably not something that the bitcoin side should have to know/care about. If you have a different model in your mind, please suggest it.
CNode is slowly moving towards an internal dumb storage class. So in this case, I don't make much distinction.
@@ -2581,30 +2581,17 @@ void CNode::AskFor(const CInv& inv) | |||
mapAskFor.insert(std::make_pair(nRequestTime, inv)); | |||
} | |||
|
|||
CDataStream CConnman::BeginMessage(CNode* pnode, int nVersion, int flags, const std::string& sCommand) | |||
void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be CSerializedNetMsg&&? It seems super strange that we're secretly destroying the object passed in, but only because the CSerializedNetMsg class doesnt have a copy-constructor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing secret is being done. This would fail, for example:
CSerializedNetMsg x = ...;
pconman->PushMessage(pnode, x);
as the binding of the actual parameter x would need a copy constructor to assign to the formal parameter msg.
Passing a && would be preferable, though. The move construction here is slower, and only adds the possibility of passing in something that is implicitly convertible to a CSerializedNetMsg (which afaik doesn't exist).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was done this way originally so that a message could be serialized once and sent to multiple peers. If an rvalue is required, we can't do that.
I don't think there's a use-case for that at the moment, though, so changing to && is fine by me.
{ | ||
CSerializedNetMsg msg; | ||
msg.data.assign(CSerializedNetMsg::header_pad_size, 0); | ||
msg.command = std::move(sCommand); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Over-use of std::move, much? Can we stick with const std::string& in the function declaration and then use operator=() to make the copy instead of creating a new object just to std::move the data over?
(Also, could definitely be missing some part of rvalue stuff, here)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is standard c++11 pass-by-value-by-default. If you pass in a const-reference, a copy is guaranteed since we're storing the result. If you pass by value, it can be moved and the copy can be skipped.
Not sure why you'd prefer a copy over 2 cheap moves.
Arguably NetMsgType should become a
std::array<char, CMessageHeader::COMMAND_SIZE>
so we can do away with the char arrays/strings, but that's a change for another day.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, didnt realize it would avoid a copy there (though I'd be OK copying the sCommand field...), anyway, comment withdrawn.
Rebased and updated for comments from @sipa and @TheBlueMatt.
|
Hmm, I would consider CNode and CConnman to be the same layer (after all
its processing related stated is moved out, and the rest is made private to
the net module).
|
Heh, well if you consider CNode to be a part of CConnman, then the type-of-message-to-send logic should be in net_processing...either way CConnman shouldnt know anything about the serialization of messages on the wire. |
Needs rebase |
Code review ACK 02d463d |
updated after a discussion with @TheBlueMatt and @sipa today, I think we're all finally aligned. @laanwj Sadly removing the "skb" was part of the compromise as it was a slight layer violation. We instead use 2 send()s (later scatter/gather) to avoid the copy. |
Needs rebase, sorry! |
This way we're not relying on messages going out after fDisconnect has been set. This should not cause any real behavioral changes, though feelers should arguably disconnect earlier in the process. That can be addressed in a later functional change.
…onnect Also, send reject messages earlier in SendMessages(), so that disconnections are processed earlier. These changes combined should ensure that no message is ever sent after fDisconnect is set.
Rebased after ca8549d. |
utACK 09a92c27d46a62d10aed95b742e891e204dcee4e |
ACK 09a92c2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You missed one fDisconnect check in SendMessages (the last one - feefilter, maybe a rebase issue?).
* @param[in] nVersionIn Serialization Version (including any flags) | ||
* @param[in] vchDataIn Referenced byte vector to overwrite/append | ||
* @param[in] nPosIn Starting position. Vector index where writes should start. If the index is past the end, the | ||
* vector will first be resized to fit. So to append, use vec.size(). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is very confusing...if nPos is 1 and vchData.size() is 1, the documentation implies that vchData will be resized so that nPos isnt past-the-end, but it will not be. Instead, vchData is resized so that, at maximum, nPos is 1-past-the-end.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, yes. Will clarify.
} | ||
size_t nMessageSize = msg.data.size(); | ||
size_t nTotalSize = nMessageSize + CMessageHeader::HEADER_SIZE; | ||
LogPrint("net", "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.command.c_str()), nTotalSize, pnode->id); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You changed the print - should print nMessageSize, not nTotalSize.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call. I assumed it'd never say "sending 0 bytes", but you're right, I'll avoid a behavioral change here.
CVectorWriter is useful for overwriting or appending an existing byte vector. CNetMsgMaker is a shortcut for creating messages on-the-fly which are suitable for pushing to CConnman.
This fixes one of the last major layer violations in the networking stack. The network side is no longer in charge of message serialization, so it is now decoupled from Bitcoin structures. Only the header is serialized and attached to the payload.
Updated and squashed for @TheBlueMatt's comments. The diff from before is very straightforward: cory@cory-i7:~/dev/bitcoin/src(connman-send)$ git diff theuni/connman-send
diff --git a/src/main.cpp b/src/main.cpp
index 4520afb..5595381 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -6983,7 +6983,7 @@ bool SendMessages(CNode* pto, CConnman& connman)
// Message: feefilter
//
// We don't want white listed peers to filter txs to us if we have -whitelistforcerelay
- if (!pto->fDisconnect && pto->nVersion >= FEEFILTER_VERSION && GetBoolArg("-feefilter", DEFAULT_FEEFILTER) &&
+ if (pto->nVersion >= FEEFILTER_VERSION && GetBoolArg("-feefilter", DEFAULT_FEEFILTER) &&
!(pto->fWhitelisted && GetBoolArg("-whitelistforcerelay", DEFAULT_WHITELISTFORCERELAY))) {
CAmount currentFilter = mempool.GetMinFee(GetArg("-maxmempool", DEFAULT_MAX_MEMPOOL_SIZE) * 1000000).GetFeePerK();
int64_t timeNow = GetTimeMicros();
diff --git a/src/net.cpp b/src/net.cpp
index e05c174..ce87150 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -2616,7 +2616,7 @@ void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg&& msg)
{
size_t nMessageSize = msg.data.size();
size_t nTotalSize = nMessageSize + CMessageHeader::HEADER_SIZE;
- LogPrint("net", "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.command.c_str()), nTotalSize, pnode->id);
+ LogPrint("net", "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.command.c_str()), nMessageSize, pnode->id);
std::vector<unsigned char> serializedHeader;
serializedHeader.reserve(CMessageHeader::HEADER_SIZE);
diff --git a/src/streams.h b/src/streams.h
index ef22620..b508784 100644
--- a/src/streams.h
+++ b/src/streams.h
@@ -81,8 +81,8 @@ class CVectorWriter
* @param[in] nTypeIn Serialization Type
* @param[in] nVersionIn Serialization Version (including any flags)
* @param[in] vchDataIn Referenced byte vector to overwrite/append
- * @param[in] nPosIn Starting position. Vector index where writes should start. If the index is past the end, the
- * vector will first be resized to fit. So to append, use vec.size().
+ * @param[in] nPosIn Starting position. Vector index where writes should start. The vector will initially
+ * grow as necessary to max(index, vec.size()). So to append, use vec.size().
*/
CVectorWriter(int nTypeIn, int nVersionIn, std::vector<unsigned char>& vchDataIn, size_t nPosIn) : nType(nTypeIn), nVersion(nVersionIn), vchData(vchDataIn), nPos(nPosIn)
{ |
utACK c7be56d |
c7be56d net: push only raw data into CConnman (Cory Fields) 2ec935d net: add CVectorWriter and CNetMsgMaker (Cory Fields) b7695c2 net: No need to check individually for disconnection anymore (Cory Fields) fedea8a net: don't send any messages before handshake or after requested disconnect (Cory Fields) d74e352 net: Set feelers to disconnect at the end of the version message (Cory Fields)
c7be56d net: push only raw data into CConnman (Cory Fields) 2ec935d net: add CVectorWriter and CNetMsgMaker (Cory Fields) b7695c2 net: No need to check individually for disconnection anymore (Cory Fields) fedea8a net: don't send any messages before handshake or after requested disconnect (Cory Fields) d74e352 net: Set feelers to disconnect at the end of the version message (Cory Fields)
c7be56d net: push only raw data into CConnman (Cory Fields) 2ec935d net: add CVectorWriter and CNetMsgMaker (Cory Fields) b7695c2 net: No need to check individually for disconnection anymore (Cory Fields) fedea8a net: don't send any messages before handshake or after requested disconnect (Cory Fields) d74e352 net: Set feelers to disconnect at the end of the version message (Cory Fields)
c7be56d net: push only raw data into CConnman (Cory Fields) 2ec935d net: add CVectorWriter and CNetMsgMaker (Cory Fields) b7695c2 net: No need to check individually for disconnection anymore (Cory Fields) fedea8a net: don't send any messages before handshake or after requested disconnect (Cory Fields) d74e352 net: Set feelers to disconnect at the end of the version message (Cory Fields)
30d5c66 net: correct addrman logging (Fuzzbawls) 8a2b7fe Don't send layer2 messages to peers that haven't completed the handshake (Fuzzbawls) dc10100 [bugfix] Making tier two thread interruptable. (furszy) 2ae76aa Move CNode::addrLocal access behind locked accessors (Fuzzbawls) 470482f Move CNode::addrName accesses behind locked accessors (Fuzzbawls) 35365e1 Move [clean|str]SubVer writes/copyStats into a lock (Fuzzbawls) d816a86 Make nServices atomic (Matt Corallo) 8a66add Make nStartingHeight atomic (Matt Corallo) 567c9b5 Avoid copying CNodeStats to make helgrind OK with buggy std::string (Matt Corallo) aea5211 Make nTimeConnected const in CNode (Matt Corallo) cf46680 net: fix a few races. (Fuzzbawls) c916fcf net: add a lock around hSocket (Cory Fields) cc8a93c net: rearrange so that socket accesses can be grouped together (Cory Fields) 6f731dc Do not add to vNodes until fOneShot/fFeeler/fAddNode have been set (Matt Corallo) 07c8d33 Ensure cs_vNodes is held when using the return value from FindNode (Matt Corallo) 110a44b Delete some unused (and broken) functions in CConnman (Matt Corallo) 08a12e0 net: log an error rather than asserting if send version is misused (Cory Fields) cd8b82c net: Disallow sending messages until the version handshake is complete (Fuzzbawls) 54b454b net: don't run callbacks on nodes that haven't completed the version handshake (Cory Fields) 2be6877 net: deserialize the entire version message locally (Fuzzbawls) 444f599 Dont deserialize nVersion into CNode (Fuzzbawls) f30f10e net: remove cs_vRecvMsg (Fuzzbawls) 5812f9e net: add a flag to indicate when a node's send buffer is full (Fuzzbawls) 5ec4db2 net: Hardcode protocol sizes and use fixed-size types (Wladimir J. van der Laan) de87ea6 net: Consistent checksum handling (Wladimir J. van der Laan) d4bcd25 net: push only raw data into CConnman (Cory Fields) b79e416 net: add CVectorWriter and CNetMsgMaker (Cory Fields) 63c51d3 net: No need to check individually for disconnection anymore (Cory Fields) 07d8c7b net: don't send any messages before handshake or after fdisconnect (Cory Fields) 9adfc7f net: Set feelers to disconnect at the end of the version message (Cory Fields) f88c06c net: handle version push in InitializeNode (Cory Fields) 04d39c8 net: construct CNodeStates in place (Cory Fields) 40a6c5d net: remove now-unused ssSend and Fuzz (Cory Fields) 681c62d drop the optimistic write counter hack (Cory Fields) 9f939f3 net: switch all callers to connman for pushing messages (Cory Fields) 8f9011d connman is in charge of pushing messages (Cory Fields) f558bb7 serialization: teach serializers variadics (Cory Fields) 01ea667 net: Use deterministic randomness for CNode's nonce, and make it const (Cory Fields) de1ad13 net: constify a few CNode vars to indicate that they're threadsafe (Cory Fields) 34050a3 Move static global randomizer seeds into CConnman (Pieter Wuille) 1ce349f net: add a flag to indicate when a node's process queue is full (Fuzzbawls) 5581b47 net: add a new message queue for the message processor (Fuzzbawls) 701b578 net: rework the way that the messagehandler sleeps (Fuzzbawls) 7e55dbf net: Add a simple function for waking the message handler (Cory Fields) 47ea844 net: record bytes written before notifying the message processor (Cory Fields) ffd4859 net: handle message accounting in ReceiveMsgBytes (Cory Fields) 8cee696 net: log bytes recv/sent per command (Fuzzbawls) 754400e net: set message deserialization version when it's time to deserialize (Fuzzbawls) d2b8e0a net: make CMessageHeader a dumb storage class (Fuzzbawls) cc24eff net: remove redundant max sendbuffer size check (Fuzzbawls) 32ab0c0 net: wait until the node is destroyed to delete its recv buffer (Cory Fields) 6e3f71b net: only disconnect if fDisconnect has been set (Cory Fields) 1b0beb6 net: make GetReceiveFloodSize public (Cory Fields) 229697a net: make vRecvMsg a list so that we can use splice() (Fuzzbawls) d2d71ba net: fix typo causing the wrong receive buffer size (Cory Fields) 50bb09d Add test-before-evict discipline to addrman (Ethan Heilman) Pull request description: This is a combination of multiple upstream PRs focused on optimizing the P2P networking flow after the introduction of CConnman encapsulation, and a few older PRs that were previously missed to support the later optimizations. The PRs are as follows: - bitcoin#9037 - net: Add test-before-evict discipline to addrman - bitcoin#5151 - make CMessageHeader a dumb storage class - bitcoin#6589 - log bytes recv/sent per command - bitcoin#8688 - Move static global randomizer seeds into CConnman - bitcoin#9050 - net: make a few values immutable, and use deterministic randomness for the localnonce - bitcoin#8708 - net: have CConnman handle message sending - bitcoin#9128 - net: Decouple CConnman and message serialization - bitcoin#8822 - net: Consistent checksum handling - bitcoin#9441 - Net: Massive speedup. Net locks overhaul - bitcoin#9609 - net: fix remaining net assertions - bitcoin#9626 - Clean up a few CConnman cs_vNodes/CNode things - bitcoin#9698 - net: fix socket close race - bitcoin#9708 - Clean up all known races/platform-specific UB at the time PR was opened - Excluded bitcoin/bitcoin@512731b and bitcoin/bitcoin@d8f2b8a, to be done in a separate PR ACKs for top commit: furszy: code ACK 30d5c66 , testnet sync from scratch went well and tested with #1829 on top as well and all good. furszy: mainnet sync went fine, ACK 30d5c66 . random-zebra: ACK 30d5c66 and merging... Tree-SHA512: 09689554f53115a45f810b47ff75d887fa9097ea05992a638dbb6055262aeecd82d6ce5aaa2284003399d839b6f2c36f897413da96cfa2cd3b858387c3f752c1
This contains a more complete version of #9117, as well as the changes promised to @TheBlueMatt in #8708. They're combined here because the PushMessage changes require that the SendVersion is sane. This will be my last time messing with PushMessage, and it's nearing the end of the net changes needed outside of actual net code...
The changes here: