p2p: Fill reconciliation sets and request reconciliation (Erlay) #26283

naumenkogs · 2022-10-08T07:29:52Z

First, this PR enables keeping track of per-peer reconciliation sets, containing those transactions which we intend to exchange efficiently. The remaining transactions are announced via flooding, as usual.

Second, this PR enables periodically initiating a reconciliation round via a new p2p message.

Erlay Project Tracking: #28646

naumenkogs · 2022-10-08T07:33:48Z

~~Marking this draft until~~

~~Previous PR is merged~~ Follow-up is merged
~~There is one in-code TODO I have to resolve (it is minor, but it has to be improved)~~
~~I add unit and functional tests for these features.~~

Another task is to sync parent PR with this version. I have a local branch that compiles, but tests of the full Erlay there needs a little care. If you want to see how this PR works in the broader context, the parent should be good enough to get it. Otherwise, I intend to update it soon.

DrahtBot · 2022-10-08T16:15:15Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#28463 (p2p: Increase inbound capacity for block-relay only connections by mzumsande)
#21515 (Erlay: bandwidth-efficient transaction relay protocol by naumenkogs)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

ariard

Just few comments as a first parse on the post-#23443 commits.

src/net_processing.cpp

src/node/txreconciliation.h

src/net_processing.cpp

src/node/txreconciliation.h

src/net_processing.cpp

src/node/txreconciliation.cpp

aureleoules

First code-review parse, left some minor comments.

src/net_processing.cpp

src/node/txreconciliation.cpp

src/protocol.h

naumenkogs · 2023-10-13T11:19:23Z

Still have to review a couple of comments, especially the fuzzing suggested by @dergoegge

The pre-mutexed version is useful for external calls, while the regular version will be used internally.

They will be used later on.

naumenkogs · 2023-10-18T16:18:36Z

This PR is good for review. Fuzzing will come in a separate PR.

ariard

will review again pr from scratch

src/node/txreconciliation.cpp

ariard · 2023-10-24T01:50:53Z

src/net_processing.cpp

    const Consensus::Params& consensusParams = m_chainparams.GetConsensus();

+    const auto current_time{GetTime<std::chrono::microseconds>()};
+
+    // We must look into the reconciliation queue early. Since the queue applies to all peers,


note the call to IsPeerNextToReconcileWith is before MaybeDiscourageAndDisconnect. if a peer is misbehaving and candidate for disconnection, maybe we should disconnect first and let the reconciliation slot be used by another peer

This would make a difference only once, which is fine for (lower-risk) outbound peers I think.

The problem with latter IsPeerNextToReconcileWith is that the queue won't be queried. Say, a peer finds a way to exploit MaybeDiscourageAndDisconnect (say in combination with something else) — so that it always returns, but never disconnects. In that case the peer will be stuck at the front of the queue forever.

I don't think what i described is currently possible (fDisconnect only can go from false to true), but I'm afraid it might be in the future.

src/net_processing.cpp

test/functional/p2p_reqtxrcncl.py

Transactions eligible for reconciliation are added to the reconciliation sets. For the remaining txs, low-fanout is used.

For a subset of reconciling peers we announce transactions via low fanout. We need to set lower intervals for that to achieve lower relay latency. Note that for privacy reasons the ratio between inbound and outbound delays matter more than the actual delays.

When we're finalizing negotiation, we should add the peers for which we will initiate reconciliations to the queue.

maflcko

left a test nit, feel free to ignore

test/functional/test_framework/messages.py

src/net_processing.cpp

When the time comes for the peer, we send a reconciliation request with the parameters which will help the peer to construct a (hopefully) sufficient reconciliation sketch for us. We will then use that sketch to find missing transactions.

naumenkogs · 2023-11-01T09:07:31Z

I decided to split this off into #28765.

naumenkogs marked this pull request as draft October 8, 2022 07:30

DrahtBot added the P2P label Oct 8, 2022

DrahtBot added the Needs rebase label Oct 13, 2022

ariard reviewed Oct 18, 2022

View reviewed changes

naumenkogs mentioned this pull request Oct 21, 2022

Erlay: bandwidth-efficient transaction relay protocol #21515

Closed

aureleoules reviewed Oct 26, 2022

View reviewed changes

naumenkogs force-pushed the 2022-10-erlay2 branch 3 times, most recently from 4afeac2 to cacf08d Compare November 4, 2022 13:21

DrahtBot added Needs rebase and removed Needs rebase labels Nov 4, 2022

naumenkogs force-pushed the 2022-10-erlay2 branch from cacf08d to 441a4f3 Compare November 8, 2022 13:16

DrahtBot removed the Needs rebase label Nov 8, 2022

DrahtBot mentioned this pull request Nov 9, 2022

p2p: Erlay support signaling follow-ups #26359

Merged

naumenkogs force-pushed the 2022-10-erlay2 branch 3 times, most recently from b0c1c43 to 913c2cb Compare November 10, 2022 12:14

aureleoules reviewed Nov 11, 2022

View reviewed changes

src/node/txreconciliation.cpp Outdated Show resolved Hide resolved

DrahtBot mentioned this pull request Sep 12, 2023

p2p: Increase inbound capacity for block-relay only connections #28463

Open

ariard reviewed Sep 21, 2023

View reviewed changes

src/protocol.h Outdated Show resolved Hide resolved

naumenkogs mentioned this pull request Oct 13, 2023

Erlay Project Tracking #28646

Closed

17 tasks

naumenkogs force-pushed the 2022-10-erlay2 branch from 180052c to 80dc9c4 Compare October 13, 2023 11:17

DrahtBot added the CI failed label Oct 13, 2023

naumenkogs added 2 commits October 17, 2023 13:56

refactor: Add a pre-mutexed version of IsPeerRegistered

fa01342

The pre-mutexed version is useful for external calls, while the regular version will be used internally.

p2p: Functions to add/remove wtxids to tx reconciliation sets

8c658b9

They will be used later on.

naumenkogs force-pushed the 2022-10-erlay2 branch 3 times, most recently from b509e9a to 863a334 Compare October 18, 2023 08:09

ariard reviewed Oct 19, 2023

View reviewed changes

src/node/txreconciliation.cpp Outdated Show resolved Hide resolved

ariard reviewed Oct 24, 2023

View reviewed changes

brunoerg reviewed Oct 24, 2023

View reviewed changes

src/net_processing.cpp Outdated Show resolved Hide resolved

brunoerg reviewed Oct 25, 2023

View reviewed changes

test/functional/p2p_reqtxrcncl.py Outdated Show resolved Hide resolved

naumenkogs added 3 commits October 27, 2023 11:53

p2p: Add transactions to reconciliation sets

b07029a

Transactions eligible for reconciliation are added to the reconciliation sets. For the remaining txs, low-fanout is used.

p2p: Add peers to reconciliation queue on negotiation

3252f54

When we're finalizing negotiation, we should add the peers for which we will initiate reconciliations to the queue.

naumenkogs force-pushed the 2022-10-erlay2 branch from 863a334 to 3637986 Compare October 27, 2023 08:59

DrahtBot removed the CI failed label Oct 27, 2023

maflcko reviewed Oct 27, 2023

View reviewed changes

test/functional/test_framework/messages.py Outdated Show resolved Hide resolved

test/functional/test_framework/messages.py Outdated Show resolved Hide resolved

brunoerg reviewed Oct 27, 2023

View reviewed changes

src/net_processing.cpp Outdated Show resolved Hide resolved

naumenkogs added 2 commits October 30, 2023 10:01

p2p: Initiate reconciliation round

c93bd55

When the time comes for the peer, we send a reconciliation request with the parameters which will help the peer to construct a (hopefully) sufficient reconciliation sketch for us. We will then use that sketch to find missing transactions.

test: functional test for reqtxrcncl

b518d26

naumenkogs force-pushed the 2022-10-erlay2 branch from 3637986 to b518d26 Compare October 30, 2023 08:13

naumenkogs closed this Nov 1, 2023

sr-gi mentioned this pull request Jun 7, 2024

Erlay Project Tracking #30249

Open

17 tasks

bitcoin locked and limited conversation to collaborators Oct 31, 2024

p2p: Fill reconciliation sets and request reconciliation (Erlay) #26283

p2p: Fill reconciliation sets and request reconciliation (Erlay) #26283

Uh oh!

Conversation

naumenkogs commented Oct 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

naumenkogs commented Oct 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Oct 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage

Reviews

Conflicts

Uh oh!

ariard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aureleoules left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

naumenkogs commented Oct 13, 2023

Uh oh!

naumenkogs commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ariard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ariard Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

naumenkogs Oct 27, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maflcko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

naumenkogs commented Nov 1, 2023

Uh oh!

Uh oh!

naumenkogs commented Oct 8, 2022 •

edited

Loading

naumenkogs commented Oct 8, 2022 •

edited

Loading

DrahtBot commented Oct 8, 2022 •

edited

Loading

naumenkogs commented Oct 18, 2023 •

edited

Loading