zmq: mempool notifications #8549

jmcorgan · 2016-08-19T17:07:54Z

This is a current rebased version of #7753 by @laanwj. Please see that PR for original discussion and existing ACKs.

Add notifications when transactions enter or leave the mempool.

These can be enabled with zmqpubmempool:

mempooladded: a transaction was added to the mempool
mempoolremoved: a transaction was removed from the mempool

[...]

It also makes it possible to keep statistics on why transactions disappear from the mempool.

Add notification signals to make it possible to subscribe to mempool changes: - NotifyEntryAdded(const CTxMemPoolEntry &)> - NotifyEntryRemoved(const CTxMemPoolEntry &, MemPoolRemovalReason)> Also add a mempool removal reason enumeration, which is passed to the removed notification based on why the transaction was removed from the mempool.

Replace factories list with calls to a function. This simplifies the code (every notifier is only created once anyway), and makes it possible to pass arguments to notifier contructors.

Add notifications when transactions enter or leave the mempool. These can be enabled with `pubmempool`: - `mempooladded`: a transaction was added to the mempool - `mempoolremoved`: a transaction was removed from the mempool This allows third-party software to keep track of what is in the mempool in real time.

Make the registration process less circuitous by making the notifiers listen to their appropriate events themselves

kanzure · 2016-08-19T17:56:44Z

src/main.cpp

@@ -2752,7 +2752,7 @@ bool static DisconnectTip(CValidationState& state, const CChainParams& chainpara
            list<CTransaction> removed;
            CValidationState stateDummy;
            if (tx.IsCoinBase() || !AcceptToMemoryPool(mempool, stateDummy, tx, false, NULL, true)) {
-                mempool.removeRecursive(tx, removed);
+              mempool.removeRecursive(tx, removed, MemPoolRemovalReason::REORG);


trivial nit: indentation symmetry between mempool.removeRecursive line and vHashUpdate.push_back line.

Emacs C++ mode FTW. Will cleanup.

jmcorgan · 2016-08-19T18:12:37Z

Hmm, looks like zmq_test.py needs some attention.

dcousens · 2016-08-24T02:49:10Z

a transaction was added to the mempool

OOI, is the case if a re-org occurs and the transaction only existed in the pre re-org chain?

jonasschnelli · 2016-10-26T07:41:46Z

Needs rebased. Thanks.

jonasschnelli · 2016-10-26T08:08:35Z

Conceptual issue:
Shouldn't there be a special ZMQ message when bitcoind starts up?
Because of the nature of the "self-healing" ZMQ connections, listeners should probabyl be informed if bitcoind restarts (which currently result in sweeping the mempool).
Maybe it could be a specific "clearmempool" message during startup.

bitkevin · 2016-10-26T08:35:47Z

Nice work!

My suggestion:

when start or restart bitcoind there should be a special event to tell consumers which listen to the ZMQ. (same as @jonasschnelli)
Add rawhex in the message. The consumers need to decode hex rawtx after get the ZMQ message, otherwise they need to do another RPC call 'getrawtransaction'.
When remove txs, they should be in order: remove children first and then the parents. ZMQ messages should also in the particular order. (Maybe the PR has already guarantee of it, I am not sure)

I patched bitcoind to the same thing: https://github.com/btccom/explorer-kv/blob/master/deploy/docker/bitcoind/v0.12.1/bitcoind-explorer-v0.12.1.patch

When bitcoind start, also need to push the latest block in ZMQ, so consumers could know if they need to sync to the latest block(with rpc call to get blocks), after that could consume txs logs from ZMQ.

dcousens · 2016-12-19T07:45:47Z

concept ACK

jtimon · 2016-12-20T01:25:03Z

Needs rebase

fanquake · 2017-01-21T00:49:40Z

@jmcorgan Are you planning on continuing this work? Needs a rebase, and nits addressed.

dcousens · 2017-01-24T11:24:32Z

Close now?

laanwj · 2017-01-24T12:21:19Z

Closing (for now). Might pick this up later.

karelbilek · 2017-03-17T04:58:01Z

This would actually be very useful, now with all the full blocks and transactions falling out of mempools more often.

Any plans for continuing with this? :) thx

edit: oh, there is also this PR, already merged. Not sure what it does 100%

#9371

jmcorgan · 2017-03-17T06:48:53Z

My own branch was only an attempt at keeping this current with progress on master; the real work was done by @laanwj. I'll defer to him whether or not he's willing to continue from here.

laanwj · 2017-03-17T09:11:02Z

@jmcorgan @Runn1ng It's on my long list. Sorry for occasionally starting projects then seemingly abandoning them. It's just that it's not always clear how much interest there is in things, and keeping things rebased while it's unclear how much attention something will get sometimes feels like a waste of time.

But it's clear that there is user interest in this and some of the changes here have been part of #9371, just not the zmq part. I'll get to that... But if someone else feels like picking this up to accelerate things (hopefully) go ahead.

Sjors · 2017-11-29T16:49:31Z

Knowing when a transaction is added to the mempool might be useful in the test framework as well, e.g. so sync_mempools can be implemented asynchronously instead of through polling (cc @jnewbery).

maflcko · 2021-11-29T09:51:18Z

picked up (sort of) in #23624

laanwj added 5 commits August 18, 2016 18:22

zmq: refactor initialization

bd7df61

Replace factories list with calls to a function. This simplifies the code (every notifier is only created once anyway), and makes it possible to pass arguments to notifier contructors.

doc: Add documentation for zmq mempool notifications

7882b9b

zmq: refactor to make all notifiers register themselves

11d20c9

Make the registration process less circuitous by making the notifiers listen to their appropriate events themselves

jmcorgan mentioned this pull request Aug 19, 2016

zmq: mempool notifications #7753

Closed

kanzure reviewed Aug 19, 2016
View reviewed changes

fanquake added the RPC/REST/ZMQ label Aug 31, 2016

laanwj mentioned this pull request Dec 19, 2016

Notify on removal #9371

Merged

laanwj closed this Jan 24, 2017

laanwj added the Up for grabs label Nov 28, 2017

promag mentioned this pull request Dec 31, 2017

expose fees via ZeroMQ for transactions in the mempool #11848

Closed

jmcorgan deleted the zmq_mempool branch August 29, 2020 14:25

maflcko removed the Up for grabs label Nov 29, 2021

bitcoin locked and limited conversation to collaborators Nov 29, 2022

zmq: mempool notifications #8549

zmq: mempool notifications #8549

Uh oh!

Conversation

jmcorgan commented Aug 19, 2016

Uh oh!

kanzure Aug 19, 2016

Choose a reason for hiding this comment

Uh oh!

jmcorgan Aug 19, 2016

Choose a reason for hiding this comment

Uh oh!

jmcorgan commented Aug 19, 2016

Uh oh!

dcousens commented Aug 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented Oct 26, 2016

Uh oh!

jonasschnelli commented Oct 26, 2016

Uh oh!

bitkevin commented Oct 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcousens commented Dec 19, 2016

Uh oh!

jtimon commented Dec 20, 2016

Uh oh!

fanquake commented Jan 21, 2017

Uh oh!

dcousens commented Jan 24, 2017

Uh oh!

laanwj commented Jan 24, 2017

Uh oh!

karelbilek commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmcorgan commented Mar 17, 2017

Uh oh!

laanwj commented Mar 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sjors commented Nov 29, 2017

Uh oh!

maflcko commented Nov 29, 2021

Uh oh!

Uh oh!

dcousens commented Aug 24, 2016 •

edited

Loading

bitkevin commented Oct 26, 2016 •

edited

Loading

karelbilek commented Mar 17, 2017 •

edited

Loading

laanwj commented Mar 17, 2017 •

edited

Loading