rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' #29418

vasild · 2024-02-11T09:32:51Z

Introduce a new RPC, getnetmsgstats to retrieve traffic bytes and count of messages possibly broken down by:

direction (sent or received)
network (ipv4, tor, etc)
connection type (outbound-full-relay, block-relay-only, etc)
message type (verack, ping, etc)

Implements: #26337 Add per message stats to getnettotals rpc

Examples:

bitcoin-cli getnetmsgstats '["direction", "network", "connection_type", "message_type"]'

{
  "bytes": 5392756,
  "count": 32683
}

bitcoin-cli getnetmsgstats '["direction", "network", "connection_type"]'

{
  "sendcmpct": {
    "bytes": 975,
    "count": 31
  },
  "sendaddrv2": {
    "bytes": 816,
    "count": 28
  },
  "headers": {
    "bytes": 1786,
    "count": 28
  },
  "feefilter": {
    "bytes": 816,
    "count": 27
  },
  "version": {
    "bytes": 4843,
    "count": 37
  },
  "ping": {
    "bytes": 1104,
    "count": 36
  },
  "getaddr": {
    "bytes": 384,
    "count": 13
  },
  "block": {
    "bytes": 5128467,
    "count": 20365
  },
  "getheaders": {
    "bytes": 29436,
    "count": 28
  },
  "pong": {
    "bytes": 1040,
    "count": 34
  },
  "getdata": {
    "bytes": 1218925,
    "count": 20008
  },
  "sendheaders": {
    "bytes": 447,
    "count": 16
  },
  "wtxidrelay": {
    "bytes": 816,
    "count": 28
  },
  "addrv2": {
    "bytes": 301161,
    "count": 25
  },
  "verack": {
    "bytes": 816,
    "count": 28
  }
}

bitcoin-cli getnetmsgstats '["direction", "connection_type", "message_type"]'

{
  "i2p": {
    "bytes": 1311861,
    "count": 8226
  },
  "onion": {
    "bytes": 3932760,
    "count": 24272
  },
  "ipv6": {
    "bytes": 631265,
    "count": 3362
  },
  "ipv4": {
    "bytes": 1300106,
    "count": 7773
  }
}

bitcoin-cli getnetmsgstats

{
  "recv": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 309,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 408,
          "count": 3
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "block": {
          "bytes": 432950,
          "count": 1794
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "addrv2": {
          "bytes": 68508,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 126,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "block": {
          "bytes": 478479,
          "count": 1896
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 195,
          "count": 6
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 527,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 639,
          "count": 5
        },
        "ping": {
          "bytes": 189,
          "count": 6
        },
        "block": {
          "bytes": 1465688,
          "count": 5957
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 120693,
          "count": 9
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 412,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "block": {
          "bytes": 138074,
          "count": 583
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "addrv2": {
          "bytes": 93436,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 126,
          "count": 1
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 252,
          "count": 2
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "block": {
          "bytes": 313988,
          "count": 1255
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "addrv2": {
          "bytes": 40,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  },
  "sent": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 66,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "getaddr": {
          "bytes": 99,
          "count": 3
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "getdata": {
          "bytes": 105382,
          "count": 1789
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "version": {
          "bytes": 127,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "getdata": {
          "bytes": 115932,
          "count": 1884
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 162,
          "count": 5
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 122,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 771,
          "count": 6
        },
        "ping": {
          "bytes": 221,
          "count": 7
        },
        "getaddr": {
          "bytes": 129,
          "count": 5
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "getdata": {
          "bytes": 363478,
          "count": 5932
        },
        "sendheaders": {
          "bytes": 129,
          "count": 5
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 231,
          "count": 4
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 88,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "getaddr": {
          "bytes": 132,
          "count": 4
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "getdata": {
          "bytes": 35492,
          "count": 584
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 254,
          "count": 2
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 508,
          "count": 4
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "getaddr": {
          "bytes": 24,
          "count": 1
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "getdata": {
          "bytes": 76531,
          "count": 1231
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  }
}

Previous incarnations of this:
#27534 rpc: add 'getnetmsgstats', new rpc to view network message statistics (Thank you, @satsie!)
#28926 rpc: add 'getnetmsgstats' RPC (Thank you, @willcl-ark!)

DrahtBot · 2024-02-11T09:32:54Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/29418.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	epiccurious

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#32822 (fuzz: Make process_message(s) more deterministic by maflcko)
#30988 (Split CConnman by vasild)
#30951 (net: option to disallow v1 connection on ipv4 and ipv6 peers by stratospher)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

typo -> established [Typo]

DrahtBot · 2024-02-11T10:51:04Z

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/21448984768}

epiccurious · 2024-02-11T22:26:43Z

Concept ACK 91c90d7.

DrahtBot · 2024-04-04T14:24:37Z

Needs rebase if still relevant

0xB10C · 2024-04-09T16:20:24Z

just fyi as this came up: You could count up message stats with the inbound and outbound message tracepoints. However, this does only gives you stats from the points on where you start hooking into the tracepoints, might not be as user friendly (RPC is easier to use than the tracepoints), and the tracepoints currently only work on Linux.

While I personally don't need it, I see why some node operators need traffic statistics. E.g. How much of my (limited monthly) traffic is used up by inbound connections?

DrahtBot · 2024-04-23T06:28:45Z

Converted to draft due to failing CI and inactivity

vasild · 2024-05-09T13:40:37Z

91c90d7595...6bfda84b92: rebase, pick changes from #29421 and adjust the newly added test.

…e constant b3efb48 protocol: make message types constexpr (Vasil Dimov) 2fa9de0 net: make the list of known message types a compile time constant (Vasil Dimov) Pull request description: Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size. --- This change is part of #29418 but it makes sense on its own and would be good to have it, regardless of the fate of #29418. Also, if this is merged, that would reduce the size of #29418, thus the current standalone PR. ACKs for top commit: achow101: ACK b3efb48 jonatack: ACK b3efb48 maflcko: utACK b3efb48 🎊 willcl-ark: ACK b3efb48 Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af

vasild · 2024-05-23T09:51:33Z

6bfda84b92...b5189f543c: rebase due to conflicts (part of this PR was merged via #29421)

vasild · 2025-01-23T10:46:49Z

f42561e667...58538c0847: rebase due to conflicts

DrahtBot · 2025-01-23T12:20:16Z

🚧 At least one of the CI tasks failed.
_{Debug: https://github.com/bitcoin/bitcoin/runs/36052867071}

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

vasild · 2025-01-23T12:43:13Z

58538c0847...a46cbdd478: pet the py linter

src/rpc/net.cpp

vasild · 2025-01-24T16:16:28Z

a46cbdd478...db442f3d6e: do #29418 (comment)

vasild · 2025-03-20T17:38:29Z

db442f3d6e...626cc06c69: rebase due to conflicts

src/rpc/net.cpp

Before this change only per-peer stats were gathered. They are lost when the peer disconnects. So, collect the traffic stats globally in `CConnman`, broken down by * direction (sent or received, (2)) * network of the peer (IPv4, IPv6, Tor, I2P, CJDNS (5)) * connection type (inbound, full outbound, feeler, etc, (6)) * message type (verack, ping, etc, (36))

Introduce a new RPC `getnetmsgstats` that renders the global traffic stats from `CConnman`. Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

Aggregation can be by either one of direction, network, connection type or message type. For example if the following ``` { "ipv4": { "ping": 3 }, "ipv6": { "ping": 4 } } ``` is aggregated by network, then the result will be ``` { "ping": 7 } ```

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

vasild · 2025-05-21T09:00:09Z

626cc06c69...528b4d6b9c: rebase and do #29418 (comment)

DrahtBot · 2025-06-25T08:32:40Z

test/functional/rpc_net.py

+        stats_before_connect = get_stats(node0)
+        node2 = node0.add_p2p_connection(P2PInterface())
+        assert_equal(len(node0.getpeerinfo()), 1)
+        # Wait for the initial PING (that is sent immediately after the connection is estabilshed) to go through.


typo -> established [Typo]

DrahtBot · 2025-07-25T10:34:40Z

🐙 This pull request conflicts with the target branch and needs rebase.

DrahtBot added the RPC/REST/ZMQ label Feb 11, 2024

vasild mentioned this pull request Feb 11, 2024

rpc: add 'getnetmsgstats' RPC #28926

Closed

DrahtBot added the CI failed label Feb 11, 2024

vasild force-pushed the getnetmsgstats branch from bc156d8 to 91c90d7 Compare February 11, 2024 12:45

DrahtBot removed the CI failed label Feb 11, 2024

vasild mentioned this pull request Feb 11, 2024

net: make the list of known message types a compile time constant #29421

Merged

This was referenced Feb 11, 2024

net, net_processing: additional and consistent disconnect logging #28521

Merged

Introduce 'getblockfileinfo' RPC command #27770

Closed

Prune locks #19463

Draft

This was referenced Feb 11, 2024

log, refactor: Allow log macros to accept context arguments #29256

Open

Stratum v2 Template Provider (take 3) #29432

Closed

DrahtBot added the CI failed label Mar 12, 2024

DrahtBot mentioned this pull request Mar 13, 2024

scripted-diff: Use LogInfo over LogPrintf [WIP, NOMERGE, DRAFT] #29641

Draft

achow101 requested a review from mzumsande April 9, 2024 15:52

DrahtBot marked this pull request as draft April 23, 2024 06:28

vasild force-pushed the getnetmsgstats branch from 91c90d7 to 6bfda84 Compare May 9, 2024 13:40

DrahtBot removed the CI failed label May 9, 2024

DrahtBot added the Needs rebase label May 21, 2024

vasild force-pushed the getnetmsgstats branch from 6bfda84 to b5189f5 Compare May 23, 2024 09:50

vasild marked this pull request as ready for review May 23, 2024 09:51

DrahtBot removed the Needs rebase label May 23, 2024

DrahtBot added CI failed and removed Needs rebase labels Jan 23, 2025

vasild force-pushed the getnetmsgstats branch from 58538c0 to a46cbdd Compare January 23, 2025 12:42

maflcko reviewed Jan 23, 2025

View reviewed changes

src/rpc/net.cpp Outdated Show resolved Hide resolved

DrahtBot removed the CI failed label Jan 23, 2025

DrahtBot mentioned this pull request Jan 23, 2025

refactor: Use std::span over Span #31519

Merged

vasild force-pushed the getnetmsgstats branch from a46cbdd to db442f3 Compare January 24, 2025 16:15

This was referenced Mar 10, 2025

[IBD] specialize block serialization #31868

Draft

Replace libevent with our own HTTP and socket-handling implementation #32061

Open

DrahtBot added the Needs rebase label Mar 20, 2025

vasild force-pushed the getnetmsgstats branch from db442f3 to 626cc06 Compare March 20, 2025 17:38

DrahtBot removed the Needs rebase label Mar 20, 2025

DrahtBot added the CI failed label May 18, 2025

maflcko reviewed May 20, 2025

View reviewed changes

src/rpc/net.cpp Outdated Show resolved Hide resolved

vasild and others added 4 commits May 21, 2025 10:56

rpc: visualize global CConnman stats in a new RPC getnetmsgstats

c005443

Introduce a new RPC `getnetmsgstats` that renders the global traffic stats from `CConnman`. Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

test: add functional tests for the new getnetmsgstats RPC

528b4d6

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

vasild force-pushed the getnetmsgstats branch from 626cc06 to 528b4d6 Compare May 21, 2025 08:59

DrahtBot removed the CI failed label May 21, 2025

DrahtBot reviewed Jun 25, 2025

View reviewed changes

DrahtBot mentioned this pull request Jun 27, 2025

fuzz: Make process_message(s) more deterministic #32822

Merged

DrahtBot added the Needs rebase label Jul 25, 2025

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' #29418

Are you sure you want to change the base?

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' #29418

Uh oh!

Conversation

vasild commented Feb 11, 2024

Uh oh!

DrahtBot commented Feb 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

LLM Linter (✨ experimental)

Uh oh!

DrahtBot commented Feb 11, 2024

Uh oh!

epiccurious commented Feb 11, 2024

Uh oh!

DrahtBot commented Apr 4, 2024

Uh oh!

0xB10C commented Apr 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Apr 23, 2024

Uh oh!

vasild commented May 9, 2024

Uh oh!

vasild commented May 23, 2024

Uh oh!

vasild commented Jan 23, 2025

Uh oh!

DrahtBot commented Jan 23, 2025

Uh oh!

vasild commented Jan 23, 2025

Uh oh!

Uh oh!

vasild commented Jan 24, 2025

Uh oh!

vasild commented Mar 20, 2025

Uh oh!

Uh oh!

vasild commented May 21, 2025

Uh oh!

DrahtBot Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

DrahtBot commented Jul 25, 2025

Uh oh!

Uh oh!

DrahtBot commented Feb 11, 2024 •

edited

Loading

0xB10C commented Apr 9, 2024 •

edited

Loading