…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d

This tracepoint was introduced in bitcoin#22006. The first argument was the
hash of the connected block as a pointer to a C-like String. The last
argument passed the hash of the connected block as pointer to 32
bytes. The hash was passed both as pointer to a string and to bytes
to allow `bpftrace` scripts to print the String. It was (incorrectly)
assumed that `bpftrace` is not able to hex-format and print the block
hash given only the hash bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash it can be printed in in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the connect_block tracepoint API, however
there exist no release including the tracepoint yet.

This tracepoint was introduced in bitcoin#22006. The first argument was the
hash of the connected block as a pointer to a C-like String. The last
argument passed the hash of the connected block as a pointer to 32
bytes. The hash was passed both as a pointer to a string and bytes
to allow `bpftrace` scripts to print the String. It was (incorrectly)
assumed that `bpftrace` cannot hex-format and print the block hash
given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the connect_block tracepoint API, however
there exist no release, including the tracepoint, yet.

The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was passed both as
a pointer to a string and bytes to allow `bpftrace` scripts to print
the String. It was (incorrectly) assumed that `bpftrace` cannot hex-
format and print the block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.

The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was only passed as
string to allow `bpftrace` scripts to print the hash. It was
(incorrectly) assumed that `bpftrace` cannot hex-format and print the
block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.

The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was only passed as
string to allow `bpftrace` scripts to print the hash. It was
(incorrectly) assumed that `bpftrace` cannot hex-format and print the
block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p -= 1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.

…alidation:block_connected` tracepoint

53c9fa9 tracing: drop block_connected hash.toString() arg (0xb10c)

Pull request description:

  The tracepoint `validation:block_connected` was introduced in #22006.
  The first argument was the hash of the connected block as a pointer
  to a C-like String. The last argument passed the hash of the
  connected block as a pointer to 32 bytes. The hash was only passed as
  string to allow `bpftrace` scripts to print the hash. It was
  (incorrectly) assumed that `bpftrace` cannot hex-format and print the
  block hash given only the hash as bytes.

  The block hash can be printed in `bpftrace` by calling
  `printf("%02x")` for each byte of the hash in an `unroll () {...}`.
  By starting from the last byte of the hash, it can be printed in
  big-endian (the block-explorer format).

  ```C
    $p = $hash + 31;
    unroll(32) {
        $b = *(uint8*)$p;
        printf("%02x", $b);
        $p -= 1;
    }
  ```

  See also: #22902 (comment)

  This is a breaking change to the block_connected tracepoint API, however
  this tracepoint has not yet been included in a release.

ACKs for top commit:
  laanwj:
    Concept and code review ACK 53c9fa9
  jb55:
    ACK 53c9fa9

Tree-SHA512: f1b9e4e0ee45aae892e8bf38e04b5ee5fbc643d6e7e27d011b829ed8701dacf966a99b7c877c46cca8666b894a375633e62582c552c8203614c6f2b9c4087585

… the `validation:block_connected` tracepoint

53c9fa9 tracing: drop block_connected hash.toString() arg (0xb10c)

Pull request description:

  The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
  The first argument was the hash of the connected block as a pointer
  to a C-like String. The last argument passed the hash of the
  connected block as a pointer to 32 bytes. The hash was only passed as
  string to allow `bpftrace` scripts to print the hash. It was
  (incorrectly) assumed that `bpftrace` cannot hex-format and print the
  block hash given only the hash as bytes.

  The block hash can be printed in `bpftrace` by calling
  `printf("%02x")` for each byte of the hash in an `unroll () {...}`.
  By starting from the last byte of the hash, it can be printed in
  big-endian (the block-explorer format).

  ```C
    $p = $hash + 31;
    unroll(32) {
        $b = *(uint8*)$p;
        printf("%02x", $b);
        $p -= 1;
    }
  ```

  See also: bitcoin#22902 (comment)

  This is a breaking change to the block_connected tracepoint API, however
  this tracepoint has not yet been included in a release.

ACKs for top commit:
  laanwj:
    Concept and code review ACK 53c9fa9
  jb55:
    ACK 53c9fa9

Tree-SHA512: f1b9e4e0ee45aae892e8bf38e04b5ee5fbc643d6e7e27d011b829ed8701dacf966a99b7c877c46cca8666b894a375633e62582c552c8203614c6f2b9c4087585

2bc51c5 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
a26e8ee [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in #20981 (comment). The first tracepoints were added in #22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

ACKs for top commit:
  laanwj:
    Code and documentation review ACK 2bc51c5

Tree-SHA512: d6b4f435d3260de4c48b36956f9311f65ab3b52cd03b1e0a4ba9cf47a774d8c4b31878e222b11e0ba5d233a68f7567f8a367b12a6392f688c10c11529341e837

76c60d7 test: validation:block_connected tracepoint test (0xb10c)
260e28e test: utxocache:* tracepoint tests (0xb10c)
34b27ba test: net:in/out_message tracepoint tests (0xb10c)
c934087 test: checks for tracepoint tests (0xb10c)

Pull request description:

  This adds functional tests for the USDT tracepoints added in bitcoin/bitcoin#22006 and bitcoin/bitcoin#22902. This partially fixes #23296. The tests **are probably skipped** on most systems as these tests require:
  - a Linux system with a kernel that supports BPF (and available kernel headers)
  - that Bitcoin Core is compiled with tracepoints for USDT support (default when compiled with depends)
  - [bcc](https://github.com/iovisor/bcc) installed
  - the tests are run with a privileged user that is able to e.g. do BPF syscalls and load BPF maps

  The tests are not yet run in our CI as the CirrusCI containers lack the required permissions (see bitcoin/bitcoin#23296 (comment)). Running the tests in a VM in the CI could work, but I haven't experimented with this yet. The priority was to get the actual tests done first to ensure the tracepoints work as intended for the v23.0 release. Running the tracepoint tests in the CI is planned as the next step to finish #23296.

  The tests can, however, be run against e.g. release candidates by hand. Additionally, they provide a starting point for tests for future tracepoints. PRs adding new tracepoint should include tests. This makes reviewing these PRs easier.

  The tests require privileges to execute BPF sycalls (`CAP_SYS_ADMIN` before Linux kernel 5.8 and `CAP_BPF` and `CAP_PERFMON` on 5.8+) and permissions to `/sys/kernel/debug/tracing/`. It's currently recommended to run the tests in a virtual machine (or on a VPS) where it's sensible to use the `root` user to gain these privileges. Never run python scripts you haven't carefully reviewed with `root` permissions! It's unclear if a non-root user can even gain the required privileges. This needs more experimenting.

  The goal here is to test the tracepoint interface to make sure the [documented interface](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#tracepoint-documentation) does not break by accident. The tracepoints expose implementation details. This means we also need to rely on implementation details of Bitcoin Core in these functional tests to trigger the tracepoints. An example is the test of the `utxocache:flush` tracepoint: On Bitcoin Core shutdown, the UTXO cache is flushed twice. The corresponding tracepoint test expects two flushes, too - if not, the test fails. Changing implementation details could cause these tests to fail and the tracepoint API to break. However, we purposefully treat the tracepoints only as [**semi-stable**](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#semi-stable-api). The tracepoints should not block refactors or changes to other internals.

ACKs for top commit:
  jb55:
    tACK 76c60d7
  laanwj:
    Tested ACK 76c60d7

Tree-SHA512: 9a63d945c68102e59d751bd8d2805ddd7b37185408fa831d28a9cb6641b701961389b55f216c475df7d4771154e735625067ee957fc74f454ad7a7921255364c

Summary:
```
Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin sweat_smile )

The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:
```

Backport of [[bitcoin/bitcoin#19866 | core#19866]].

```
This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).
```

Backport of [[bitcoin/bitcoin#22006 | core#22006]].

Test Plan:
  cmake -GNinja .. -DENABLE_TRACING=ON
  ninja
  ./src/bitcoind -daemon
  sudo bpftrace ../contrib/tracing/log_p2p_traffic.bt
  sudo ../contrib/tracing/p2p_monitor.py ./src/bitcoind

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Subscribers: PiRK

Differential Revision: https://reviews.bitcoinabc.org/D12623