Skip to content

Conversation

virtu
Copy link
Owner

@virtu virtu commented Aug 23, 2024

  • Consider non-standard ports bad
  • Consider connection times during node quality check
    • IPv4, IPv6 and CJDNS: Bitcoin Core defaults to a 5s timeout for these networks. Since responsive seed nodes are preferred, this
      threshold is reduced by 50%.
    • Onion and I2P: Bitcoin Core defaults to a 20s timeout for
      socks5 connections. Since Onion and I2P network performance are prone to variations
      (intraday fluctuations caused by usage profiles during working ours of different
      geographic regions, attacks on the networks, etc.), connection times are granted 20%
      of slack.

@virtu virtu force-pushed the judge_node_connection_time branch from 5aa10aa to 19b95d9 Compare August 23, 2024 06:40
@virtu virtu merged commit f2072bb into master Aug 23, 2024
@virtu virtu deleted the judge_node_connection_time branch August 23, 2024 06:41
achow101 added a commit to bitcoin/bitcoin that referenced this pull request Aug 27, 2024
…rements for Onion and I2P nodes

b061b35 seeds: Regenerate mainnet seeds (virtu)
02dc45c seeds: Pull nodes from Luke's seeder (virtu)
7a2068a seeds: Pull nodes from virtu's crawler (virtu)

Pull request description:

  This builds on #30008 and adds data [exported](https://github.com/virtu/seed-exporter) by [my crawler](https://github.com/virtu/p2p-crawler) an additional source for seed nodes. Data covers all supported network types.

  [edit: Added Luke's seeder as input as well.]

  ### Motivation
  - Further decentralizes the seed node selection process (in the long term potentially enabling an _n_-source threshold for nodes to prevent a single source from entering malicious nodes)
  - No longer need to manually curate seed node list for any network type: See last paragraph of OP in #30008. My crawler has been [discovering the handful of available cjdns nodes](https://21.ninja/reachable-nodes/nodes-by-net-type/) for around two months, all but one of which meet the reliability criteria.
  - Alignment of uptime requirements for Onion and I2P nodes with those of clearnet nodes to 50%: If I'm reading the code correctly, seeders appear to optimize for up-to-dateness by using [lower connection timeouts](https://github.com/achow101/dnsseedrs/blob/3c1a63c6723819871d76fe0fbd2155fe5a5bb171/src/crawl.rs#L349) than [Bitcoin Core](https://github.com/bitcoin/bitcoin/blob/bc87ad98543299e1990ee1994d0653df3ac70093/src/netbase.cpp#L40C27-L40C48) to maximize throughput. Since my crawler does not have the same timeliness requirements, it opts for accuracy by using generous timeouts. As a result, its data contains additional eligible Onion (and other darknet nodes), as is shown in the histogram below. Around 4500 Onion nodes are discovered so far (blue); my data adds ~6400 more (orange); ~ 1500 nodes take longer than the default 20-second Bitcoin Core timeout and won't qualify as "good".

  ![Connection time histogram for Onion nodes](https://github.com/user-attachments/assets/c3513604-aa48-4c75-b51d-13421eaed9eb)

  Here's the current results with 512 nodes for all networks except cjdns:
  <details>
  <summary>Using the extra data</summary>

  ```
  IPv4   IPv6  Onion  I2P    CJDNS Pass
  10335   2531  11545   1589     10 Initial
  10335   2531  11545   1589     10 Skip entries with invalid address
  5639   1431  11163   1589      8 After removing duplicates
  5606   1417  11163   1589      8 Enforce minimal number of blocks
  5606   1417  11163   1589      8 Require service bit 1
  4873   1228  11163   1589      8 Require minimum uptime
  4846   1225  11161   1588      8 Require a known and recent user agent
  4846   1225  11161   1588      8 Filter out hosts with multiple bitcoin ports
  512    512    512    512      8 Look up ASNs and limit results per ASN and per net
  ```
  </details>
  <details>
  <summary>Before</summary>

  ```
  IPv4   IPv6  Onion  I2P    CJDNS Pass
  5772   1323    443      0      2 Initial
  5772   1323    443      0      2 Skip entries with invalid address
  4758   1110    443      0      2 After removing duplicates
  4723   1094    443      0      2 Enforce minimal number of blocks
  4723   1094    443      0      2 Require service bit 1
  3732    867    443      0      2 Require minimum uptime
  3718    864    443      0      2 Require a known and recent user agent
  3718    864    443      0      2 Filter out hosts with multiple bitcoin ports
   512    409    443      0      2 Look up ASNs and limit results per ASN and per net
  ```
  </details>

  ### To dos
  - [x] Remove manual nodes and update README
  - [x] Mark nodes with connection times exceeding Bitcoin Core's default as bad in [exporter](https://github.com/virtu/seed-exporter): [done](virtu/seed-exporter#12)
  - [x] Regenerate mainnet seeds
  - [x] Rebase, then remove WIP label once #30008 gets merged

ACKs for top commit:
  achow101:
    ACK b061b35
  fjahr:
    utACK b061b35

Tree-SHA512: 63e86220787251c7e8d2d5957bad69352e19ae17d7b9b2d27d8acddfec5bdafe588edb68d77d19c57f25f149de723e2eeadded0c8cf13eaca22dc33bd8cf92a0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant