feat: telemetry plugin #10866
Merged
feat: telemetry plugin #10866
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, IP Shipyard, with the help of Probelab, monitor and extract Amino/IPFS public network metrics with the use of DHT crawlers and bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and their AgentVersion/Addresses obtained from the `identify` protocol, which provides insights into protocol usage, total number of peers etc. We would like to increase the ability to obtain more insights from the network by collecting some more information in the future, but also to give users more control over this collection (i.e. opt-out). The information collected will not allow unique identification of anyone and is only used for aggregation. Now, this PR explores a way of moving in this direction: * A new "telemetry" fx plugin is in charge of dealing with telemetry * The FX plugin allows to plug and make decisions / take actions during the setup phase: * We can inspect whether we are using Private Networks before the libp2p.Host has been initialized. * We can send telemetry after the libp2p Host is initialized. * Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code. As for behaviour: * The user can opt-in/out via EnvVar, file in the repo path or plugin configuration. * Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored. * Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out. The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer.
* On first run it generates a UUID and shows a message to the user. * UUID is persistend to "telemetry_uuid" * Sends telemetry 1 minute after boot and every 24h * LogEvent is the thing containing all the telemetry that is sent * Opt-out possible via env-var or plugin configuration TODO: server URL
hsanjuan
commented
Jul 14, 2025
lidel
reviewed
Jul 21, 2025
lidel
reviewed
Jul 21, 2025
lidel
reviewed
Jul 21, 2025
making it more obvious nothing was sent yet and that user had 15m to out-out plus some debug logs that confirm opt-out
lidel
approved these changes
Aug 12, 2025
There was a problem hiding this comment.
LGTM. @hsanjuan @gammazero if CI is green and changes I pushed look ok, feel free to Merge (squash):
- Improved error handling if our endpoint is down or misbehaves (warning + 30s timeout)
- Added tests to ensure opt-out is respected and UUID is removed
- Updated
ipfs daemonone-time notice to make it more clear that nothing was sent yet, and user had 15m to opt-out:$ ipfs init $ ipfs daemon Initializing daemon... Kubo version: 0.37.0-dev-0c1dc4d97 Repo version: 16 System version: amd64/linux Golang version: go1.24.6 PeerID: 12D3KooWPP1uvLNhscFFQ1GgwCFNHhKX1653QwWdQUGaeCxRumB7 Swarm listening on 127.0.0.1:4001 (TCP+UDP) Swarm listening on [::1]:4001 (TCP+UDP) Run 'ipfs id' to inspect announced and discovered multiaddrs of this node. ℹ️ Anonymous telemetry will be enabled in 15m0s Kubo will collect anonymous usage data to help improve the software: • What: Feature usage and configuration (no personal data) Use GOLOG_LOG_LEVEL="telemetry=debug" to inspect collected data • When: First collection in 15m0s, then every 24h • How: HTTP POST to https://telemetry.ipshipyard.dev Anonymous ID: a21fc379-c3d9-4011-8aec-c8a648aee1d8 No data sent yet. To opt-out before collection starts: • Set environment: IPFS_TELEMETRY_PLUGIN_MODE=optout • Or run: ipfs config Plugins.Plugins.telemetry.Config.Mode optout • Then restart daemon This message is shown only once. Learn more: https://github.com/ipfs/kubo/blob/master/docs/telemetry.md RPC API server listening on /ip4/127.0.0.1/tcp/5001 WebUI: http://127.0.0.1:5001/webui Gateway server listening on /ip4/127.0.0.1/tcp/8080 Daemon is ready
ipfs-webui tests weare trying to use addAbortListener from node:events, which requires Node.js v20.5.0+, but the CI is using v18.14.0
lidel
reviewed
Aug 12, 2025
gammazero
approved these changes
Aug 13, 2025
gammazero
approved these changes
Aug 15, 2025
lidel
reviewed
Aug 18, 2025
- rename info mode to auto - clarify auto mode behavior in docs
- add emoji to telemetry section title - move telemetry to end of highlights section - improve title and description clarity
lidel
approved these changes
Aug 18, 2025
There was a problem hiding this comment.
Lgtm.
Pushed cosmetic changes moving changelog entry bit down the list, and renaming IPFS_TELEMETRY_PLUGIN_MODE=optin|optout|info to IPFS_TELEMETRY=on|off|auto so it is more clear when its on/off and also easier to control default implicit behavior by following convention we have for other things.
Merging to include in 0.37.0-rc1 to gather feedback.
hsanjuan
commented
Aug 20, 2025
| - `info`: Like `optin`, but logs an informative message about telemetry (default in the first run). | ||
| - `on`: Enables telemetry. | ||
| - `off`: Disables telemetry. | ||
| - `auto`: Like `on`, but logs an informative message about telemetry and gives user 15 minutes to opt-out before first collection. Used automatically on first run and when `IPFS_TELEMETRY` is not set. |
There was a problem hiding this comment.
(Note the 15 minutes are not just during auto or during first run.)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, IP Shipyard, with the help of Probelab, monitor and extract Amino/IPFS public network metrics with the use of DHT crawlers and bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and their AgentVersion/Addresses obtained from the
identifyprotocol, which provides insights into protocol usage, total number of peers etc.We would like to increase the ability to obtain more insights from the network by collecting some more information but also to give users more control over this collection (i.e. opt-out). The information collected will not allow unique identification of anyone and is only used for aggregation.
This PR adds:
Telemetry submission consists in a POST request to a telemetry-collecting HTTP endpoint.
This work is part of roadmap sponsored by IPFS Foundation: