Skip to content

Conversation

lidel
Copy link
Member

@lidel lidel commented Jul 24, 2025

Note

🔍 REVIEW HIGHLIGHT: this PR looks scary (>10k of lines) but its mostly very very comprehensive tests :)

This PR contains three distinct components:

  1. boxo/autoconf/ library is in feat: autoconf client library boxo#997
  2. config.AutoConf.URL integration - Kubo integration using https://config.ipfs-mainnet.org/autoconf.json by default with explicit "auto" placeholders (no implicit defaults), plus --expand-auto CLI flag for viewing resolved values
  3. Embedded migration 16→17 - Template for future migrations, migrates existing users to explicit "auto" placeholders matching fresh ipfs init

Directory Tree & Review Focus

PR FOCUS AREAS:

Kubo Integration (~1.2k LOC):
├── config/
│   ├── autoconf.go                  # Config expansion & helpers
│   │   ├── DNSResolversWithAutoConf()
│   │   ├── BootstrapWithAutoConf()
│   │   ├── DelegatedRoutersWithAutoConf()
│   │   └── DelegatedPublishersWithAutoConf()
│   └── autoconf_client.go           # Client management & caching

Runtime Integration:
├── core/commands/
│   ├── config.go                    # --expand-auto CLI flag
│   └── bootstrap.go                 # Bootstrap --expand-auto support
├── core/node/libp2p/
│   ├── host.go                      # Bootstrap peer expansion
│   └── routingopt.go                # Delegated routing expansion

Migration 16→17 (~800 LOC):
├── repo/fsrepo/migrations/
│   ├── embedded.go                  # Embedded migration framework
│   ├── atomicfile/                  # Atomic file operations (moved here)
│   └── fs-repo-16-to-17/
│       └── migration/
│           ├── migration.go         # Core migration logic
│           └── migration_test.go    # Migration tests

Testing (~3k LOC):
├── test/cli/
│   ├── autoconf/                    # AutoConf integration tests
│   └── migrations/                  # Migration tests
└── docs/config.md                   # Configuration documentation

Problem

Hardcoded defaults accumulated over time make it difficult to:

  • Truly disable defaults in custom swarms
  • Swap endpoints during third-party outages (#771, #772)
  • Understand which values are dynamic vs static

Solution

Replace implicit hardcoded values with explicit "auto" placeholders resolved from remote autoconf.json manifest.

New ipfs init behavior

{
  "Bootstrap": ["auto"],
  "DNS": {
    "Resolvers": {".": "auto"}
  },
  "Routing": {
    "DelegatedRouters": ["auto"]
  },
  "Ipns": {
    "DelegatedPublishers": ["auto"]
  },
  "AutoConf": {
    "Enabled": true,
    "URL": "https://config.ipfs-mainnet.org/autoconf.json"
  }
}

Implementation Highlights

  • Non-intrusive integration: Robust fallback system prevents daemon from ever being bricked - falls back to last valid cache or user-provided fallbacks
  • Explicit "auto" placeholders: No implicit defaults - all dynamic values use visible "auto" placeholders in config
  • Versioned cache: Multiple config versions stored with configurable TTL and automatic cleanup
  • Background refresh: Daemon handles network updates every 24h, CLI commands use cache-only for speed
  • Zero breaking changes: Existing configurations work unchanged, custom user-config is respected

CLI Usage

# View current config (shows "auto" placeholders)
ipfs config Bootstrap

# View resolved values
ipfs config Bootstrap --expand-auto
ipfs bootstrap list --expand-auto
ipfs config show --expand-auto

Migration 16→17

  • Replaces hardcoded defaults with "auto" while preserving user-set values
  • Runs instantly using embedded code (no remote binary downloads)
    • Template for future embedded migrations
    • 💡 This is very important for future-proofing Kubo's existence. Security best practices of operating systems changed in the past decade and fetching binaries and executing them behind user's back triggers too many red flags these days. At some point it may even be fully forbidden (e.g. macOS is heading in that direction)
  • ipfs repo migrate now supports target version: ipfs repo migrate --to=17

Documentation

See AutoConf section in docs/config.md

References



"The Integration of AutoConfig in late Kubo" (1509-1511) by Raphael

lidel added 5 commits July 18, 2025 18:22
tdb, we may extract this later for use outside of Kubo
this needs more testing and work, but we will do that
after autoconfig is wired in future commits
- tests need manual inspection / being more DRY / reusable for future
  migrations
- need to see if all CI tests pass
@lidel lidel mentioned this pull request Jul 24, 2025
51 tasks
@lidel lidel changed the title feat(config): AutoConfig for IPFS Mainnet feat(config): AutoConfig Jul 24, 2025
lidel added 7 commits July 25, 2025 05:53
identified some test gaps - will fix in future commit
ensuring we do only one http get per AutoConfig.RefreshInterval
use file and dont depend on filesystem timestamps (less problematic
across platforms and deployments, some people mount filesystems without
atime or mtime  for performance)
+ backward compatibility for `ipfs bootstrap add default`
- follows existing --allow-* convention for degraded modes
- clarifies --allow-offline behavior in docs
- improves test clarity and removes duplication
- adds test to verify PUT payload matches routing get
@lidel lidel force-pushed the feat-mainnet-autoconfig branch from 9e06f4d to a7b0b21 Compare August 19, 2025 13:48
@lidel
Copy link
Member Author

lidel commented Aug 19, 2025

Things to agree before merge:

  • do we hardcode BOTH AutoConf.Enabled and AutoConf.URL in config, or have implicit defaults like we do for AutoTLS.Enabled and AutoTLS.DomainSuffix.
    • I'm leaning towards following what we did for AutoTLS, and having explicit values only if user applied explicit profile like test, autoconf-on or autoconf-off
  • Update migrations based on the decision.

lidel added 3 commits August 19, 2025 20:20
- change AutoConf.URL to *OptionalString with omitempty
- use WithDefault(DefaultAutoConfURL) for implicit defaults
- update migration to omit URL field for cleaner config
- remove explicit empty field validation

this makes AutoConf work like AutoTLS where defaults are
implicit unless explicitly overridden by the user.
use slices.Sorted(maps.Keys()) for consistent alphabetical ordering
the test profile explicitly disables AutoConf, so we need to
enable it for the 'bootstrap add default' test to work correctly
@lidel lidel force-pushed the feat-mainnet-autoconfig branch from c84f145 to 8ec830b Compare August 19, 2025 19:23
lidel added 3 commits August 19, 2025 22:18
- add SetDoesNotUseRepo(true) to repoMigrateCmd to enable direct filesystem access
- migrations need filesystem access without daemon API connection
- manual lock checking at lines 427-433 still protects against concurrent access
- fix t0066-migration.sh test by removing explicit AutoConf.Enabled=false
  The test profile disables AutoConf by default, which prevents the daemon
  from starting after v16-to-17 migration adds 'auto' values. Since sharness
  tests use the test profile by default, we need to remove the explicit
  AutoConf.Enabled=false to allow implicit defaults that work with 'auto' values.
- remove Migration initialization from config/init.go (no longer in new configs)
- add omitempty to Migration.DownloadSources and Migration.Keep fields
- these fields now only appear in JSON when explicitly set
- update docs to clarify Migration is deprecated (only for repos <v16)
- emphasize that only HTTPS is supported for legacy migrations
@lidel lidel force-pushed the feat-mainnet-autoconfig branch 14 times, most recently from 0d89566 to d028a23 Compare August 20, 2025 03:00
@lidel
Copy link
Member Author

lidel commented Aug 20, 2025

All green, merging to master. We will cut 0.37 RC1 on Thursday.

migration now creates empty AutoConf{} section instead of setting
explicit values. fields use implicit defaults via WithDefault():
- Enabled defaults to true
- RefreshInterval defaults to 24h
- TLSInsecureSkipVerify defaults to false
@lidel lidel force-pushed the feat-mainnet-autoconfig branch from 7447d93 to 2b32db3 Compare August 20, 2025 03:45
@lidel lidel merged commit ccb49de into master Aug 20, 2025
15 checks passed
@lidel lidel deleted the feat-mainnet-autoconfig branch August 20, 2025 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip/changelog This change does NOT require a changelog entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Routing.Type=delegated to only use Routing.DelegatedRouters Migrating users to new configuration (protocols, bootstrap peers etc.)
2 participants