fix(gateway): prevent blocked CIDs in CAR responses #1019
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #1019 +/- ##
==========================================
- Coverage 60.53% 60.50% -0.04%
==========================================
Files 267 267
Lines 33269 33349 +80
==========================================
+ Hits 20140 20177 +37
- Misses 11467 11509 +42
- Partials 1662 1663 +1
... and 7 files with indirect coverage changes 🚀 New features to boost your workflow:
|
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 30, 2025
Updates boxo to include the fix for blocked CIDs appearing in CAR responses and enables the previously commented test cases. Uses boxo PR #1019 which ensures content filtering policies are enforced for CAR format responses by properly separating DAGService usage during traversal. Ref: ipfs/boxo#1019 Closes #10361
The gateway was including blocked CIDs in CAR format responses,
bypassing content filtering policies.
The fix separates the DAGService usage in GetCAR:
- nodeGetterToCarExporer continues wrapping for path resolution
- Original dagService is now used for blockOpener during traversal
- blockOpener returns traversal.SkipMe{} ONLY for blocked content
- NotFound errors are properly propagated (not skipped)
This ensures blocked content is filtered from CAR responses while
properly failing when blocks are genuinely missing (broken DAG).
Closes ipfs/kubo#10361
334247a to
dba0bc4
Compare
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 30, 2025
Updates boxo to include the fix for blocked CIDs appearing in CAR responses and enables the previously commented test cases. Uses boxo PR #1019 which ensures content filtering policies are enforced for CAR format responses by properly separating DAGService usage during traversal. Ref: ipfs/boxo#1019 Closes #10361
- fixed HTTP 410 for blocked root CIDs in CAR requests - prevented blocked content from appearing in CAR output during traversal - removed DAG service session that was bypassing blocking checks - added proper terminal block handling without recursive fetches - ensured CAR generation respects content blocking at all levels fixes #458 (empty CAR with 200 status) fixes ipfs/kubo#10361 (blocked CIDs in CAR output)
Added TODO comments linking to ipfs-shipyard/nopfs#34 to track where we need to restore .Session(ctx) calls once sessions properly support denylist checks. Sessions provide performance benefits but currently bypass content blocking by accessing the blockstore directly.
This was referenced Aug 31, 2025
feat: relax CAR tests to accept both 200 and 404 for non-existing paths
ipfs/gateway-conformance#244
Merged
- use blockservice.NewSession for automatic session reuse - properly enforce denylist checks via wrapped blockstore/exchange - improve performance with request-level session batching requires ipfs-shipyard/nopfs#50 fixes #458 fixes ipfs/kubo#10361
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 31, 2025
- nopfs: SessionExchange support and optimized blocking checks ipfs-shipyard/nopfs#50 - boxo: restored session usage in CAR streaming with proper blocking ipfs/boxo#1019 fixes #10361
removed special case handling for DagScopeAll since comprehensive nopfs wrappers now handle blocking at service level - removed cw parameter from walkGatewaySimpleSelector - unified CAR writing through blockGetter for all scopes
wrap HTTP error responses from remote CAR fetcher with ErrorStatusCode to ensure proper error classification in gateway handlers - 404 responses are now recognized by isErrNotFound() - 410 responses are now recognized by isErrContentBlocked() - limit error message reading to 512 bytes to prevent memory exhaustion - properly close/drain response body to avoid connection leaks - fixes directory listing failures with remote-car-backend
move header setting after GetCAR call to ensure proper HTTP status codes (404/410) are returned when paths don't exist, instead of sending 200 with X-Stream-Error header this allows gateway-conformance tests to properly handle non-existing paths in CAR responses
lidel
added a commit
to ipfs/gateway-conformance
that referenced
this pull request
Sep 1, 2025
add temporary support for legacy 200 with X-Stream-Error header behavior when missing blocks are detected during streaming. this ensures compatibility with current boxo/gateway implementations that defer header setting. relates to ipfs/boxo@1ac242a6 which fixes CAR handler to properly return 404 status codes instead of 200 with X-Stream-Error TODO: remove once Kubo ships with ipfs/boxo#1019
lidel
added a commit
to ipfs/gateway-conformance
that referenced
this pull request
Sep 1, 2025
* test: add third option for CAR non-existing path responses add temporary support for legacy 200 with X-Stream-Error header behavior when missing blocks are detected during streaming. this ensures compatibility with current boxo/gateway implementations that defer header setting. relates to ipfs/boxo@1ac242a6 which fixes CAR handler to properly return 404 status codes instead of 200 with X-Stream-Error TODO: remove once Kubo ships with ipfs/boxo#1019 * docs: add v0.8.3 changelog entry document temporary support for legacy X-Stream-Error header behavior in CAR responses for non-existing paths
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Warning
This wip experimentation to see gateway-conformance regressions, not ready for review yet.
fix for ipfs/kubo#10361
The gateway was including blocked CIDs in CAR format responses, bypassing content filtering policies.
The fix separates the DAGService usage in GetCAR:
nodeGetterToCarExporercontinues wrapping for path resolutiondagServiceis now used forblockOpenerduring traversalblockOpenerreturnstraversal.SkipMe{}for blocked contentThis ensures blocked content is filtered from CAR responses while allowing partial CAR generation when internal blocks are blocked.
fix for #458
We wait with headers until first block, and return 410 / 404 when we know we can't serve the data.