blockstorage: XOR blocksdir *.dat files #28052
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
Concept ACK. Certainly worth doing. |
|
Oops, I clicked wrong. I just meant to Concept ACK. |
|
AFAIK there is software which reads these files intentionally. IMO they can be expected to adapt, but having the option to disable it seems like a good idea in case they aren't ready right away. |
|
Strong Concept ACK. |
|
Thanks for the feedback so far. I've kept the option to disable, but made it a simple bool instead of allowing the user to pick the XOR-key. If someone is interested, a new commit can be added to upgrade the linearize scripts to read the XOR-key. |
This was referenced Jul 12, 2023
|
Concept ACK |
This was referenced Jul 14, 2023
|
Rebased on the latest ##28060 |
fanquake
added a commit
to bitcoin-core/gui
that referenced
this pull request
Aug 1, 2023
fa633aa streams: Teach AutoFile how to XOR (MarcoFalke) 000019e Add AutoFile::detail_fread member function (MarcoFalke) fa7724b refactor: Modernize AutoFile (MarcoFalke) fa8d227 doc: Remove comments that just repeat what the code does (MarcoFalke) fafe2ca refactor: Remove redundant file check from AutoFile shift operators (MarcoFalke) 9999a49 Extract util::Xor, Add key_offset option, Add bench (MarcoFalke) Pull request description: This allows `AutoFile` to roll an XOR pattern while reading or writing to the underlying file. This is needed for bitcoin/bitcoin#28052, but can also be used in any other place. Also, there are tests, so I've split this up from the larger pull to make review easier, hopefully. ACKs for top commit: Crypt-iQ: crACK fa633aa willcl-ark: Lightly tested ACK fa633aa jamesob: reACK fa633aa ([`jamesob/ackr/28060.4.MarcoFalke.util_add_xorfile`](https://github.com/jamesob/bitcoin/tree/ackr/28060.4.MarcoFalke.util_add_xorfile)) Tree-SHA512: 6d66cad0a089a096d3f95e4f2b28bce80b349d4b76f53d09dc9a0bea4fc1b7c0652724469c37971ba27728c7d46398a4c1d289c252af4c0f83bb2fcbc6f8e90b
achow101
added a commit
that referenced
this pull request
Aug 12, 2024
….py script 77ff0ec contrib: support reading XORed blocks in linearize-data.py script (Sebastian Falbesoner) Pull request description: This PR is a small follow-up for #28052, adding support for the block linearization script to handle XORed blocksdir *.dat files. Note that if no xor.dat file exists, the XOR pattern is set to all-zeros, in order to still support blockdirs that have been created with versions earlier than 28.x. Partly fixes issue #30599. ACKs for top commit: achow101: ACK 77ff0ec tdb3: ACK 77ff0ec hodlinator: ACK 77ff0ec Tree-SHA512: 011eb02e2411de373cbbf4b26db4640fc693a20be8c2430529fba6e36a3a3abfdfdc3b005d330f9ec2846bfad9bfbf34231c574ba99289ef37dd51a68e6e7f3d
glozow
added a commit
that referenced
this pull request
Aug 16, 2024
…-blocksxor` option) faa1b9b test: add functional test for XORed block/undo files (`-blocksxor`) (Sebastian Falbesoner) 6b3676b test: refactor: move `read_xor_key`/`util_xor` helpers to util module (Sebastian Falbesoner) Pull request description: This PR adds a dedicated functional test for XORed block data/undo file support (bitcoind option `-blocksxor`, see PR #28052). In order to verify that the XOR pattern has been applied, the {blk,rev}*.dat files are rewritten un-XORed manually by the test while the node is shut down; the node is then started again with `-blocksxor=0`, and both the data and undo files are verified via the `verifychain` RPC (with checklevel=2). Note that starting bitcoind with `-blocksxor=0` fails if a xor key is present already, which is also tested explicitly. Fixes #30599. ACKs for top commit: glozow: ACK faa1b9b maflcko: ACK faa1b9b ismaelsadeeq: Tested ACK faa1b9b Tree-SHA512: e1df106f6b4e3ba67eca108e36d762f1b991673b881934b84cd36946496a09ce9c329c1363c36aa29409137ae4881e2d177e651359686511632ddf2870f7ca8e
PastaPastaPasta
pushed a commit
to PastaPastaPasta/dash
that referenced
this pull request
Oct 25, 2024
…ze-data.py script 77ff0ec contrib: support reading XORed blocks in linearize-data.py script (Sebastian Falbesoner) Pull request description: This PR is a small follow-up for bitcoin#28052, adding support for the block linearization script to handle XORed blocksdir *.dat files. Note that if no xor.dat file exists, the XOR pattern is set to all-zeros, in order to still support blockdirs that have been created with versions earlier than 28.x. Partly fixes issue bitcoin#30599. ACKs for top commit: achow101: ACK 77ff0ec tdb3: ACK 77ff0ec hodlinator: ACK 77ff0ec Tree-SHA512: 011eb02e2411de373cbbf4b26db4640fc693a20be8c2430529fba6e36a3a3abfdfdc3b005d330f9ec2846bfad9bfbf34231c574ba99289ef37dd51a68e6e7f3d
phlip9
added a commit
to phlip9/Blockstream-electrs
that referenced
this pull request
Nov 15, 2024
shesek
pushed a commit
to Blockstream/electrs
that referenced
this pull request
Nov 23, 2024
liuchengxu
added a commit
to liuchengxu/Rusty-Bitcoin-Explorer
that referenced
this pull request
Dec 2, 2024
This was referenced Feb 9, 2025
DashCoreAutoGuix
pushed a commit
to DashCoreAutoGuix/dash
that referenced
this pull request
Jul 27, 2025
fa633aa streams: Teach AutoFile how to XOR (MarcoFalke) 000019e Add AutoFile::detail_fread member function (MarcoFalke) fa7724b refactor: Modernize AutoFile (MarcoFalke) fa8d227 doc: Remove comments that just repeat what the code does (MarcoFalke) fafe2ca refactor: Remove redundant file check from AutoFile shift operators (MarcoFalke) 9999a49 Extract util::Xor, Add key_offset option, Add bench (MarcoFalke) Pull request description: This allows `AutoFile` to roll an XOR pattern while reading or writing to the underlying file. This is needed for bitcoin#28052, but can also be used in any other place. Also, there are tests, so I've split this up from the larger pull to make review easier, hopefully. ACKs for top commit: Crypt-iQ: crACK fa633aa willcl-ark: Lightly tested ACK fa633aa jamesob: reACK fa633aa ([`jamesob/ackr/28060.4.MarcoFalke.util_add_xorfile`](https://github.com/jamesob/bitcoin/tree/ackr/28060.4.MarcoFalke.util_add_xorfile)) Tree-SHA512: 6d66cad0a089a096d3f95e4f2b28bce80b349d4b76f53d09dc9a0bea4fc1b7c0652724469c37971ba27728c7d46398a4c1d289c252af4c0f83bb2fcbc6f8e90b
DashCoreAutoGuix
pushed a commit
to DashCoreAutoGuix/dash
that referenced
this pull request
Jul 28, 2025
…en*File() fa56c42 Return CAutoFile from BlockManager::Open*File() (MarcoFalke) 9999b89 Make BufferedFile to be a CAutoFile wrapper (MarcoFalke) fa389d9 refactor: Drop unused fclose() from BufferedFile (MarcoFalke) Pull request description: This is required for bitcoin#28052, but makes sense on its own, because offloading logic to `CAutoFile` instead of re-implementing it allows to delete code and complexity. ACKs for top commit: TheCharlatan: Re-ACK fa56c42 willcl-ark: tACK fa56c42 Tree-SHA512: fe4638f3a6bd3f9d968cfb9ae3259c9d6cd278fe2912cbc90289851311c8c781099db4c160e775960975c4739098d9af801a8d2d12603f371f8edfe134d8f85a
DashCoreAutoGuix
pushed a commit
to DashCoreAutoGuix/dash
that referenced
this pull request
Aug 2, 2025
fa633aa streams: Teach AutoFile how to XOR (MarcoFalke) 000019e Add AutoFile::detail_fread member function (MarcoFalke) fa7724b refactor: Modernize AutoFile (MarcoFalke) fa8d227 doc: Remove comments that just repeat what the code does (MarcoFalke) fafe2ca refactor: Remove redundant file check from AutoFile shift operators (MarcoFalke) 9999a49 Extract util::Xor, Add key_offset option, Add bench (MarcoFalke) Pull request description: This allows `AutoFile` to roll an XOR pattern while reading or writing to the underlying file. This is needed for bitcoin#28052, but can also be used in any other place. Also, there are tests, so I've split this up from the larger pull to make review easier, hopefully. ACKs for top commit: Crypt-iQ: crACK fa633aa willcl-ark: Lightly tested ACK fa633aa jamesob: reACK fa633aa ([`jamesob/ackr/28060.4.MarcoFalke.util_add_xorfile`](https://github.com/jamesob/bitcoin/tree/ackr/28060.4.MarcoFalke.util_add_xorfile)) Tree-SHA512: 6d66cad0a089a096d3f95e4f2b28bce80b349d4b76f53d09dc9a0bea4fc1b7c0652724469c37971ba27728c7d46398a4c1d289c252af4c0f83bb2fcbc6f8e90b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently the *.dat files in the blocksdir store the data received from remote peers as-is. This may be problematic when a program other than Bitcoin Core tries to interpret them by accident. For example, an anti-virus program or other program may scan them and move them into quarantine, or delete them, or corrupt them. This may cause Bitcoin Core to fail a reorg, or fail to reply to block requests (via P2P, RPC, REST, ...).
Fix this, similar to #6650, by rolling a random XOR pattern over the dat files when writing or reading them.
Obviously this can only protect against programs that accidentally and unintentionally are trying to mess with the dat files. Any program that intentionally wants to mess with the dat files can still trivially do so.
The XOR pattern is only applied when the blocksdir is freshly created, and there is an option to disable it (on creation), so that people can disable it, if needed.