-
Notifications
You must be signed in to change notification settings - Fork 37.7k
wallet: bugfix, 'wallet_load_ckey' unit test fails with bdb #26644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
ACK a588708 |
a588708
to
8e666df
Compare
8e666df
to
c977a56
Compare
…yCursor If the batch ptr is not passed, the cursor will not use the db active txn context which could lead to a deadlock if the code tries to modify the db while it is traversing it. E.g. the 'EraseRecords()' function.
so we erase all the records atomically or abort the entire procedure. and, at the same time, we can share the same db txn context for the db cursor and the erase functionality. extra note from the Db.cursor doc: "If transaction protection is enabled, cursors must be opened and closed within the context of a transaction" thus why added a `CloseCursor` call before calling to `TxnAbort/TxnCommit`.
As in the test we encrypt the wallet, and the encryption process requires a db rewrite + environment reload, we cannot use mock dbs (mock dbs are memory-only).
c977a56
to
32ffea2
Compare
Would it be easier to do this on top of #26715? |
Was waiting for #27556 CI to close this in favor of #26715 actually. The heart of this PR is 32ffea2 which is just a test code re-ordering to by-pass the memory-only db limitations in the db rewrite + environment reload events executed in the wallet encryption process. Reality is that we don't care about which db engine the test uses, the test exercises loading invalid data into the wallet. With #26715 inclusion, the test db is just a raw map in-memory and not a temp bdb, so the issue does no longer exist. |
33e2b82 wallet, bench: Remove unused database options from WalletBenchLoading (Andrew Chow) 80ace04 tests: Modify records directly in wallet ckey loading test (Andrew Chow) b3bb17d tests: Update DuplicateMockDatabase for MockableDatabase (Andrew Chow) f0eecf5 scripted-diff: Replace CreateMockWalletDB with CreateMockableWalletDB (Andrew Chow) 075962b wallet, tests: Include wallet/test/util.h (Andrew Chow) 14aa4cb wallet: Move DummyDatabase to salvage (Andrew Chow) f67a385 wallet, tests: Replace usage of dummy db with mockable db (Andrew Chow) 33c6245 Introduce MockableDatabase for wallet unit tests (Andrew Chow) Pull request description: For the wallet's unit tests, we currently use either `DummyDatabase` or memory-only versions of either BDB or SQLite. The tests that use `DummyDatabase` just need a `WalletDatabase` so that the `CWallet` can be constructed, while the tests using the memory-only databases just need a backing data store. There is also a `FailDatabase` that is similar to `DummyDatabase` except it fails be default or can have a configured return value. Having all of these different database types can make it difficult to write tests, particularly tests that work when either BDB or SQLite is disabled. This PR unifies all of these different unit test database classes into a single `MockableDatabase`. Like `DummyDatabase`, most functions do nothing and just return true. Like `FailDatabase`, the return value of some functions can be configured on the fly to test various failure cases. Like the memory-only databases, records can actually be written to the `MockableDatabase` and be retrieved later, but all of this is still held in memory. Using `MockableDatabase` completely removes the need for having BDB or SQLite backed wallets in the unit tests for the tests that are not actually testing specific database behaviors. Because `MockableDatabase`s can be created by each unit test, we can also control what records are stored in the database. Records can be added and removed externally from the typical database modification functions. This will give us greater ability to test failure conditions, particularly those involving corrupted records. Possible alternative to #26644 ACKs for top commit: furszy: ACK 33e2b82 TheCharlatan: ACK 33e2b82 Tree-SHA512: c2b09eff9728d063d2d4aea28a0f0e64e40b76483e75dc53f08667df23bd25834d52656cd4eafb02e552db0b9e619cfdb1b1c65b26b5436ee2c971d804768bcc
69d4390 test: add coverage for wallet read write db deadlock (furszy) 12daf6f walletdb: scope bdb::EraseRecords under a single db txn (furszy) 043fcb0 wallet: bugfix, GetNewCursor() misses to provide batch ptr to BerkeleyCursor (furszy) Pull request description: Decoupled from #26644 so it can closed in favor of #26715. Basically, with bdb, we can't make a write operation while we are traversing the db with the same db handler. These two operations are performed in different txn contexts and cause a deadlock. Added coverage by using `EraseRecords()` which is the simplest function that executes this process. To replicate it, need bdb support and drop the first commit. The test will run forever. ACKs for top commit: achow101: ACK 69d4390 hebasto: re-ACK 69d4390 Tree-SHA512: b3773be78925f674e962f4a5c54b398a9d0cfe697148c01c3ec0d68281cc5c1444b38165960d219ef3cf1a57c8ce6427f44a876275958d49bbc0808486e19d7d
69d4390 test: add coverage for wallet read write db deadlock (furszy) 12daf6f walletdb: scope bdb::EraseRecords under a single db txn (furszy) 043fcb0 wallet: bugfix, GetNewCursor() misses to provide batch ptr to BerkeleyCursor (furszy) Pull request description: Decoupled from bitcoin#26644 so it can closed in favor of bitcoin#26715. Basically, with bdb, we can't make a write operation while we are traversing the db with the same db handler. These two operations are performed in different txn contexts and cause a deadlock. Added coverage by using `EraseRecords()` which is the simplest function that executes this process. To replicate it, need bdb support and drop the first commit. The test will run forever. ACKs for top commit: achow101: ACK 69d4390 hebasto: re-ACK 69d4390 Tree-SHA512: b3773be78925f674e962f4a5c54b398a9d0cfe697148c01c3ec0d68281cc5c1444b38165960d219ef3cf1a57c8ce6427f44a876275958d49bbc0808486e19d7d
Currently, the test
wallet_load_verif_crypted_key_checksum
fails if configured without sqlite.There are few reasons for it (chained bugs):
As in the test we encrypt the wallet, and the encryption process
requires a db rewrite + environment reload, we cannot use mock dbs
(mock dbs are memory-only).
Once the first issue gets solved, the next one is a deadlock inside
DeleteRecords
due a lack of sync between the opened cursor andeach of the
Erase
calls (they happen on different db txn contextsso while we traverse the db, we aren't able to erase records).
Note: I do have a pending research here. Because this should
be affecting other places as well.. (ideally, read-only operations
shouldn't lock the db)
Extra Info:
Have renamed the long and ugly
wallet_load_verif_crypted_key_checksum
test name to
wallet_load_ckey
.