Skip to content

Conversation

furszy
Copy link
Member

@furszy furszy commented Dec 5, 2022

Currently, the test wallet_load_verif_crypted_key_checksum fails if configured without sqlite.

There are few reasons for it (chained bugs):

  1. As in the test we encrypt the wallet, and the encryption process
    requires a db rewrite + environment reload, we cannot use mock dbs
    (mock dbs are memory-only).

  2. Once the first issue gets solved, the next one is a deadlock inside
    DeleteRecords due a lack of sync between the opened cursor and
    each of the Erase calls (they happen on different db txn contexts
    so while we traverse the db, we aren't able to erase records).

    Note: I do have a pending research here. Because this should
    be affecting other places as well.. (ideally, read-only operations
    shouldn't lock the db)

Extra Info:
Have renamed the long and ugly wallet_load_verif_crypted_key_checksum
test name to wallet_load_ckey.

@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 5, 2022

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
Stale ACK achow101

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #26715 (Introduce MockableDatabase for wallet unit tests by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@DrahtBot DrahtBot added the Wallet label Dec 5, 2022
@achow101
Copy link
Member

achow101 commented Dec 8, 2022

ACK a588708

furszy added 3 commits May 1, 2023 10:22
…yCursor

If the batch ptr is not passed, the cursor will not use the db active
txn context which could lead to a deadlock if the code tries to modify
the db while it is traversing it.

E.g. the 'EraseRecords()' function.
so we erase all the records atomically or abort the entire
procedure.

and, at the same time, we can share the same db txn context
for the db cursor and the erase functionality.

extra note from the Db.cursor doc:
"If transaction protection is enabled, cursors must be
opened and closed within the context of a transaction"

thus why added a `CloseCursor` call before calling to
`TxnAbort/TxnCommit`.
As in the test we encrypt the wallet, and the encryption process
requires a db rewrite + environment reload, we cannot use mock dbs
(mock dbs are memory-only).
@achow101
Copy link
Member

achow101 commented May 2, 2023

Would it be easier to do this on top of #26715?

@furszy
Copy link
Member Author

furszy commented May 2, 2023

Would it be easier to do this on top of #26715?

Was waiting for #27556 CI to close this in favor of #26715 actually. The heart of this PR is 32ffea2 which is just a test code re-ordering to by-pass the memory-only db limitations in the db rewrite + environment reload events executed in the wallet encryption process. Reality is that we don't care about which db engine the test uses, the test exercises loading invalid data into the wallet.

With #26715 inclusion, the test db is just a raw map in-memory and not a temp bdb, so the issue does no longer exist.

@furszy furszy closed this May 2, 2023
fanquake added a commit that referenced this pull request May 15, 2023
33e2b82 wallet, bench: Remove unused database options from WalletBenchLoading (Andrew Chow)
80ace04 tests: Modify records directly in wallet ckey loading test (Andrew Chow)
b3bb17d tests: Update DuplicateMockDatabase for MockableDatabase (Andrew Chow)
f0eecf5 scripted-diff: Replace CreateMockWalletDB with CreateMockableWalletDB (Andrew Chow)
075962b wallet, tests: Include wallet/test/util.h (Andrew Chow)
14aa4cb wallet: Move DummyDatabase to salvage (Andrew Chow)
f67a385 wallet, tests: Replace usage of dummy db with mockable db (Andrew Chow)
33c6245 Introduce MockableDatabase for wallet unit tests (Andrew Chow)

Pull request description:

  For the wallet's unit tests, we currently use either `DummyDatabase` or memory-only versions of either BDB or SQLite. The tests that use `DummyDatabase` just need a `WalletDatabase` so that the `CWallet` can be constructed, while the tests using the memory-only databases just need a backing data store. There is also a `FailDatabase` that is similar to `DummyDatabase` except it fails be default or can have a configured return value. Having all of these different database types can make it difficult to write tests, particularly tests that work when either BDB or SQLite is disabled.

  This PR unifies all of these different unit test database classes into a single `MockableDatabase`. Like `DummyDatabase`, most functions do nothing and just return true. Like `FailDatabase`, the return value of some functions can be configured on the fly to test various failure cases. Like the memory-only databases, records can actually be written to the `MockableDatabase` and be retrieved later, but all of this is still held in memory. Using `MockableDatabase` completely removes the need for having BDB or SQLite backed wallets in the unit tests for the tests that are not actually testing specific database behaviors.

  Because `MockableDatabase`s can be created by each unit test, we can also control what records are stored in the database. Records can be added and removed externally from the typical database modification functions. This will give us greater ability to test failure conditions, particularly those involving corrupted records.

  Possible alternative to #26644

ACKs for top commit:
  furszy:
    ACK 33e2b82
  TheCharlatan:
    ACK 33e2b82

Tree-SHA512: c2b09eff9728d063d2d4aea28a0f0e64e40b76483e75dc53f08667df23bd25834d52656cd4eafb02e552db0b9e619cfdb1b1c65b26b5436ee2c971d804768bcc
achow101 added a commit that referenced this pull request May 18, 2023
69d4390 test: add coverage for wallet read write db deadlock (furszy)
12daf6f walletdb: scope bdb::EraseRecords under a single db txn (furszy)
043fcb0 wallet: bugfix, GetNewCursor() misses to provide batch ptr to BerkeleyCursor (furszy)

Pull request description:

  Decoupled from #26644 so it can closed in favor of #26715.

  Basically, with bdb, we can't make a write operation while we are traversing the db with the same db handler. These two operations are performed in different txn contexts and cause a deadlock.

  Added coverage by using `EraseRecords()` which is the simplest function that executes this process.

  To replicate it, need bdb support and drop the first commit. The test will run forever.

ACKs for top commit:
  achow101:
    ACK 69d4390
  hebasto:
    re-ACK 69d4390

Tree-SHA512: b3773be78925f674e962f4a5c54b398a9d0cfe697148c01c3ec0d68281cc5c1444b38165960d219ef3cf1a57c8ce6427f44a876275958d49bbc0808486e19d7d
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request May 19, 2023
69d4390 test: add coverage for wallet read write db deadlock (furszy)
12daf6f walletdb: scope bdb::EraseRecords under a single db txn (furszy)
043fcb0 wallet: bugfix, GetNewCursor() misses to provide batch ptr to BerkeleyCursor (furszy)

Pull request description:

  Decoupled from bitcoin#26644 so it can closed in favor of bitcoin#26715.

  Basically, with bdb, we can't make a write operation while we are traversing the db with the same db handler. These two operations are performed in different txn contexts and cause a deadlock.

  Added coverage by using `EraseRecords()` which is the simplest function that executes this process.

  To replicate it, need bdb support and drop the first commit. The test will run forever.

ACKs for top commit:
  achow101:
    ACK 69d4390
  hebasto:
    re-ACK 69d4390

Tree-SHA512: b3773be78925f674e962f4a5c54b398a9d0cfe697148c01c3ec0d68281cc5c1444b38165960d219ef3cf1a57c8ce6427f44a876275958d49bbc0808486e19d7d
@furszy furszy deleted the 2022_walletdb_fix_bdb_deadlock branch May 27, 2023 01:46
@bitcoin bitcoin locked and limited conversation to collaborators May 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants