-
Notifications
You must be signed in to change notification settings - Fork 37.7k
refactor: Remove CAddressBookData::destdata #27224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
src/wallet/walletdb.cpp
Outdated
if (strKey.compare("used") == 0) { | ||
data.SetUsed(true); | ||
} else if (strKey.compare(0, 2, "rr") == 0) { | ||
data.SetReceiveRequest(strKey.substr(2), std::move(strValue)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
src/wallet/walletdb.cpp:616:58: error: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg,-warnings-as-errors]
data.SetReceiveRequest(strKey.substr(2), std::move(strValue));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Thanks, fixed now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated 2a44c4e -> 6388b30 (pr/nodest.15
-> pr/nodest.16
, compare) to fix clang-tidy std::move
warning https://cirrus-ci.com/task/4799988845248512?logs=ci#L4742
src/wallet/walletdb.cpp
Outdated
if (strKey.compare("used") == 0) { | ||
data.SetUsed(true); | ||
} else if (strKey.compare(0, 2, "rr") == 0) { | ||
data.SetReceiveRequest(strKey.substr(2), std::move(strValue)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Thanks, fixed now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated 6388b30 -> fe367f2 (pr/nodest.16
-> pr/nodest.17
, compare) adding suggested erase method, also adding more comments and removing some unnecessary changes to existing code after the rebase.
ACK fe367f2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK
I read through the motivating examples you listed and I think this change makes a lot of sense. Also great to see more test coverage!
Still reading and testing to make sure I fully understand the change but left a small suggestion in the meanwhile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated fe367f2 -> 0ad6a53 (pr/nodest.17
-> pr/nodest.18
, compare) using more brace initialization as suggested
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 0ad6a53
I left a suggestion about using DataStream&&
instead of Span<std::byte>
, but not blocking
src/wallet/bdb.h
Outdated
@@ -205,6 +206,7 @@ class BerkeleyBatch : public DatabaseBatch | |||
bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override; | |||
bool EraseKey(DataStream&& key) override; | |||
bool HasKey(DataStream&& key) override; | |||
bool ErasePrefix(Span<std::byte> prefix) override; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps there's some nuance that I'm missing, but it seems strange to have this be Span<std::byte>
when the rest of the methods take DataStream&&
. I changed the ErasePrefix
method to instead take DataStream&&
as an argument and was able to compile and pass the unit and functional tests. Seems like it would be preferable to match the other methods unless you have a reason to prefer Span<std::byte>
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: https://github.com/bitcoin/bitcoin/pull/27224/files#r1150232826
you have a reason to prefer
Span<std::byte>
?
Well it should actually take a span of const bytes, so I fixed this to add const.
Passing a span makes more sense than passing a datastream because the function just needs a read-only array of bytes, it doesn't need a read/write vector or a stream object or anything more complicated.
The other functions are using CDataStream just because they are called by older code which might be counting on them to call memory_cleanse, and would require extra work to convert to spans, though it was tried previously in #19320
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gotcha, thanks for the explanation!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code review 0ad6a53, left few points.
src/wallet/wallet.cpp
Outdated
@@ -2439,11 +2439,7 @@ bool CWallet::DelAddressBook(const CTxDestination& address) | |||
return false; | |||
} | |||
// Delete destdata tuples associated with address |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No longer the case.
Same for the comment that is above this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No longer the case. Same for the comment that is above this one.
Thanks, I basically just replaced "destdata" with "address data" in these comments, but you can let me know if the issue was something else
src/wallet/wallet.cpp
Outdated
@@ -2821,65 +2817,46 @@ unsigned int CWallet::ComputeTimeSmart(const CWalletTx& wtx, bool rescanning_old | |||
|
|||
bool CWallet::SetAddressUsed(WalletBatch& batch, const CTxDestination& dest, bool used) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really for this PR, but.. the used
arg is never false.
SetAddressUsed
is only called from SetSpentKeyState
which is only called from AddToWallet
, which provides used=true
.
Which.. means that the address is not reverted to a "not used" state if the tx gets abandoned/discarded.
Could also work on it on a follow-up. Probably after #26836 so changes don't end up conflicting that much.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Sure, I think it would make sense to clear this flag when a transaction is abandoned. I think this could be a separate issue, and it should be pretty easy to implement anytime IMO
src/wallet/wallet.cpp
Outdated
void CWallet::LoadDestData(const CTxDestination &dest, const std::string &key, const std::string &value) | ||
{ | ||
m_address_book[dest].destdata.insert(std::make_pair(key, value)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This method was removed from the cpp but not from the header.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
This method was removed from the cpp but not from the header.
Thanks applied your suggested change (furszy@2195b1e)
src/wallet/walletdb.cpp
Outdated
CAddressBookData& data{pwallet->m_address_book[DecodeDestination(strAddress)]}; | ||
if (strKey.compare("used") == 0) { | ||
data.previously_spent = true; | ||
} else if (strKey.compare(0, 2, "rr") == 0) { | ||
data.receive_requests[strKey.substr(2)] = std::move(strValue); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Would be good to keep the encapsulation and don't access
m_address_book
from outside of the wallet class (otherwise we are reverting #25337 and, future, #26836).
Thanks, applied your change (furszy@2195b1e)
// Transaction argument to cursor is only needed when using the cursor to | ||
// write to the database. Read-only cursors do not need a txn pointer. | ||
int ret = database.m_db->cursor(batch ? batch->txn() : nullptr, &m_cursor, 0); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
src/wallet/test/wallet_tests.cpp
Outdated
std::vector<bilingual_str> warnings; | ||
auto database{MakeWalletDatabase(name, options, status, error)}; | ||
auto wallet{std::make_shared<CWallet>(chain.get(), "", std::move(database))}; | ||
wallet->LoadWallet(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could
BOOST_CHECK_EQUAL(wallet->LoadWallet(), DBErrors::::LOAD_OK);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK_EQUAL(wallet->LoadWallet(), DBErrors::::LOAD_OK);
Thanks, added this check
src/wallet/test/wallet_tests.cpp
Outdated
for (DatabaseFormat format : DATABASE_FORMATS) { | ||
const std::string name{strprintf("receive-requests-%i", format)}; | ||
TestLoadWallet(name, format, [](std::shared_ptr<CWallet> wallet) EXCLUSIVE_LOCKS_REQUIRED(wallet->cs_wallet) { | ||
BOOST_CHECK(!wallet->m_address_book[PKHash()].previously_spent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above, would be good to not access m_address_book
directly from outside of the wallet class.
The map [] operator usage in a random place of the sources could easily screw things up.
Could change it to
BOOST_CHECK(!wallet->IsAddressUsed(PKHash()));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK(!wallet->IsAddressUsed(PKHash()));
Thanks applied your suggested change (furszy@2195b1e)
src/wallet/test/wallet_tests.cpp
Outdated
BOOST_CHECK(wallet->m_address_book[PKHash()].previously_spent); | ||
BOOST_CHECK(wallet->m_address_book[ScriptHash()].previously_spent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BOOST_CHECK(wallet->IsAddressUsed(PKHash()));
BOOST_CHECK(wallet->IsAddressUsed(ScriptHash()));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK(wallet->IsAddressUsed(PKHash())); BOOST_CHECK(wallet->IsAddressUsed(ScriptHash()));
Thanks applied your suggested change (furszy@2195b1e)
src/wallet/test/wallet_tests.cpp
Outdated
BOOST_CHECK(!wallet->m_address_book[PKHash()].previously_spent); | ||
BOOST_CHECK(!wallet->m_address_book[ScriptHash()].previously_spent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BOOST_CHECK(!wallet->IsAddressUsed(PKHash()));
BOOST_CHECK(!wallet->IsAddressUsed(ScriptHash()));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK(!wallet->IsAddressUsed(PKHash())); BOOST_CHECK(!wallet->IsAddressUsed(ScriptHash()));
Thanks applied your suggested change (furszy@2195b1e)
@furszy, thanks for the thorough review. I will start implementing your suggestions, but do you think you could help me with your comments about encapsulation like #27224 (comment)? The comments are just a little vague from my perspective so it'd be great if you could provide specific code suggestions and I will apply or implement them. For background, I've been a little detached from the address encapsulation attempt, and personally like m_address_book.find(dest) / m_addressbook[dest] lookups and for loops a little better than calling FindAddress/FindOrInsertAddress/ForAddresses wrapper functions. But I don't have a strong opinion and am happy to use wrappers, I just think you would be better at suggesting with names / parameters since you have been designing the API and know more about the goals. |
Yeah ok. The main purpose of address book encapsulation is prevent people from accidentally modifying the ‘purpose’ field in an RPC command or the GUI, or from creating a new entry by misusing the map's [] operator. Which it could lead, in the worst-case scenario, to a "send" address being relabeled as a "receive" address, resulting in a user sending funds to an address they do not own while thinking they do. I know that it didn’t happen so far but.. it’s a big project and I think that we shouldn’t assume that everyone touching the GUI and/or the RPC commands are aware of the internal address book usage (e.g. #25979). Simpler to just forget about this possible issue by letting people access entries through an interface. Also, the encapsulation will let us use another mutex for the address book, so commands like
Sadly, the cleanest implementation would come after #24914, when we load the wallet in-order, first read the address book entry, then “dest data” etc. About the API naming, we still have a mixture of names there. Then, in the future, to not allow calling this But anyway, future stuff. Another idea: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated 0ad6a53 -> d343235 (pr/nodest.18
-> pr/nodest.19
, compare) applying most of the suggested changes. Thanks for the reviews!
src/wallet/bdb.h
Outdated
@@ -205,6 +206,7 @@ class BerkeleyBatch : public DatabaseBatch | |||
bool WriteKey(DataStream&& key, DataStream&& value, bool overwrite = true) override; | |||
bool EraseKey(DataStream&& key) override; | |||
bool HasKey(DataStream&& key) override; | |||
bool ErasePrefix(Span<std::byte> prefix) override; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: https://github.com/bitcoin/bitcoin/pull/27224/files#r1150232826
you have a reason to prefer
Span<std::byte>
?
Well it should actually take a span of const bytes, so I fixed this to add const.
Passing a span makes more sense than passing a datastream because the function just needs a read-only array of bytes, it doesn't need a read/write vector or a stream object or anything more complicated.
The other functions are using CDataStream just because they are called by older code which might be counting on them to call memory_cleanse, and would require extra work to convert to spans, though it was tried previously in #19320
src/wallet/test/wallet_tests.cpp
Outdated
std::vector<bilingual_str> warnings; | ||
auto database{MakeWalletDatabase(name, options, status, error)}; | ||
auto wallet{std::make_shared<CWallet>(chain.get(), "", std::move(database))}; | ||
wallet->LoadWallet(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK_EQUAL(wallet->LoadWallet(), DBErrors::::LOAD_OK);
Thanks, added this check
src/wallet/walletdb.cpp
Outdated
CAddressBookData& data{pwallet->m_address_book[DecodeDestination(strAddress)]}; | ||
if (strKey.compare("used") == 0) { | ||
data.previously_spent = true; | ||
} else if (strKey.compare(0, 2, "rr") == 0) { | ||
data.receive_requests[strKey.substr(2)] = std::move(strValue); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Would be good to keep the encapsulation and don't access
m_address_book
from outside of the wallet class (otherwise we are reverting #25337 and, future, #26836).
Thanks, applied your change (furszy@2195b1e)
src/wallet/test/wallet_tests.cpp
Outdated
BOOST_CHECK(wallet->m_address_book[PKHash()].previously_spent); | ||
BOOST_CHECK(wallet->m_address_book[ScriptHash()].previously_spent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK(wallet->IsAddressUsed(PKHash())); BOOST_CHECK(wallet->IsAddressUsed(ScriptHash()));
Thanks applied your suggested change (furszy@2195b1e)
src/wallet/test/wallet_tests.cpp
Outdated
BOOST_CHECK(!wallet->m_address_book[PKHash()].previously_spent); | ||
BOOST_CHECK(!wallet->m_address_book[ScriptHash()].previously_spent); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
BOOST_CHECK(!wallet->IsAddressUsed(PKHash())); BOOST_CHECK(!wallet->IsAddressUsed(ScriptHash()));
Thanks applied your suggested change (furszy@2195b1e)
src/wallet/wallet.cpp
Outdated
@@ -2439,11 +2439,7 @@ bool CWallet::DelAddressBook(const CTxDestination& address) | |||
return false; | |||
} | |||
// Delete destdata tuples associated with address |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No longer the case. Same for the comment that is above this one.
Thanks, I basically just replaced "destdata" with "address data" in these comments, but you can let me know if the issue was something else
src/wallet/wallet.cpp
Outdated
@@ -2821,65 +2817,46 @@ unsigned int CWallet::ComputeTimeSmart(const CWalletTx& wtx, bool rescanning_old | |||
|
|||
bool CWallet::SetAddressUsed(WalletBatch& batch, const CTxDestination& dest, bool used) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
Sure, I think it would make sense to clear this flag when a transaction is abandoned. I think this could be a separate issue, and it should be pretty easy to implement anytime IMO
src/wallet/wallet.cpp
Outdated
void CWallet::LoadDestData(const CTxDestination &dest, const std::string &key, const std::string &value) | ||
{ | ||
m_address_book[dest].destdata.insert(std::make_pair(key, value)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
This method was removed from the cpp but not from the header.
Thanks applied your suggested change (furszy@2195b1e)
This new function is not used yet this commit, but next commit adds usages and test coverage for both BDB and sqlite.
This is cleanup that doesn't change external behavior. - Removes awkward `StringMap` intermediate representation - Simplifies CWallet code, deals with used address and received request serialization in walletdb.cpp - Adds test coverage and documentation - Reduces memory usage This PR doesn't change externally observable behavior. Internally, only change in behavior is that EraseDestData deletes directly from database because the `StringMap` is gone. This is more direct and efficient because it uses a single btree lookup and scan instead of multiple lookups Motivation for this cleanup is making changes like bitcoin#18550, bitcoin#18192, bitcoin#13756 easier to reason about and less likely to result in unintended behavior and bugs Co-authored-by: furszy <matiasfurszyfer@protonmail.com>
Rebased d343235 -> a5986e8 ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code ACK a5986e8
bool CWallet::EraseAddressReceiveRequest(WalletBatch& batch, const CTxDestination& dest, const std::string& id) | ||
{ | ||
if (!batch.EraseAddressReceiveRequest(dest, id)) return false; | ||
m_address_book[dest].receive_requests.erase(id); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to check that the map contains the element prior accessing it. Otherwise the GUI could be adding a new address book entry by mistake.
ACK a5986e8 |
/** | ||
* Map containing data about previously generated receive requests | ||
* requesting funds to be sent to this address. Only present for IsMine | ||
* addresses. Map keys are decimal numbers uniquely identifying each |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was momentarily confused by this because (IIUC) map keys are the int64_t id
field of RecentRequestEntry
but actually encoded into std::string
by a few methods in RecentRequestsTableModel
. So I'm wondering: Is there a reason we need to keep those IDs as strings? Is it like, at this point, going back to int64 would be a needless refactor?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #27224 (comment)
So I'm wondering: Is there a reason we need to keep those IDs as strings? Is it like, at this point, going back to int64 would be a needless refactor?
Only reason that the CAddressBookData::receive_requests
map containing the requests has string keys and string values is that the setAddressReceiveRequest
API adding the requests takes string keys and string values.
I think it is best for the map types to match the API types, so wallet code can store the values it is passed without parsing them. But definitely the setAddressReceiveRequest
API could be improved, and there was discussion about this in #27224 (comment). Not only could the string keys be replaced by integer keys, but it might be possible to drop the keys and the map entirely if the GUI doesn't need to store multiple requests for a single address. I'm not sure how valuable this refactoring would be, but it should be easier to do now that receive requests are stored separately from other address data.
EDIT: I was curious what replacing strings with ints would look like so I implemented it in commit 2877c80 (branch). I think the cleanup probably isn't that valuable standalone, but it might be useful as part of a larger change.
a5986e8 refactor: Remove CAddressBookData::destdata (Ryan Ofsky) 5938ad0 wallet: Add DatabaseBatch::ErasePrefix method (Ryan Ofsky) Pull request description: This is cleanup that doesn't change external behavior. Benefits of the cleanup are: - Removes awkward `StringMap` intermediate representation for wallet address metadata. - Simplifies `CWallet`, deals with used address and received request serialization in `walletdb.cpp` instead of higher level wallet code - Adds test coverage and documentation This PR doesn't change externally observable behavior. Internally, the only change in behavior is that `EraseDestData` deletes rows directly from the database because they are no longer stored in memory. This is more direct and efficient because it uses a single lookup and scan instead of multiple lookups. Motivation for this cleanup is making changes like bitcoin#18550, bitcoin#18192, bitcoin#13756 easier to reason about and less likely to result in unintended behavior and bugs --- This PR is a rebased copy of bitcoin#18608. For some reason that PR is locked and couldn't be reopened or commented on. This PR is an alternative to bitcoin#27215 with differences described in bitcoin#27215 (review) ACKs for top commit: achow101: ACK a5986e8 furszy: Code ACK a5986e8 Tree-SHA512: 6bd3e402f1f60263fbd433760bdc29d04175ddaf8307207c4a67d59f6cffa732e176ba57886e02926f7a1615dce0ed9cda36c8cbc6430aa8e5b56934c23f3fe7
This is cleanup that doesn't change external behavior. Benefits of the cleanup are:
StringMap
intermediate representation for wallet address metadata.CWallet
, deals with used address and received request serialization inwalletdb.cpp
instead of higher level wallet codeThis PR doesn't change externally observable behavior. Internally, the only change in behavior is that
EraseDestData
deletes rows directly from the database because they are no longer stored in memory. This is more direct and efficient because it uses a single lookup and scan instead of multiple lookups.Motivation for this cleanup is making changes like #18550, #18192, #13756 easier to reason about and less likely to result in unintended behavior and bugs
This PR is a rebased copy of #18608. For some reason that PR is locked and couldn't be reopened or commented on.
This PR is an alternative to #27215 with differences described in #27215 (review)