-
Notifications
You must be signed in to change notification settings - Fork 37.7k
[wallet] Don't shut down after encrypting the wallet #11678
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested ACK on the RPC, I'm having a look into the GUI hang at the moment, it successfully stops, closes and opens the wallets and then hangs which I find weird
@@ -2386,8 +2387,10 @@ UniValue encryptwallet(const JSONRPCRequest& request) | |||
// BDB seems to have a bad habit of writing old data into | |||
// slack space in .dat files; that is bad if the old data is | |||
// unencrypted private keys. So: | |||
StartShutdown(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Above on line 2347 it says "Note that this will shutdown the server.\n"
which should be removed now
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
db7a0c9
to
5145625
Compare
I resolved the hanging issue, but now there's another issue. The hanging issue was because I forgot to set the CWallet pointers
|
5145625
to
e47b890
Compare
Resolved the other issue too. I forgot to also reset the signals to use the new wallets too. No longer WIP. |
Concept ACK. I get
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good to fix whatever bugs prevent qt from being able to deal with opened and closed wallets, but it doesn't seem right that every wallet should be closed and reopened when one wallet is encrypted. I don't even see a reason why the wallet that is being encrypted needs have all it's in-memory state thrown away and then the same data reloaded from disk.
If the problem is just BDB "writing old data", it seems like we could add a CDBEnv::Reopen() method that would just close and reopen the DbEnv* and Db* pointers in the CDBEnv object. This might be a little tricky to implement (it would probably need to add a condition variable and wait for mapFileUseCount counters to go to 0, see CDB::PeriodicFlush), but I think it would add very little code and be faster and less buggy than unloading and reloading all kinds of things that don't need to be reloaded.
This was easier to do since there are some things in the GUI that look specifically for
I thought that there were still some in-memory things that needed to be discarded (e.g. unencrypted keys) to ensure security of the wallet. |
Concept ACK.
|
Needs rebase |
e47b890
to
2da96f0
Compare
Rebased. |
2da96f0
to
e6bb6bb
Compare
Rebased this. The issue that @promag pointed out is because the GUI does not reset the walletmodel as it needs to when reloading the wallet when the rpc console is used to give the command. I'm not quite sure what the best approach for enabling that is though. |
Insteading of shutting down the entire software after encrypting a wallet, instead just stop and close all of the wallets and then reopen them. This will flush the wallets, clear them from memory, and then reopen them to ensure that no unencrypted keys remain after encrypting.
e6bb6bb
to
7cb328d
Compare
Rebased. I believe I fixed the issue mentioned earlier, although it is an extremely ugly hack that involves string comparisons. If anyone has a better suggestion, I'm all ears. |
Will test. |
9f69b53
to
ca09b2b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NACK the current implementation. Models are instanced with the given wallet and should not be changed, instead new models should be instanced.
My suggestion is to build on top of #10740.
Closing this so I can implement it the right way. |
I think before anything else, we should figure out exactly what needs to be cleared. For example, is there information in the DBEnv that needs to be cleared? If so, opening and closing the wallet will not clear it and a shutdown will still be needed (or the database improvements I suggested above). |
@ryanofsky The problem is described here: https://bitcointalk.org/index.php?topic=51474.0 I think the approach I was using works since it closes the wallets, resets the database environment, and then reopens the wallets. Specifically, I think the key part is the I wrote and used this script to check if there were any private keys being written to the log files: https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d |
@achow101, then what were you referring to when you wrote "I thought that there were still some in-memory things that needed to be discarded"? I'd be interested to know why encryption can't be handled at the db/wallet layer instead of involving qt, rpc, hooks, callbacks, shared pointers or whatever else might be entailed in a new effort to "implement it the right way." |
I was mistaken.
The database environment needs to be reloaded, and AFAIK, you can't do this from within an individual wallet because reloading the database environment requires closing the databases of all wallets and reopening them. So it has to be done externally instead of within the db/wallet itself. |
Would adding a method to close and reopen the database environment not work? This is what I was trying to suggest here: #11678 (review). I don't see what the problem would be. It would seem to require some interaction with the flush thread but otherwise could just be implemented at just the lowest layers and leave qt, rpc, and higher level wallet code untouched. |
I'm not sure that that would work. The I can try that approach and see if I get anywhere with it. |
@ryanofsky I think it actually is possible to do that. I'll try it and make a PR if it works. It appears that I have misunderstood how the wallet handles Db stuff. |
Sounds good. I think you probably already found this, but the CDB objects are temporary, so you could just wait for them to be closed with |
…ting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for #11678 which implements @ryanofsky's [suggestion](#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
…of shutting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
…of shutting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
…of shutting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
…of shutting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
…of shutting down c1dde3a No longer shutdown after encrypting the wallet (Andrew Chow) d7637c5 After encrypting the wallet, reload the database environment (Andrew Chow) 5d296ac Add function to close all Db's and reload the databae environment (Andrew Chow) a769461 Move BerkeleyEnvironment deletion from internal method to callsite (Andrew Chow) Pull request description: This is the replacement for bitcoin#11678 which implements @ryanofsky's [suggestion](bitcoin#11678 (review)). Shutting down the software was to prevent the BDB environment from writing unencrypted private keys to disk in the database log files, as was noted [here](https://bitcointalk.org/index.php?topic=51474.msg616068#msg616068). This PR replaces the shutdown behavior with a CDBEnv flush, close, and reopen which achieves the same effect: everything is cleanly flushed and closed, the log files are removed, and then the environment reopened to continue normal operation. To ensure that no unencrypted private keys are in the log files after encrypting the wallet, I wrote [this script](https://gist.github.com/achow101/7f7143e6c3d3fdc034d3470e72823e9d) to pull private keys from the original wallet file and searches for these keys in the log files (note that you will have to change your file paths to make it work on your own machine). As for concerns about private keys being written to slack space or being kept in memory, these behaviors no longer exist after the original wallet encryption PR and the shutting down solution from 2011. cc @ryanofsky Tree-SHA512: 34b894283b0677a873d06dee46dff8424dec85a2973009ac9b84bcf3d22d05f227c494168c395219d9aee3178e420cf70d4b3eeacc9785aa86b6015d25758e75
Instead of shutting down the entire software after encrypting a wallet, instead just stop and close all of the wallets and then reopen them. This will flush the wallets, clear them from memory, and then reopen them to ensure that no unencrypted keys remain after encrypting.
This is marked as WIP because there are a few bugs that I am still trying to figure out. ~~~is a locking bug that I'm still trying to figure out. After encrypting a wallet from the GUI, the GUI freezes. This appears to be a lock contention issue with
cs_KeyStore
at wallet/crypter.cpp:160. However I can't figure out why this is happening.~~~