forked from bitcoin/bitcoin
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
backport: partial bitcoin#27920, #28799, #29007 - to improve CI experience #6240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r spkm descriptor ID BACKPORT NOTICE It includes only this commit: 97a965d refactor: extract descriptor ID calculation from spkm GetID() This allows us to verify the descriptor ID on the descriptors unit tests in different software versions without requiring to use the entire DescriptorScriptPubKeyMan machinery. Note: The unit test changes are introduced after the bugfix commit but this commit + the unit test commit can be cherry-picked on top of the v25 branch to verify IDs correctness. IDs must be the same for v25 and after the bugfix commit.
…eated descriptor string creation BACKPORT NOTICE: It doesn't include changes related to wallet_miniscript.py 5e6bc6d test: remove custom rpc timeout for `wallet_miniscript.py`, reorder in test_runner (Sebastian Falbesoner) f811a24 wallet: cache descriptor ID to avoid repeated descriptor string creation (Sebastian Falbesoner) Pull request description: Right now a wallet descriptor is converted to its string representation (via `Descriptor::ToString`) repeatedly at different instances: - on finding a `DescriptorScriptPubKeyMan` for a given descriptor (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the `importdescriptors` RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`) - whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in `TopUp` or any instances where a descriptor is written to the DB to determine the database key, also at less obvious places like `FastWalletRescanFilter` etc. As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to `WalletDescriptor` and calculate it immediately at initialization (or deserialization). `HasWalletDescriptor` is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that. This speeds up the functional test `wallet_miniscript.py` by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance. Fixes bitcoin#28800. ACKs for top commit: Sjors: ACK 5e6bc6d S3RK: Code Review ACK 5e6bc6d achow101: ACK 5e6bc6d BrandonOdiwuor: ACK 5e6bc6d Tree-SHA512: 98b43963a5dde6055bb26cecd3b878dadd837d6226af4c84142383310495da80b3c4bd552e73b9107f2f2ff1c11f5e18060c6fd3d9e44bbd5224114c4d245c1c
…e functional tests BACKPORT NOTICE It includes only this commit: be25ac3 [init] Remove -addrmantest command line arg -addrmantest is only used in `p2p_node_network_limited.py` test to test if the node self-advertises a hard-coded local address (which wouldn't be advertised in the tests because it's unroutable without the test-only code path) to check pruning-related services are correct in that addr. Remove -addrmantest because the self advertisement happens because of hard coded test path logic, and expected services are nominal due to how easily the test-only code could diverge from mainnet logic. It's also being used only in 1 test.
5 tasks
PastaPastaPasta
approved these changes
Aug 29, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 2eadcf2
UdjinM6
approved these changes
Aug 30, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK 2eadcf2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
Lately our CI for tsan is flapping many functional tests and take long times.
This PR has several important changes backported from the latest bitcoin's version to improve CI experience
What was done?
This PR has several backports that improved CI experience drastically.
Firstly, it aims to fix flapping test p2p_node_network_limited.py
For example: https://gitlab.com/dashpay/dash/-/jobs/7692635307
p2p_node_network_limited.py | ✖ Failed | 28 s
Secondly, it improves performance of Descriptor wallets significantly for case of
tsan
CI. It is tiny improvement for Release build and local runs, but some fucnctional tests run as fast as twice:https://gitlab.com/dashpay/dash/-/jobs/7694458953
https://gitlab.com/dashpay/dash/-/jobs/7665132625
See performance investigation here: #6226
How Has This Been Tested?
Run unit/functional tests
Breaking Changes
N/A
Checklist: