-
Notifications
You must be signed in to change notification settings - Fork 37.7k
serialization: c++20 endian/byteswap/clz modernization #29263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
6d28845
to
60be777
Compare
Here's a godbolt test that shows how various compilers perform: https://gcc.godbolt.org/z/nTadqEP83 To test with/without builtins, comment/un-comment the From my tests:
|
60be777
to
4221e05
Compare
Added a quick note about |
Concept ACK - I think moving to using the builtins is ok. With the eventual plan to migrate to @aureleoules everytime I visit the coverage/benchmarks for this PR, I get a 500 error. Can you take a look? |
@fanquake thanks for reaching out, I fixed the issue. |
@aureleoules Thanks! These numbers really don't make much sense to me and don't match what I've seen locally, but I'll work on trying to reproduce. |
From some more local tests, it looks like it's the clz changes slowing things down for whatever reason. Still investigating. |
4221e05
to
2c1a73e
Compare
I've dropped the clz changes as a test and kept only the endian/byteswap change. Locally on my machine the benchmarks look the same before and after. If the corecheck benchmarks look better I'll update the title/description here. Edit: looks good. |
Aha, I finally tracked down the culprit! No wonder the benchmarks weren't making sense! The problem was the removal of this from #if defined(HAVE_CONFIG_H)
#include <config/bitcoin-config.h>
#endif Turns out, I'm going to put the clz changes back to match the PR title/description, but this time with the include fixed up. Hopefully after that the benchmarks will make sense and all will be good :) I suppose there might be other compilation units that were depending on the include indirectly as well. I'll figure out a way to check for that. |
2c1a73e
to
6c84885
Compare
6c84885
to
7ffc688
Compare
Converted to a draft while I'm still messing with this. I added a new commit to add the The benchmarks still show some regressions, though it's not nearly as bad as it was before. Will continue poking. |
7ffc688
to
cfbf466
Compare
I've created #26972 to track ideas on how to detect those silent build issues, or at least make them easier to spot. |
I've pushed a commit which temporarily puts the includes back in the low-level headers for the sake of addressing the benchmarks first, setting aside the possibility of missing defines. @aureleoules's benchmarks show 2 small regressions, though I am unable to reproduce those locally. My tests show:
This PR:
Essentially no change. I'm curious if @maflcko sees the same? |
@theuni yeah the You can see ignored benchmarks here in the meantime: https://github.com/corecheck/frontend/blob/master/src/routes/%5Bowner%5D/%5Brepo%5D/pulls/%5Bnumber%5D/Benchmarks.svelte#L334 |
@aureleoules Thanks, that's helpful. |
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
THis change mirrors changes from bitcoin#29263.
This change mirrors changes from bitcoin#29263.
8e17f00 build, msvc: Cleanup `bitcoin_config.h.in` (Hennadii Stepanov) Pull request description: This PR mirrors changes from #29263 into the MSVC build system. ACKs for top commit: fanquake: ACK 8e17f00 Tree-SHA512: b8e5cca015ff112c2969a60436524e97007ff2c559b3c12425d0549af694b16248311cc3e7c33f798bc095a679933641496836bb846eee6a2a377956ef53f56e
This change mirrors changes from bitcoin#29263.
This change mirrors changes from bitcoin#29263.
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
33a454e fixup! cmake: Check system symbols (Hennadii Stepanov) 3cb2e65 fixup! cmake: Check system headers (Hennadii Stepanov) Pull request description: This PR backports build system changes from bitcoin#29263. ACKs for top commit: pablomartin4btc: ACK 33a454e Tree-SHA512: 1793c6504a7190134c0ce075e959d22c4a3640d54a4d141f5117975bed267952cc8c7da488426e48022eba1eb77d3353783d77a20907b0cfa183e0b68d824133
This change mirrors changes from bitcoin/bitcoin#29263.
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263 Add negative test case for OP_LE64TOSCRIPTNUM Use OP_EQUAL rather than OP_EQUALVERIFY Fix rebase
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263 Add negative test case for OP_LE64TOSCRIPTNUM Use OP_EQUAL rather than OP_EQUALVERIFY Fix rebase
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263 Fix tests after rebase
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263 Add negative test case for OP_LE64TOSCRIPTNUM Use OP_EQUAL rather than OP_EQUALVERIFY Fix rebase
…ESSTHANOREQUAL64, OP_GREATERTHAN64, OP_GREATERTHANOREQUAL64, OP_SCRIPTNUMTOLE64, OP_LE64TOSCRIPTNUM, OP_LE32TOLE64 Remove liquid args WIP Get simple OP_1 functional test case working Get tests for arithmetic and comparison opcodes working Get all functional tests passing Rename test case to Arithmetic64bitTest Rename file to feature_64bit_arithmetic_opcodes.py, add it to test_runner.py Get tests passing in feature_taproot.py Remove unused push_le4 Revert test fixture setup Cleanup Fix linting test: Add leaf_version parameter to taproot_tree_helper() Fix bug Fix bugs Fix compile Fix missing sigversion checks Fix htole64 -> htole64_internal due to bitcoin#29263
, bitcoin#26159, bitcoin#26489, bitcoin#26825, bitcoin#28127, bitcoin#29275, bitcoin#29263, partial bitcoin#27790, bitcoin#27978 (c++20 backports) 8b14cd6 fix: add header to `test/util/wallet.cpp` to define `ENABLE_WALLET` (Kittywhiskers Van Gogh) 0c69fb6 merge bitcoin#29263: c++20 endian/byteswap/clz modernization (Kittywhiskers Van Gogh) 26c1553 merge bitcoin#29275: Fix prevector iterator concept issues (Kittywhiskers Van Gogh) bda57aa merge bitcoin#28127: Remove C-style const-violating cast, Use reinterpret_cast (Kittywhiskers Van Gogh) 48ed8c0 partial bitcoin#27978: Drop unsafe AsBytePtr function (Kittywhiskers Van Gogh) 7185aff partial bitcoin#27790: Add PrefixCursor (Kittywhiskers Van Gogh) 2b2869b merge bitcoin#26825: remove already tested headers from AC_CHECK_HEADERS (Kittywhiskers Van Gogh) e9d23b3 merge bitcoin#26489: Split overly large util_tests.cpp file (Kittywhiskers Van Gogh) b682aea merge bitcoin#26159: Remove `stdlib.h` from header checks (Kittywhiskers Van Gogh) 3c2c6da merge bitcoin#26150: remove stdio.h from header checks (Kittywhiskers Van Gogh) 043bc04 merge bitcoin#26135: remove strings.h from header checks (Kittywhiskers Van Gogh) 97f3c6d merge bitcoin#24989: rename BytePtr to AsBytePtr (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * Developers are required to rerun `autogen.sh` if switching between this PR and `develop` due to changes in `configure.ac`, failing to do so will result in a myriad of compiler errors (see below) <details> <summary>Build errors:</summary> ``` In file included from /usr/include/x86_64-linux-gnu/sys/types.h:176, from /usr/include/stdlib.h:514, from /usr/include/c++/13/cstdlib:79, from /usr/include/c++/13/ext/string_conversions.h:43, from /usr/include/c++/13/bits/basic_string.h:4109, from /usr/include/c++/13/string:54, from ./blockfilter.h:9, from blockfilter.cpp:8: ./compat/endian.h:156:17: error: redefinition of 'uint16_t __bswap_16(uint16_t)' 156 | inline uint16_t htobe16(uint16_t host_16bits) | ^~~~~~~ In file included from /usr/include/endian.h:35: /usr/include/x86_64-linux-gnu/bits/byteswap.h:34:1: note: '__uint16_t __bswap_16(__uint16_t)' previously defined here 34 | __bswap_16 (__uint16_t __bsx) | ^~~~~~~~~~ ./compat/endian.h:163:17: error: redefinition of 'uint16_t __uint16_identity(uint16_t)' 163 | inline uint16_t htole16(uint16_t host_16bits) | ^~~~~~~ In file included from /usr/include/endian.h:36: /usr/include/x86_64-linux-gnu/bits/uintn-identity.h:33:1: note: '__uint16_t __uint16_identity(__uint16_t)' previously defined here 33 | __uint16_identity (__uint16_t __x) | ^~~~~~~~~~~~~~~~~ ``` </details> ## Breaking Changes None expected. ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 8b14cd6 PastaPastaPasta: ACK 8b14cd6; reviewed compiled locally Tree-SHA512: 4f0ad19006df965aba68e28227bbffb340e7bd578594970bc295737929bdb6e9f0782c8fa5b0e699bc8df3b3861d655783275456976ef0dacc8c85e62c522aa5
This replaces #28674, #29036, and #29057. Now ready for testing and review.
Replaces platform-specific endian and byteswap functions. This is especially useful for kernel, as it means that our deep serialization code no longer requires bitcoin-config.h.
I apologize for the size of the last commit, but it's hard to avoid making those changes at once.
All platforms now use our internal functions rather than libc or platform-specific ones, with the exception of MSVC.
Sadly, benchmarking showed that not all compilers are capable of detecting and optimizing byteswap functions, so compiler builtins are instead used where possible. However, they're now detected via macros rather than autoconf checks.
This matches how libc++ implements std::byteswap for c++23.
I suggest we move/rename
compat/endian.h
, but I left that out of this PR to avoid bikeshedding.#29057 pointed out some irregularities in benchmarks. After messing with various compilers and configs for a few weeks with these changes, I'm of the opinion that we can't win on every platform every time, so we should take the code that makes sense going forward. That said, if any real-world slowdowns are caused here, we should obviously investigate.