Skip to content

Conversation

sipa
Copy link
Member

@sipa sipa commented Apr 11, 2023

There is no strict need for any of the changes in v0.3.1 (compared to the v0.3.0 that's currently subtreed) for Bitcoin Core release builds, but if anyone may compile Bitcoin Core from source using Clang v14+, this will prevent known timing leaks in the signing/keygen logic.

This also includes a CI fix from libsecp256k1 master (on top of 0.3.1) which fixes Wycheproof test vector generation.

I also had to amend some of the linters to avoid enforcing their rules on the .py files in the secp256k1 subtree.

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 11, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK real-or-random, fanquake

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #23432 (BIP324: CKey encode/decode to elligator-swift by dhruv)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@real-or-random
Copy link
Contributor

Build failure on CI here. This is during make distdir.

 (cd secp256k1 && make  top_distdir=../../bitcoin-i686-pc-linux-gnu distdir=../../bitcoin-i686-pc-linux-gnu/src/secp256k1 \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[4]: Entering directory '/tmp/cirrus-ci-build/ci/scratch/build/src/secp256k1'
make  distdir-am
make[5]: Entering directory '/tmp/cirrus-ci-build/ci/scratch/build/src/secp256k1'
python3 tools/tests_wycheproof_generate.py /tmp/cirrus-ci-build/src/secp256k1/src/wycheproof/ecdsa_secp256k1_sha256_bitcoin_test.json > src/wycheproof/ecdsa_secp256k1_sha256_bitcoin_test.h
/bin/dash: 1: cannot create src/wycheproof/ecdsa_secp256k1_sha256_bitcoin_test.h: Directory nonexistent

https://cirrus-ci.com/task/4665749793406976?logs=ci#L986

So, apparently the wycheproof directory doesn't exist. That's why (re)creating ecdsa_secp256k1_sha256_bitcoin_test.h fails.

The correct behavior would be that the wycheproof dir just exists (including ecdsa_secp256k1_sha256_bitcoin_test.h), so it wouldn't need to be created in the first place.

I have no idea why this fails. make distdir works for me locally, even in a out-of-tree build.

@real-or-random
Copy link
Contributor

Okay, I think I understand what's going on here:

  1. make distdir is supposed to create a directory with all files to be distributed
  2. the distdir target depends on all files to be distributed
  3. these file include src/wycheproof/secdsa_secp256k1_sha256_bitcoin_test.h because it's listed in EXTRA_DIST (perhaps noinst_HEADERS would be more appropriate, but the result is the same)
  4. src/wycheproof/secdsa_secp256k1_sha256_bitcoin_test.h has a dependency on src/wycheproof/secdsa_secp256k1_sha256_bitcoin_test.json
  5. if the .json file in the src dir appears to be newer than the .h file in the build dir (e.g., due to meaningless timestamps after copying the source dir), make distdir tries to rebuild the .h in the build dir
  6. but this fails because not even the src/wycheproof subdirectory exists in the build dir (in a out-of-tree/VPATH build)

Puh.

Item 4 is a bug in libsecp256k1. While this dependency exists in fact, we don't want to list in the Makefile because the .h file is checked in the repo and only meant to be rebuilt by developers/maintainers of the library. (We similarly omit explicit dependencies for other generated source files.) @sipa My suggestion above fixes exactly this. Can you to commit this here to see if it makes the problem disappear, just for a quick and dirty check of CI? If that works, we can create a proper fix in libsecp256k1 and update this PR accordingly.

Item 7 is another bug in libsecp256k1, which is however only relevant to developers of the library. (Well, and users like Gentoo who insist on rebuilding everything from scratch in an out-of-tree build, see bitcoin-core/secp256k1#1160.) The fix should simply be another mkdir -p call.

@achow101 achow101 added this to the 25.0 milestone Apr 13, 2023
real-or-random added a commit to real-or-random/secp256k1 that referenced this pull request Apr 14, 2023
Pregenerated files that we distribute should not have dependencies
in Makefile.am. For rationale, see the comments about the precomputed
table files.

See also bitcoin/bitcoin#27445 (comment) .
real-or-random added a commit to real-or-random/secp256k1 that referenced this pull request Apr 14, 2023
@RandomLattice
Copy link

Okay, I think I understand what's going on here

Great investigation, thanks! And sorry this broke the build.

I see you already put hotfixes in bitcoin-core/secp256k1#1276 for the autotools issue. Thanks. I can put a hotfix for the linting issue + open an issue for a longer-term fix for linting.

real-or-random added a commit to bitcoin-core/secp256k1 that referenced this pull request Apr 14, 2023
06c67de autotools: Don't regenerate Wycheproof header automatically (Tim Ruffing)

Pull request description:

  This is a hot fix for bitcoin/bitcoin#27445 .

  ---

  Pregenerated files that we distribute should not have dependencies in Makefile.am. For rationale, see the comments about the precomputed table files.

  See also bitcoin/bitcoin#27445 (comment) .

ACKs for top commit:
  hebasto:
    ACK 06c67de
  RandomLattice:
    ACK 06c67de

Tree-SHA512: fa7f44eaa1c7e42ecba5829ac1b8ae8b5826d1a1551e01c3caf37af780bd5c102c8f54e88520723937f7016d93c67b62a334c7a28b96c4f422a38fcf8e6a1984
@real-or-random
Copy link
Contributor

@sipa bitcoin-core/secp256k1#1276 has been merged, can you update this PR to include the current secp256k1 master?

sipa added 3 commits April 14, 2023 10:35
4258c54 Merge bitcoin-core/secp256k1#1276: autotools: Don't regenerate Wycheproof header automatically
06c67de autotools: Don't regenerate Wycheproof header automatically
3bab71c Merge bitcoin-core/secp256k1#1268: release cleanup: bump version after 0.3.1
656c6ea release cleanup: bump version after 0.3.1
346a053 Merge bitcoin-core/secp256k1#1269: changelog: Fix link
6a37b2a changelog: Fix link
ec98fce Merge bitcoin-core/secp256k1#1266: release: Prepare for 0.3.1
898e1c6 release: Prepare for 0.3.1
1d9a13f changelog: Remove inconsistent newlines
0e09166 changelog: Catch up in preparation of 0.3.1
7b7503d Merge bitcoin-core/secp256k1#1245: tests: Add Wycheproof ECDSA vectors
145078c Merge bitcoin-core/secp256k1#1118: Add x-only ecmult_const version with x specified as n/d
e5de454 tests: Add Wycheproof ECDSA vectors
0f86420 Add exhaustive tests for ecmult_const_xonly
4485926 Add x-only ecmult_const version for x=n/d
a0f4644 Merge bitcoin-core/secp256k1#1252: Make position of * in pointer declarations in include/ consistent
4e68262 Merge bitcoin-core/secp256k1#1226: Add CMake instructions to release process
2d51a45 Merge bitcoin-core/secp256k1#1257: ct: Use volatile "trick" in all fe/scalar cmov implementations
4a496a3 ct: Use volatile "trick" in all fe/scalar cmov implementations
3d1f430 Make position of * in pointer declarations in include/ consistent
2bca0a5 Merge bitcoin-core/secp256k1#1241: build: Improve `SECP_TRY_APPEND_DEFAULT_CFLAGS` macro
afd8b23 Merge bitcoin-core/secp256k1#1244: Suppress `-Wunused-parameter` when building for coverage analysis
1d8f367 Merge bitcoin-core/secp256k1#1250: No need to subtract 1 before doing a right shift
3e43041 No need to subtract 1 before doing a right shift
3addb4c build: Improve `SECP_TRY_APPEND_DEFAULT_CFLAGS` macro
0c07c82 Add CMake instructions to release process
464a911 Merge bitcoin-core/secp256k1#1242: Set ARM ASM symbol visibility to `hidden`
f16a709 Merge bitcoin-core/secp256k1#1247: Apply Checks only in VERIFY mode.
70be3ca Merge bitcoin-core/secp256k1#1246: Typo
4ebd828 Apply Checks only in VERIFY mode.
d1e7ca1 Typo
5bb03c2 Replace `SECP256K1_ECMULT_TABLE_VERIFY` macro by a function
9c8c4f4 Merge bitcoin-core/secp256k1#1238: build: bump CMake minimum requirement to 3.13
0cf2fb9 Merge bitcoin-core/secp256k1#1243: build: Ensure no optimization when building for coverage analysis
fd2a408 Set ARM ASM symbol visibility to `hidden`
4429a8c Suppress `-Wunused-parameter` when building for coverage analysis
8e79c7e build: Ensure no optimization when building for coverage analysis
96dd062 build: bump CMake minimum requirement to 3.13
427bc3c Merge bitcoin-core/secp256k1#1236: Update comment for secp256k1_modinv32_inv256
647f0a5 Update comment for secp256k1_modinv32_inv256
5658209 Merge bitcoin-core/secp256k1#1228: release cleanup: bump version after 0.3.0
28e63f7 release cleanup: bump version after 0.3.0

git-subtree-dir: src/secp256k1
git-subtree-split: 4258c54
@sipa
Copy link
Member Author

sipa commented Apr 14, 2023

Updated, and also improved the linter logic to decide which files to lint.

FILES_ARGS = ['git', 'ls-files', 'test/functional/*.py', 'contrib/devtools/*.py']

# All .py files, except those in src/ (to exclude subtrees there)
FLAKE_FILES_ARGS = ['git', 'ls-files', '*.py', ':!:src/*.py']
Copy link
Member Author

@sipa sipa Apr 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for reviewers: this should just be excluding the following files:

  • src/crc32c/.ycm_extra_conf.py
  • src/minisketch/tests/pyminisketch.py
  • src/secp256k1/tools/tests_wycheproof_generate.py

I believe that's the correct course of action, as even if the projects supplying those files would want to comply with Bitcoin Core's linting standards, it shouldn't be up to Bitcoin Core to enforce that.

Copy link
Contributor

@real-or-random real-or-random left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 621c178 subtree matches. diff to linter script looks good

@fanquake
Copy link
Member

cc @jonasnick too

@fanquake
Copy link
Member

Guix Build:

a750ae4390cbc5386cd263097da9470c1437d798379fe3f5f10a5086e48fc6fb  guix-build-621c17869d37/output/aarch64-linux-gnu/SHA256SUMS.part
1458929f13ef9d52ebca28948d83f7e1ed9bd2e992fe0334482cdaf3fa32aab6  guix-build-621c17869d37/output/aarch64-linux-gnu/bitcoin-621c17869d37-aarch64-linux-gnu-debug.tar.gz
74458df1b50213bff9231ed87f0d80f7fe2ab2e6b83d1a959a28b7f1297339b4  guix-build-621c17869d37/output/aarch64-linux-gnu/bitcoin-621c17869d37-aarch64-linux-gnu.tar.gz
1ed57572087d7b09003aa1fc9c541c0d637be517622a03c69531c82bf9218196  guix-build-621c17869d37/output/arm-linux-gnueabihf/SHA256SUMS.part
5b535acac7fb477e3891963d08d8eb7c53a613ea9673c8e94f9846a06502b455  guix-build-621c17869d37/output/arm-linux-gnueabihf/bitcoin-621c17869d37-arm-linux-gnueabihf-debug.tar.gz
e6096969efb61d0777c41f65ca82e0fbddfdbad0a2021e337838d13ee1bb0ced  guix-build-621c17869d37/output/arm-linux-gnueabihf/bitcoin-621c17869d37-arm-linux-gnueabihf.tar.gz
0dd1cad5f18a053f7db1e46ed74274d58eb5a06031096823847737ea6225c040  guix-build-621c17869d37/output/arm64-apple-darwin/SHA256SUMS.part
c44f98a1ad6c866d5d20ebbdc0a827c532212740c210a68ba6ffbbfe03d05ff2  guix-build-621c17869d37/output/arm64-apple-darwin/bitcoin-621c17869d37-arm64-apple-darwin-unsigned.dmg
d4cbf66fb6950570e52edfc6df2b80284e99dcc49e43c1b14cbbb0c3d75f1512  guix-build-621c17869d37/output/arm64-apple-darwin/bitcoin-621c17869d37-arm64-apple-darwin-unsigned.tar.gz
133bef091207b07b724aa3c28b4b470823fd005c84c5bb27ebda0cab8c79f23f  guix-build-621c17869d37/output/arm64-apple-darwin/bitcoin-621c17869d37-arm64-apple-darwin.tar.gz
13607ade159c952f2731e283c9a6c1761604a85ac25c81c2f3329d162e846591  guix-build-621c17869d37/output/dist-archive/bitcoin-621c17869d37.tar.gz
546ba0fab2be1793199ba2421fdcb01c046830d6a2ff015294491c87697cd566  guix-build-621c17869d37/output/powerpc64-linux-gnu/SHA256SUMS.part
88bbf9156c4000a087f4e692338195255c5a707a04030a2aee2542083fd819db  guix-build-621c17869d37/output/powerpc64-linux-gnu/bitcoin-621c17869d37-powerpc64-linux-gnu-debug.tar.gz
631f01957a64ca3d690f996a956453e324aa847ce15c53b6067ed68fc81e8131  guix-build-621c17869d37/output/powerpc64-linux-gnu/bitcoin-621c17869d37-powerpc64-linux-gnu.tar.gz
d82f2c33c2852efd77b35df821a41dc2c7125ad1e3b5a5c7b3d5407bf6de50fc  guix-build-621c17869d37/output/powerpc64le-linux-gnu/SHA256SUMS.part
c5d9b108f93c5551acd607e0f12c2b9ec9e4ed6aad71ac7d325497d6e1495118  guix-build-621c17869d37/output/powerpc64le-linux-gnu/bitcoin-621c17869d37-powerpc64le-linux-gnu-debug.tar.gz
4d0b503f9b7db8d7342d3b69b4bff27d151674db7ac697796afab22e33699ba3  guix-build-621c17869d37/output/powerpc64le-linux-gnu/bitcoin-621c17869d37-powerpc64le-linux-gnu.tar.gz
7380f0a47d68d6c9bdeb61ac60724eb178083828a8d08e945f737165f2c26ac8  guix-build-621c17869d37/output/riscv64-linux-gnu/SHA256SUMS.part
6c3cc75978ed504b839a471334f43b116e040af6096b9c1f89c4318afe7f0c5f  guix-build-621c17869d37/output/riscv64-linux-gnu/bitcoin-621c17869d37-riscv64-linux-gnu-debug.tar.gz
832744857f34affaf78cb823155b9cdded71be3796fa53143ddaca9cae85c2c6  guix-build-621c17869d37/output/riscv64-linux-gnu/bitcoin-621c17869d37-riscv64-linux-gnu.tar.gz
9897e69fa8aed38f49550b6015711968be12826590f5289c85daeb25f526c09a  guix-build-621c17869d37/output/x86_64-apple-darwin/SHA256SUMS.part
9dee78bd51b4cc1b3b3cb17f5bebd6f8ffc2d51f6f88e3cde989c14eb5a20da9  guix-build-621c17869d37/output/x86_64-apple-darwin/bitcoin-621c17869d37-x86_64-apple-darwin-unsigned.dmg
a613b69fe28596f8bf53197c4dd9abfd28967ad2e7d409ca5da151d008504675  guix-build-621c17869d37/output/x86_64-apple-darwin/bitcoin-621c17869d37-x86_64-apple-darwin-unsigned.tar.gz
f94ef742c25463270aaaf2715ce6cd8fa11309c8e0114cd5cbcf275c95687f57  guix-build-621c17869d37/output/x86_64-apple-darwin/bitcoin-621c17869d37-x86_64-apple-darwin.tar.gz
954163b38343400282d61e2ceaf6ef0802fbab2e0671a898a5bbd21401519c35  guix-build-621c17869d37/output/x86_64-linux-gnu/SHA256SUMS.part
9c6653a0a4d1fad8acdfc21a52f82f00ba320a9b508a15e52319722990c958c3  guix-build-621c17869d37/output/x86_64-linux-gnu/bitcoin-621c17869d37-x86_64-linux-gnu-debug.tar.gz
08a5d67947eece9af4c4e1aef4089474e6f3c94169e895b52b0ec2e2c9c09e91  guix-build-621c17869d37/output/x86_64-linux-gnu/bitcoin-621c17869d37-x86_64-linux-gnu.tar.gz
bce081ba263d970d968c74bd6e87e2ba46a08a3624fbc3c1331580343af7fed7  guix-build-621c17869d37/output/x86_64-w64-mingw32/SHA256SUMS.part
a38af023d3b6682db933910f535136a5ddab9d7ced513382b239423a24132791  guix-build-621c17869d37/output/x86_64-w64-mingw32/bitcoin-621c17869d37-win64-debug.zip
3b2dcc19bb75ff990eab621707ecd02e7145b709e47d57805df4e0a4ee8035a2  guix-build-621c17869d37/output/x86_64-w64-mingw32/bitcoin-621c17869d37-win64-setup-unsigned.exe
00a133efd3c588e66271d67b842b3aa38eb59882c166fca1610a638dce6e9758  guix-build-621c17869d37/output/x86_64-w64-mingw32/bitcoin-621c17869d37-win64-unsigned.tar.gz
b0a8ede2f34d5d45a63e942e4322b594413fb7989789cdb6b93c3efd7e9f491c  guix-build-621c17869d37/output/x86_64-w64-mingw32/bitcoin-621c17869d37-win64.zip

Copy link
Member

@fanquake fanquake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 621c178

@fanquake fanquake merged commit 3650e74 into bitcoin:master Apr 15, 2023
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 17, 2023
real-or-random added a commit to real-or-random/secp256k1 that referenced this pull request Apr 19, 2023
real-or-random added a commit to real-or-random/secp256k1 that referenced this pull request Apr 19, 2023
real-or-random added a commit to real-or-random/secp256k1 that referenced this pull request Apr 25, 2023
dderjoel pushed a commit to dderjoel/secp256k1 that referenced this pull request May 23, 2023
Pregenerated files that we distribute should not have dependencies
in Makefile.am. For rationale, see the comments about the precomputed
table files.

See also bitcoin/bitcoin#27445 (comment) .
dderjoel pushed a commit to dderjoel/secp256k1 that referenced this pull request May 23, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 8, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 9, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 9, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 24, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 28, 2023
kwvg added a commit to kwvg/dash that referenced this pull request Sep 28, 2023
UdjinM6 pushed a commit to kwvg/dash that referenced this pull request Nov 15, 2023
UdjinM6 pushed a commit to kwvg/dash that referenced this pull request Nov 16, 2023
UdjinM6 pushed a commit to kwvg/dash that referenced this pull request Nov 20, 2023
PastaPastaPasta pushed a commit to kwvg/dash that referenced this pull request Nov 21, 2023
PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Nov 21, 2023
@bitcoin bitcoin locked and limited conversation to collaborators Apr 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants