Skip to content

Conversation

TheCharlatan
Copy link
Contributor

@TheCharlatan TheCharlatan commented Dec 20, 2023

The script/bitcoinconsensus module defines the public interface for the bitcoinconsensus library. Even though this module is only required by the tests and the bitcoinconsensus library, it is currently compiled into the static internal libbitcoin_consensus library, and therefore used by a bunch of build targets that do not require it.

Since it is always part of the internal library, the HAVE_CONSENSUS_LIB define used by the tests is essentially
meaningless, since the module is always available. This can be verified on master by removing the HAVE_CONSENSUS_LIB checks in the tests, configuring with --with-libs=no, and running make check.

Improve all of this by including the bitcoinconsensus module only where it is required and removing the HAVE_CONSENSUS_LIB define.

An alternative to this patch was discussed in hebasto#41: Actually linking the test binaries with the external consensus library if it is available. This has the disadvantage though that if the dynamic consensus library is built, it has to be available for the test binaries to load whenever and wherever they are being run.

The `script/bitcoinconsensus` module defines the public interface for
the `bitcoinconsensus` library. Even though this module is only required
by the tests and the `bitcoinconsensus` library, it is currently
compiled into the static internal `libbitcoin_consensus` library, and
therefore used by a bunch of build targets that do not require it.

Since it is always part of the internal library, the
`HAVE_CONSENSUS_LIB` define used by the tests is essentially
meaningless, since the module is always available. This can be verified
on master by removing the `HAVE_CONSENSUS_LIB` checks in the tests,
configuring with `--with-libs=no`, and running `make check`.
@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 20, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK hebasto, vasild
Concept ACK theuni

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #29265 (Add SigVersion parameter to IsOpSuccess() by Christewart)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@TheCharlatan
Copy link
Contributor Author

cc @hebasto @theuni

@theuni
Copy link
Member

theuni commented Dec 20, 2023

Concept ACK.

I think another way of looking at this is that script/bitcoinconsensus.cpp is moving out into its own convenience lib, it just happens to be a single file. We could imagine if the api were written in multiple translation units, we'd actually want to build that convenience lib. But the question is: absent that, is it worth the trouble?

Honestly I don't really care because it's kinda funky either way. Curious to see if anyone else has a preference.

@hebasto
Copy link
Member

hebasto commented Dec 21, 2023

Since it is always part of the internal library, the HAVE_CONSENSUS_LIB define used by the tests is essentially
meaningless, since the module is always available.

It seems worth noting that we do not use the HAVE_CONSENSUS_LIB macro in /src/test/fuzz/script_bitcoin_consensus.cpp in the master branch.

@hebasto
Copy link
Member

hebasto commented Dec 21, 2023

Concept ACK on moving the script/bitcoinconsensus.cpp out from the internal libbitcoin_consensus.a library.

@theuni

I think another way of looking at this is that script/bitcoinconsensus.cpp is moving out into its own convenience lib, it just happens to be a single file. We could imagine if the api were written in multiple translation units, we'd actually want to build that convenience lib.

That is very similar to what I suggested in #24994.

But the question is: absent that, is it worth the trouble?

For example, to fix the symbol visibility issue.

@theuni
Copy link
Member

theuni commented Dec 21, 2023

@hebasto #24994 counted on linker side-effects to fix things IIRC. As do your other suggestions as far as I can see. If there's an issue with a compiler/stdlib symbol visibility, let's track it down and get the issues reported upstream as opposed to working around them with brittle workarounds.

@hebasto
Copy link
Member

hebasto commented Dec 21, 2023

@theuni

If there's an issue with a compiler/stdlib symbol visibility, let's track it down and get the issues reported upstream as opposed to working around them with brittle workarounds.

#26698 (comment):

The underlying problem is described in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=36022, and it was not considered as a bug.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK b37a171.

@DrahtBot DrahtBot requested a review from theuni December 21, 2023 17:42
Copy link
Contributor

@vasild vasild left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK b37a171

@DrahtBot
Copy link
Contributor

Guix builds (on x86_64)

File commit 4b1196a
(master)
commit 162a169
(master and this pull)
SHA256SUMS.part 7aefd803a584a152... c9c88507e2b1c29c...
*-aarch64-linux-gnu-debug.tar.gz 735f38fb37b239f8... e8751fcba6a3857d...
*-aarch64-linux-gnu.tar.gz 83a257c74e184b96... 524f9c0663e78d16...
*-arm-linux-gnueabihf-debug.tar.gz 513acf435b18fab2... ddab696a8c6d3861...
*-arm-linux-gnueabihf.tar.gz 581f911fcca5ae02... 777f8c934210246c...
*-arm64-apple-darwin-unsigned.tar.gz 7a3f2b4e01f27896... d2a1614bc2186e5a...
*-arm64-apple-darwin-unsigned.zip dff45b014236e990... 58709693ef349088...
*-arm64-apple-darwin.tar.gz 0d9696b7bd76b36f... d8d72316859b7f5d...
*-powerpc64-linux-gnu-debug.tar.gz dfe6cfdcd6ad3c6f... a6cbbf09fc81e90f...
*-powerpc64-linux-gnu.tar.gz 54c013201daad1ff... ab8d50fd98bb7b35...
*-powerpc64le-linux-gnu-debug.tar.gz 8e944b002273b678... 20f4572fe366c3a7...
*-powerpc64le-linux-gnu.tar.gz 1ba8199984d7c43a... df39769bd8b934e2...
*-riscv64-linux-gnu-debug.tar.gz e9abe48aeb7acc68... 16da8f0afde2a02e...
*-riscv64-linux-gnu.tar.gz f4d1484dec65b967... 8f6d556eda7ee5fc...
*-x86_64-apple-darwin-unsigned.tar.gz 915650aa35e6e322... 9d5b2c3aafeefc35...
*-x86_64-apple-darwin-unsigned.zip 6a6139cb415de743... 0e8e2ec7ab7ecceb...
*-x86_64-apple-darwin.tar.gz c479d099d1df9a85... f20ed8546f6a72cf...
*-x86_64-linux-gnu-debug.tar.gz 9e27161179f18a4a... 5585b7fdf4aef067...
*-x86_64-linux-gnu.tar.gz f69632ca3dc98896... 3e5c37e6b2def850...
*.tar.gz c04997c9f9e51d80... 1dde913fa303d6e4...
guix_build.log e5ef1439ef10a87a... 052d056473d09f7b...
guix_build.log.diff 2de91436d107e8a2...

theuni added a commit to theuni/bitcoin that referenced this pull request Jan 5, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessatating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto#41
bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v26. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
theuni added a commit to theuni/bitcoin that referenced this pull request Jan 5, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto#41
bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v26. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
theuni added a commit to theuni/bitcoin that referenced this pull request Jan 5, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto#41
bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v27. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
@fanquake
Copy link
Member

Maybe draft for now, while we hash out #29189?

@TheCharlatan TheCharlatan marked this pull request as draft January 12, 2024 15:09
theuni added a commit to theuni/bitcoin that referenced this pull request Jan 30, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto#41
bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v27. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
theuni added a commit to theuni/bitcoin that referenced this pull request Jan 30, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto#41
bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v27. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
fanquake added a commit that referenced this pull request Feb 1, 2024
25dc87e libconsensus: deprecate (Cory Fields)

Pull request description:

  This library has existed for nearly 10 years with very little known uptake or impact. It has become a maintenance burden. In several cases it dictates our code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as well as build-system procedures (building multiple copies of object files especially for the lib).

  Several discussions have arisen wrt migrating it to CMake and it has become difficult to justify adding more complexity for a library that is virtually unused anyway.

  See for example the discussions:
  hebasto#41
  #29123

  And here: #29180
  Where it is pointed out that the libbitcoinconsensus functions are slower than those the internal bitcoind equivalents due to the missing sha2 implementations.

  Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not migrating it to CMake and letting it end with v27. Any remaining use-cases could be handled in the future by libbitcoinkernel.

  If there are any users currently using libbitcoinconsensus, please chime in with your use-case!

  Edit: Corrected final release to be v27.

ACKs for top commit:
  TheCharlatan:
    ACK 25dc87e
  fanquake:
    ACK 25dc87e - this library has very little, if any impactful real world usage. It has been entirely broken (on various platforms) for long periods of its existence, where nobody even noticed. Pruning this out to save porting, and starting anew with the kernel, is the sane thing to do.

Tree-SHA512: baff2b3c4f76f520c96021035f751fdcb51bedf00e767660249e92a7bc7c5c176786bcf2c4cfe2d2351c200f932b39eb886bcfb22fbec824a41617590d6a1638
@fanquake
Copy link
Member

fanquake commented Feb 1, 2024

Could turn this PR into a full removal for 28.x? Otherwise I guess close, and we can followup with that in future?

@TheCharlatan
Copy link
Contributor Author

Seems cleaner to close and do the removal in a separate one. Might be good to open a PR for the full removal soon, so it can get enough review before branch-off.

janus pushed a commit to BitgesellOfficial/bitgesell that referenced this pull request Apr 6, 2024
This library has existed for nearly 10 years with very little known uptake or
impact. It has become a maintenance burden. In several cases it dictates our
code/library structure (for example necessitating LIBBITCOIN_CRYPTO_BASE), as
well as build-system procedures (building multiple copies of object files
especially for the lib).

Several discussions have arisen wrt migrating it to CMake and it has become
difficult to justify adding more complexity for a library that is virtually
unused anyway.

See for example the discussions:
hebasto/bitcoin#41
bitcoin/bitcoin#29123

Instead, we (fanquake, hebasto, TheCharlatan, and I) propose simply not
migrating it to CMake and letting it end with v27. Any remaining use-cases
could be handled in the future by libbitcoinkernel.
@bitcoin bitcoin locked and limited conversation to collaborators Jan 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants