Skip to content

Conversation

fanquake
Copy link
Member

@fanquake fanquake commented Apr 26, 2021

This switches us to using a LLD based toolchain for macOS builds.

Benefits

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 26, 2021

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK theuni
Concept ACK hebasto, michaelfolkson

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #30022 (releases: use LLVM 18 for macOS by fanquake)
  • #29881 (guix: use GCC 13 to builds releases by fanquake)
  • #29072 (build: use -no_exported_symbols on macOS by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@ryanofsky
Copy link
Contributor

ryanofsky commented Apr 26, 2021

This definitely seems like a good thing to add support for, but it seems sad to have to drop support for the open source compiler at the same time as adding support for the closed source one. I don't know enough about this to have an informed opinion, though.

EDIT: Just a misunderstanding

@fanquake
Copy link
Member Author

but it seems sad to have to drop support for the open source compiler at the same time as adding support for the closed source one.

I might be misunderstanding your comment, but the opposite is happening here. We're migrating away from using Apples, irregularly open sourced tools (cctools, ld64), to just vanilla LLVM (lld), which is entirely open source.

@ryanofsky
Copy link
Contributor

I might be misunderstanding your comment, but the opposite is happening here. We're migrating away from using Apples, irregularly open sourced tools (cctools, ld64), to just vanilla LLVM (lld), which is entirely open source.

My bad! I thought this was switching from an unsupported apple open-source toolchain to an supported apple closed source toolchain, not to a vanilla toolchain. My blinding hate of apple has kept me from seeing reality! Thanks for the clarification. This all seems very good.

@theuni
Copy link
Member

theuni commented Apr 26, 2021

Very nice! Strong concept ACK, but obviously we can't use this until all features we need are supported. But it's great to see that linking finally works!

For a little extra context beyond what @fanquake said above, the macOS clang+cctools build has always been a wonky outlier because we have to build our own (very unsupported) toolchain first. LD64 is apple's libre/open-source linker, and there is a community project that maintains a build environment for cross builds.

For a few years, there has been slow development on a darwin functionality for llvm's lld, but it's only really picked up speed and become usable in the last year or two. This PR would bring our Apple builds much more inline with others in that it would finally be using a supported libre/open-source compiler/linker combo.

@fanquake Is cctools still needed at all? LLVM ships bitcode equivalents of most binutils tools (ar, nm, ranlib, etc), I wonder if we can use those instead. Maybe they require lto?

@fanquake
Copy link
Member Author

Is cctools still needed at all? LLVM ships bitcode equivalents of most binutils tools (ar, nm, ranlib, etc), I wonder if we can use those instead. Maybe they require lto?

@theuni no cctools is no longer required. I drop the native_cctools as part of this PR and we just use the LLVM tools.

@theuni
Copy link
Member

theuni commented Apr 28, 2021

@theuni no cctools is no longer required. I drop the native_cctools as part of this PR and we just use the LLVM tools.

Apologies for reading too quickly, I see now that I only looked at the first commit. Turns out the 3rd one is the one I expected :)

@fanquake
Copy link
Member Author

fanquake commented May 1, 2021

Rebased for #21793, and rebased on top of #19817. Made some additional changes, such as renaming the native_cctools package to native_llvm, and removed some now-redundant info from the macdeploy documentation.

Note: I'm pretty sure the use of ZERO_AR_DATE can now be dropped entirely, but have left that out for now.

fanquake added a commit to fanquake/bitcoin that referenced this pull request Jun 13, 2024
luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 13, 2024
I need to try and patch around Qts buildsystem to keep bitcoin#21778 moving
along (the issue being that even when you tell Qt to build using
Clang on Linux, it still calls out to GCC, breaking our ability to have
a macOS release build env that doesn't have a GCC toolchain installed,
and thus no ld binary).

Before trying to patch Qt any further, update to the latest LTS
release, and update the current patch set.

Github-Pull: bitcoin#29732
Rebased-From: 430f319
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jun 14, 2024
fanquake added a commit that referenced this pull request Jun 18, 2024
…or `mmacos-version-min`

7c298fe doc: rewrite some of the macdeploy docs (fanquake)
d042230 depends: swap mmacosx-version-min for mmacos-version-min (fanquake)

Pull request description:

  Whilst `-mmacosx-version-min` and `-mmacos-version-min` remain aliases for each other, the later is preferred,
  and I assume the former will be removed at some point in the future; see: llvm/llvm-project#95374.

  Somewhat of a followup to #21778. Rewrite some of the mac deploy docs.

ACKs for top commit:
  theuni:
    utACK 7c298fe
  TheCharlatan:
    ACK 7c298fe
  hebasto:
    ACK 7c298fe.

Tree-SHA512: 6493f087fde93e0eec319af0e105d163b3f047d8a03f7d4b0d6cd7c64b58d0a978b7d67c6b8dba5c6ccf8b10e188aab5dc98eec400b0546dc9ee801a689b4332
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 23, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compiler another
GCC toolchain.
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 23, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compiler another
GCC toolchain.
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 23, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 24, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jul 25, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
fanquake added a commit that referenced this pull request Jul 25, 2024
d1592d2 guix: use gcc-12 to compile winpthreads (fanquake)
b23690e guix: use GCC 12.4.0 over 12.3.0 (fanquake)
8b41ede guix: consolidate back to GCC 12 toolchain for all HOSTS (fanquake)

Pull request description:

  This PR contains 3 changes:

  * Bump GCC in Guix from [12.3.0 to 12.4.0](https://gcc.gnu.org/gcc-12/). A patch was sent upstream, https://lists.gnu.org/archive/html/guix-patches/2024-06/msg01025.html, but has not landed.
  * Consolidate all build environments back to using a GCC 12 toolchain. After #21778, the macOS environment is no-longer pinned to 11 (12 would otherwise cause issues building cctools). So, instead of requiring all builders to compile an additional GCC toolchain, use 12.
  * Use GCC 12 to compile winpthreads. Currently, GCC 11 is used; which became apparent in #30452 (comment).

ACKs for top commit:
  TheCharlatan:
    ACK d1592d2
  hebasto:
    ACK d1592d2.

Tree-SHA512: e3aa1fa3e69500c93180e07cb4684661247ec6bc45245f746538d81406ff1d8777131590307496dda3287a112b6633e4991168586ca4c2036fa3a57b1efa9c87
davidgumberg pushed a commit to davidgumberg/bitcoin that referenced this pull request Jul 29, 2024
Using GCC 11 for the macOS build hasn't been required since bitcoin#21778, and
at this point, given a toolchain is still needed (bitcoin#30206), it makes more
sense to (re-)use 12, rather than make all builders compile another
GCC toolchain.
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 24, 2024
4fe5f3c depends: remove redundant stdlib option (Cory Fields)

Pull request description:

  Like bitcoin#27628, this is another dependency of bitcoin#21778, though it doesn't become obvious until used with a newer clang.

  This should be a no-op.

  Use of -stdlib++-isystem gets rid of any system c++ header include paths and negates the need for this option. In newer versions of clangs the combo produces an annoying warning that actually causes problems during configure.

ACKs for top commit:
  fanquake:
    ACK 4fe5f3c

Tree-SHA512: 904072f2b13dffbbeab2bc9ff20a74969888502fa1ea35d9030784dd397c6751e31233e6ec7dc34e9fd42574950be733b25d6653c2a93e214cc5e4eef2e0133a
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 24, 2024
67aacc7 build: cleanup comments after adding yet another libtool hack (Cory Fields)
283d955 build: Fix shared lib linking for darwin with lld (Cory Fields)

Pull request description:

  Solves one of the last remaining blockers for bitcoin#21778. Fixes lld linking shared libs for macos via libtool.

  lld fails one of libtool's earliest checks [because it happens to output a warning that contains a specific string](https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4#n999):

  >     # If there is a non-empty error log, and "single_module"
  >     # appears in it, assume the flag caused a linker warning

  And here is the test being run:
  > x86_64-apple-darwin-ld: warning: Option `-single_module' is deprecated in ld64:
  > x86_64-apple-darwin-ld: warning: Unnecessary option: this is already the default

  Because the warning is printed the test fails. So libtool falls back to a very primitive and broken link-line for shared libs.

  Arguably this should be worked-around in upstream lld by changing the warning string, as otherwise every libtool project will fail to link with it.

  Like many other libtool hacks, the solution is to simply disable the check and hard-code the answer we know to be correct.

ACKs for top commit:
  hebasto:
    re-ACK 67aacc7

Tree-SHA512: 792e4d208a3a4921edb5f267f43ecd052b5b650df0db5cb2788ee1e4f3c4087413f354b22e407ff5fa2f99a22a16154ec6826d14c6654a57c00aae3b3e744bca
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 24, 2024
4fe5f3c depends: remove redundant stdlib option (Cory Fields)

Pull request description:

  Like bitcoin#27628, this is another dependency of bitcoin#21778, though it doesn't become obvious until used with a newer clang.

  This should be a no-op.

  Use of -stdlib++-isystem gets rid of any system c++ header include paths and negates the need for this option. In newer versions of clangs the combo produces an annoying warning that actually causes problems during configure.

ACKs for top commit:
  fanquake:
    ACK 4fe5f3c

Tree-SHA512: 904072f2b13dffbbeab2bc9ff20a74969888502fa1ea35d9030784dd397c6751e31233e6ec7dc34e9fd42574950be733b25d6653c2a93e214cc5e4eef2e0133a
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 25, 2024
4fe5f3c depends: remove redundant stdlib option (Cory Fields)

Pull request description:

  Like bitcoin#27628, this is another dependency of bitcoin#21778, though it doesn't become obvious until used with a newer clang.

  This should be a no-op.

  Use of -stdlib++-isystem gets rid of any system c++ header include paths and negates the need for this option. In newer versions of clangs the combo produces an annoying warning that actually causes problems during configure.

ACKs for top commit:
  fanquake:
    ACK 4fe5f3c

Tree-SHA512: 904072f2b13dffbbeab2bc9ff20a74969888502fa1ea35d9030784dd397c6751e31233e6ec7dc34e9fd42574950be733b25d6653c2a93e214cc5e4eef2e0133a
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 25, 2024
67aacc7 build: cleanup comments after adding yet another libtool hack (Cory Fields)
283d955 build: Fix shared lib linking for darwin with lld (Cory Fields)

Pull request description:

  Solves one of the last remaining blockers for bitcoin#21778. Fixes lld linking shared libs for macos via libtool.

  lld fails one of libtool's earliest checks [because it happens to output a warning that contains a specific string](https://git.savannah.gnu.org/cgit/libtool.git/tree/m4/libtool.m4#n999):

  >     # If there is a non-empty error log, and "single_module"
  >     # appears in it, assume the flag caused a linker warning

  And here is the test being run:
  > x86_64-apple-darwin-ld: warning: Option `-single_module' is deprecated in ld64:
  > x86_64-apple-darwin-ld: warning: Unnecessary option: this is already the default

  Because the warning is printed the test fails. So libtool falls back to a very primitive and broken link-line for shared libs.

  Arguably this should be worked-around in upstream lld by changing the warning string, as otherwise every libtool project will fail to link with it.

  Like many other libtool hacks, the solution is to simply disable the check and hard-code the answer we know to be correct.

ACKs for top commit:
  hebasto:
    re-ACK 67aacc7

Tree-SHA512: 792e4d208a3a4921edb5f267f43ecd052b5b650df0db5cb2788ee1e4f3c4087413f354b22e407ff5fa2f99a22a16154ec6826d14c6654a57c00aae3b3e744bca
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 25, 2024
019ad73 depends: set RANLIB for CMake (fanquake)
43cfb42 depends: set NM for CMake (fanquake)
1e4412b depends: set AR for CMake (fanquake)

Pull request description:

  Needed for bitcoin#21778. Should be more correct in any case.

ACKs for top commit:
  theuni:
    utACK 019ad73. I didn't test, but I tried this approach on a few deps and it seemed to work as expected.
  TheCharlatan:
    ACK 019ad73

Tree-SHA512: 78cc8981456f7476cafca0e40fcc569e474b92004c8024d1c4268b6aab53175074a06ab17ebded8d706bf0a7f77401642dd38bb7ce2e4b04abdcd149d3d69969
PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Nov 10, 2024
, bitcoin#26470, bitcoin#27296, bitcoin#27179, bitcoin#27813, bitcoin#27811, bitcoin#28069, bitcoin#28294, bitcoin#28324, bitcoin#28328, bitcoin#29987 (guix backports: part 4)

1edd121 merge bitcoin#29987: build with glibc 2.31 (Kittywhiskers Van Gogh)
0949825 revert: add exception for 32-bit ARM builds which need glibc 2.28 (Kittywhiskers Van Gogh)
0ac2531 merge bitcoin#28328: update time-machine (Kittywhiskers Van Gogh)
ef9c4bd merge bitcoin#28324: pre time-machine bump changes (Linux) (Kittywhiskers Van Gogh)
b45a127 merge bitcoin#28294: pre time-machine bump changes (Windows) (Kittywhiskers Van Gogh)
ed1f7fe merge bitcoin#28069: Remove librt usage from release binaries (Kittywhiskers Van Gogh)
5d51aa9 merge bitcoin#27811: Clean up manifest (Kittywhiskers Van Gogh)
d439e46 merge bitcoin#27813: Update `python-lief` package to 0.13.2 (Kittywhiskers Van Gogh)
70e6283 merge bitcoin#27179: use osslsigncode 2.5 (Kittywhiskers Van Gogh)
3799509 merge bitcoin#27296: import/sync python-lief (0.12.3) package definition from upstream (Kittywhiskers Van Gogh)
ac8bd5a refactor: move lief definitions to expected location (Kittywhiskers Van Gogh)
eb0ae08 merge bitcoin#26470: Clean up `libexec/build.sh` (Kittywhiskers Van Gogh)
d3d7a05 merge bitcoin#24031: don't compress macOS DMG (Kittywhiskers Van Gogh)
6a54603 merge bitcoin#27670: remove redundant glibc patches (Kittywhiskers Van Gogh)
0c988f0 merge bitcoin#21089: Add support for powerpc64{,le} (Kittywhiskers Van Gogh)
808d215 fix: set correct locale in guix ci container (UdjinM6)

Pull request description:

  ## Motivation

  In preparation for migrating to C++20 and bumping our minimum required compiler to ensure that we have access to greater portions of the C++20 spec, this pull request first of two pull requests aiming to upgrade our Guix setup to use GCC 12 (and GCC 11 for the macOS toolchain).

  Upgrades to the Clang toolchain for macOS and moving to GCC 12 for the macOS toolchain will be addressed in a separate pull request as the latter requires [bitcoin#21778](bitcoin#21778), which is well outside the scope of this PR.

  ## Additional Notes

  * Dependency for #6383

  * Newer versions of GCC will not behave as expected with `test-security-check.py` as the stack protector (listed as `Canary` in the test) now behaves as expected and no longer fails. This causes an error when running it as the test expects failure (see below). This isn't an issue upstream as they backported [bitcoin#29987](bitcoin#29987), which gets rid of those tests ([source](bitcoin@b5fc6d4#diff-52aa0cda44721f089e53b128cb1232a876006ef257b211655456b17dfb2ec712)).

    Therefore, we have backported that pull request as well.

    <details>

    <summary>Build failure:</summary>

    ```
    ======================================================================
    FAIL: test_ELF (__main__.TestSecurityChecks)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/distsrc-base/distsrc-22.0.0-beta.1-70-g7907fab39c17-x86_64-linux-gnu/./contrib/devtools/test-security-check.py", line 61, in test_ELF
        self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-zexecstack','-fno-stack-protector','-Wl,-znorelro','-no-pie','-fno-PIE', '-Wl,-z,separate-code']),
    AssertionError: Tuples differ: (1, 'test1: failed PIE NX RELRO CONTROL_FLOW') != (1, 'test1: failed PIE NX RELRO Canary CONTROL_FLOW')

    First differing element 1:
    'test1: failed PIE NX RELRO CONTROL_FLOW'
    'test1: failed PIE NX RELRO Canary CONTROL_FLOW'

    - (1, 'test1: failed PIE NX RELRO CONTROL_FLOW')
    + (1, 'test1: failed PIE NX RELRO Canary CONTROL_FLOW')
    ?                                +++++++

    ----------------------------------------------------------------------
    Ran 1 test in 0.126s

    FAILED (failures=1)
    ```

    </details>

    * The backport has the effect of bumping the target glibc version to 2.31, which as the release notes say, cuts off support for RHEL 8 and Ubuntu 18.04 LTS (`bionic`) (i.e. our Guix binaries won't run on these distros anymore, users running those distros will have to compile it themselves).

    This shouldn't be a problem as full support for RHEL 8 ended May 31, 2024 ([source](https://access.redhat.com/support/policy/updates/errata#Life_Cycle_Dates)) and standard support for `bionic` ended April 2023 ([source](https://ubuntu.com/about/release-cycle)).

    **Dash Core will still work on Ubuntu 20.04 LTS (`focal`) as it ships with glibc 2.31 ([source](https://packages.ubuntu.com/focal/glibc-doc))**

  ## Breaking Changes

  None expected

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    utACK 1edd121

Tree-SHA512: 194a35a72ede5fb5488cdc41eb15320df7a5fa1fb8815c7342dd15b3983efff417fb1a525fdf497a98ede91958d0a5ed18f3e5f0f27c5cbc5dc25eb58116dbd5
Fabcien added a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2024
Summary:
```
This switches us to using a LLD based toolchain for macOS builds.
Benefits

 - Less complicated macOS toolchain.
 - No longer beholden to Apple releasing it's source for cctools, ld64 & libtapi.
 - No more reliance on third parties to modify those sources for us. i.e apple-libtapi, cctools-port (cctools + ld64).
```

Backport of [[bitcoin/bitcoin#21778 | core#21778]].

Note this needs some adaptation or our cmake build, as well as some update in build.sh to work around a cmake bug with llvm-strip.
The Cross build for OSX will be broken on CI (because it lacks the proper clang version). I paused the build for now and will unbreak in a follow-up.

Test Plan: Run the guix osx build.

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D17261
Fabcien added a commit to Bitcoin-ABC/secp256k1 that referenced this pull request Dec 1, 2024
Summary:
```
This switches us to using a LLD based toolchain for macOS builds.
Benefits

 - Less complicated macOS toolchain.
 - No longer beholden to Apple releasing it's source for cctools, ld64 & libtapi.
 - No more reliance on third parties to modify those sources for us. i.e apple-libtapi, cctools-port (cctools + ld64).
```

Backport of [[bitcoin/bitcoin#21778 | core#21778]].

Note this needs some adaptation or our cmake build, as well as some update in build.sh to work around a cmake bug with llvm-strip.
The Cross build for OSX will be broken on CI (because it lacks the proper clang version). I paused the build for now and will unbreak in a follow-up.

Test Plan: Run the guix osx build.

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D17261
roqqit pushed a commit to doged-io/doged that referenced this pull request Dec 19, 2024
Summary:
```
This switches us to using a LLD based toolchain for macOS builds.
Benefits

 - Less complicated macOS toolchain.
 - No longer beholden to Apple releasing it's source for cctools, ld64 & libtapi.
 - No more reliance on third parties to modify those sources for us. i.e apple-libtapi, cctools-port (cctools + ld64).
```

Backport of [[bitcoin/bitcoin#21778 | core#21778]].

Note this needs some adaptation or our cmake build, as well as some update in build.sh to work around a cmake bug with llvm-strip.
The Cross build for OSX will be broken on CI (because it lacks the proper clang version). I paused the build for now and will unbreak in a follow-up.

Test Plan: Run the guix osx build.

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D17261
kwvg added a commit to kwvg/dash that referenced this pull request Dec 31, 2024
kwvg added a commit to kwvg/dash that referenced this pull request Jan 2, 2025
kwvg added a commit to kwvg/dash that referenced this pull request Jan 3, 2025
PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Jan 7, 2025
, bitcoin#28757, bitcoin#28845, bitcoin#28932, bitcoin#28973, bitcoin#21778, bitcoin#30204, bitcoin#29765, bitcoin#30201, bitcoin#30287, bitcoin#30511 (toolchain backports: part 3)

ee04e1f merge bitcoin#30511: GCC 12 consolidation (Kittywhiskers Van Gogh)
cc5c30e merge bitcoin#30287: rewrite some docs & swap `mmacosx-version-min` for `mmacos-version-min` (Kittywhiskers Van Gogh)
632b8b4 merge bitcoin#30201: remove `FORCE_USE_SYSTEM_CLANG` (Kittywhiskers Van Gogh)
994e7b1 merge bitcoin#29765: Use clang-18 (Kittywhiskers Van Gogh)
7e7d7bd merge bitcoin#30204: consolidate dependency docs (Kittywhiskers Van Gogh)
061a608 merge bitcoin#21778: LLD based macOS toolchain (Kittywhiskers Van Gogh)
ea891d4 fix: use `LDFLAGS` when compiling `gmp`'s `configure` test programs (Kittywhiskers Van Gogh)
c2fe7c6 merge bitcoin#28973: remove `libz-dev` from macOS build deps (Kittywhiskers Van Gogh)
62e671d merge bitcoin#28932: remove `python3-setuptools` from macOS build deps (Kittywhiskers Van Gogh)
5bef1e5 merge bitcoin#28845: remove `PYTHONPATH` from config.site (Kittywhiskers Van Gogh)
c501a0d merge bitcoin#28757: Zip needs to include all files and set time to SOURCE_DATE_EPOCH (Kittywhiskers Van Gogh)
122a61f merge bitcoin#28432: Produce a `.zip` for macOS distribution (Kittywhiskers Van Gogh)
407deab merge bitcoin#25612: default to using GCC tool wrappers for LTO (with GCC) (Kittywhiskers Van Gogh)
f0a0040 fix: ensure that `dsymutil` available even when `FORCE_USE_SYSTEM_CLANG=1` (Kittywhiskers Van Gogh)
416e154 merge bitcoin#24866: No longer need to hack the PATH variable in `config.site` (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Depends on #6385

  * Omissions related to explicitly defining the location of `DSYMUTIL` (see [dash#6384](#6384) for more information) have been remedied and are now included (completing [bitcoin#24866](bitcoin#24866) and [bitcoin#25612](bitcoin#25612) in this pull request), courtesy of a fix that defines `DSYMUTIL` even when `FORCE_USE_SYSTEM_CLANG` is defined ([commit](f0a0040)).
    * This fix is effectively reverted in [bitcoin#21778](bitcoin#21778) as the backport ensures that `DSYMUTIL` is defined in the `FORCE_USE_SYSTEM_CLANG` case and `cctools` is no longer used.
    * It still remains to allow testing and validation of commits pre-[bitcoin#21778](bitcoin#21778).

  * GMP runs its test programs during build configuration sans `LDFLAGS` ([source](https://github.com/gmp-mirror/gmp/blob/141ed4f98a50e4a3c1a95a1758cb599b7499ac39/acinclude.m4#L1755)), which are quite important when using an LLD-based toolchain as we rely on it to take on `ld64`'s duties ([source](https://lld.llvm.org/MachO/index.html)) using `-fuse-ld=lld` ([source](https://lld.llvm.org/#using-lld)). This has been remedied by patching `acinclude.m4` and `configure.ac` to include `LDFLAGS`.

    <details>

    <summary>configure failure:</summary>

     ```
     checking build system type... x86_64-pc-linux-gnu
     checking host system type... x86_64-apple-darwin
     checking for a BSD-compatible install... /home/debian/.guix-profile/bin/install -c
     checking whether build environment is sane... yes
     checking for x86_64-apple-darwin-strip... no
     checking for strip... strip
     configure: WARNING: using cross tools not prefixed with host triplet
     checking for a thread-safe mkdir -p... /home/debian/.guix-profile/bin/mkdir -p
     checking for gawk... gawk
     checking whether make sets $(MAKE)... yes
     checking whether make supports nested variables... yes
     checking whether to enable maintainer-specific portions of Makefiles... no
     checking ABI=64
     checking whether env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks is gcc... yes
     checking compiler env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -pipe -std=c11 -mmacos-version-min=11.0 -mlinker-version=711 -O2 -march=x86-64    -I/dash/depends/x86_64-apple-darwin/include... no
     checking ABI=x32
     checking whether env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks is gcc... yes
     checking compiler env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -pipe -std=c11 -mmacos-version-min=11.0 -mlinker-version=711 -O2 -march=x86-64    -I/dash/depends/x86_64-apple-darwin/include... no
     checking ABI=32
     checking whether env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks is gcc... yes
     checking compiler env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -pipe -std=c11 -mmacos-version-min=11.0 -mlinker-version=711 -O2 -march=x86-64    -I/dash/depends/x86_64-apple-darwin/include... no
     configure: error: could not find a working compiler, see config.log for details
     make: *** [funcs.mk:302: /dash/depends/x86_64-apple-darwin/.gmp_stamp_configured] Error 1
     make: Leaving directory '/dash/depends'
     ```

    </details>

    <details>

    <summary>config.log excerpt:</summary>

     ```
     This file contains any messages produced by compilers while
     running configure, to aid debugging if configure makes a mistake.

     It was created by GNU MP configure 6.3.0, which was
     generated by GNU Autoconf 2.69.  Invocation command line was

     [...]

     ## --------- ##
     ## Platform. ##
     ## --------- ##

     hostname = [...]
     uname -m = x86_64
     uname -r = 6.6.32[...]
     uname -s = Linux
     uname -v = #1 SMP PREEMPT_DYNAMIC Mon Jul  8 16:11:58 UTC 2024

     /usr/bin/uname -p = unknown
     /bin/uname -X     = unknown

     /bin/arch              = unknown
     /usr/bin/arch -k       = unknown
     /usr/convex/getsysinfo = unknown
     /usr/bin/hostinfo      = unknown
     /bin/machine           = unknown
     /usr/bin/oslevel       = unknown
     /bin/universe          = unknown

     PATH: /dash/depends/x86_64-apple-darwin/native/bin
     PATH: /home/debian/.guix-profile/bin
     PATH: /home/debian/.guix-profile/sbin

     ## ----------- ##
     ## Core tests. ##
     ## ----------- ##

     [...]
     configure:5889: env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -c conftest.c >&5
     configure:5892: $? = 0
     configure:5895: checking whether env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks is gcc
     configure:5897: result: yes
     configure:5911: env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks 2>&1 | grep xlc >/dev/null
     configure:5914: $? = 1
     configure:5968: checking compiler env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -pipe -std=c11 -mmacos-version-min=11.0 -mlinker-version=711 -O2 -march=x86-64   -I/dash/depends/x86_64-apple-darwin/include
     Test compile:
     configure:5982: env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH /home/debian/.guix-profile/bin/clang --target=x86_64-apple-darwin -isysroot/dash/depends/SDKs/Xcode-15.0-15A240d-extracted-SDK-with-libcxx-headers -nostdlibinc -iwithsysroot/usr/include -iframeworkwithsysroot/System/Library/Frameworks -pipe -std=c11 -mmacos-version-min=11.0 -mlinker-version=711 -O2 -march=x86-64   -I/dash/depends/x86_64-apple-darwin/include conftest.c >&5
     ld: unrecognized option '-dynamic'
     ld: use the --help option for usage information
     clang: error: linker command failed with exit code 1 (use -v to see invocation)
     configure:5985: $? = 1
     failed program was:

     int main () { return 0; }
     configure:7072: result: no
     [...]
     configure:7310: error: could not find a working compiler, see config.log for details
     [...]

     ## ---------------- ##
     ## Cache variables. ##
     ## ---------------- ##

     [...]
     ac_cv_env_LDFLAGS_set=set
     ac_cv_env_LDFLAGS_value='-Wl,-platform_version,macos,11.0,14.0 -Wl,-no_adhoc_codesign -fuse-ld=lld  -L/dash/depends/x86_64-apple-darwin/lib'
     [...]
     LD=''
     LDFLAGS='-Wl,-platform_version,macos,11.0,14.0 -Wl,-no_adhoc_codesign -fuse-ld=lld  -L/dash/depends/x86_64-apple-darwin/lib'
     [...]

     ## ----------- ##
     ## confdefs.h. ##
     ## ----------- ##

     /* confdefs.h */
     #define PACKAGE_NAME "GNU MP"
     #define PACKAGE_TARNAME "gmp"
     #define PACKAGE_VERSION "6.3.0"
     #define PACKAGE_STRING "GNU MP 6.3.0"
     #define PACKAGE_BUGREPORT "gmp-bugs@gmplib.org (see https://gmplib.org/manual/Reporting-Bugs.html)"
     #define PACKAGE_URL "http://www.gnu.org/software/gmp/"
     #define PACKAGE "gmp"
     #define VERSION "6.3.0"
     #define WANT_FFT 1
     #define HAVE_HOST_CPU_x86_64 1

     configure: exit 1
     ```

     **Note:** `lld` itself will never respond like this, as it will exit with a catch-all message requesting you to invoke it with the correct name

     ```
     $ lld -dynamic
     lld is a generic driver.
     Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead
     ```

     Nor is it calling the Unix-specific variant mistakenly

     ```
     $ ld.lld -dynamic
     ld.lld: error: unknown argument '-dynamic', did you mean '-Bdynamic'
     ```

     The error displayed above is consistent with GNU `ld`
     ```
     $ ld -dynamic
     ld: unrecognized option '-dynamic'
     ld: use the --help option for usage information
     ```

    </details>

    * A similar problem is observed for Qt, which was resolved upstream by adding it to `CXXFLAGS` ([source](https://github.com/bitcoin/bitcoin/blob/e8c25e8a35e333e90514945c592557615641553f/depends/patches/qt/mac-qmake.conf#L16)) (but this comes at the downside of an unused argument warnings whenever the compiler doesn't intend to do any linking and is why we didn't just modify `darwin_{CC,CXX}` instead)

      <details>

      <summary>Warning messages when building Qt:</summary>

      ```
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      compiling kernel/qplatformcursor.cpp
      compiling kernel/qplatformclipboard.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      compiling kernel/qplatformsurface.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      4 warnings generated.
      compiling kernel/qcursor.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      compiling kernel/qguivariant.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      compiling kernel/qshortcutmap.cpp
      compiling kernel/qtouchdevice.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      compiling kernel/qplatformservices.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      kernel/qplatformcursor.cpp:622:17: warning: variable 'w' set but not used [-Wunused-but-set-variable]
      622 |     int x = -1, w = 0;
          |                 ^
      compiling kernel/qplatformsessionmanager.cpp
      clang++: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
      ```

      </details>

  * Clang has been bumped to 18 as [bitcoin#30201](bitcoin#30201) drops `native_llvm` (formerly known as `native_clang`) and mandates the presence of Clang 18 or higher for cross-compilation ([source](https://github.com/bitcoin/bitcoin/blob/7cbfd7a7ce45ac68d6041f42f468862f5c193d8c/depends/README.md?plain=1#L54-L57)).

  ## Breaking Changes

  None expected

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  PastaPastaPasta:
    utACK ee04e1f
  UdjinM6:
    utACK ee04e1f

Tree-SHA512: 0b8be036419a53eda5be77c9abdf214d8f5baa088526f9249daba31e650c50f831f54c389845f828f8b7cf50458967f211a6d42d12b8dae8546b265d7f37a968
@bitcoin bitcoin locked and limited conversation to collaborators May 30, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants