build: macOS toolchain bump #19817

fanquake · 2020-08-27T08:11:38Z

Bumps our macOS toolchain to be using the following:

Clang 10.0.1 (gitian) & Clang 10.0.0 (Guix)
ld64 609
libtapi 1100.0.11
cctools 973.0.1
Xcode 12.1
macOS SDK 10.15.6

which are currently the most recent releases available as open source. See upstream cctools and libtapi.

This should improve the possibility of Apple ARM cross-compilation in depends.

This also removes our patching out of pthreads usage in ld64. There have been multiple changes since ld64 450.3, which have likely fixed the non-determinism we were working around. i.e from InputFiles.cpp:

// <rdar://problem/15002251> make implicit dylib order be deterministic by sorting by install_name
		std::sort(implicitDylibs.begin(), implicitDylibs.end(), DylibByInstallNameSorter());

// <rdar://problem/42675402> ld64 output is not deterministic due to dylib processing order
		std::sort(unprocessedDylibs.begin(), unprocessedDylibs.end(), [](const ld::dylib::File* lhs, const ld::dylib::File* rhs) {
			return strcmp(lhs->path(), rhs->path()) < 0;
		});

Guix Build:

find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5  guix-build-a5550f877a2c/output/dist-archive/bitcoin-a5550f877a2c.tar.gz
4954dcf563c2d496b8d9fecd48f8e3f7fba2f319ffa254a5bc8ee12cfee6acf0  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.dmg
8f6095b445c7f1a8e6accd86bb7f0696d5849402084927d2b726b7d557831c3a  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.tar.gz
cc40f25477b4defc1617ae694313d80f307ddf6742fe6cc85c6bc0e215ef8be0  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx64.tar.gz

Gitian Build:

Generating report
506a8abdefe559999b43dd9f14905b9b2b5a3363b1cd013d45ae47acc1f7ef6c  bitcoin-a5550f877a2c-osx-unsigned.dmg
f606997f74026dd12d110d683c6f116b40df324836904ef507dd7ac787e6ebe2  bitcoin-a5550f877a2c-osx-unsigned.tar.gz
5b495ef15f2c3260c2950921b61326912a9bf533cccd51e13818809fd225489e  bitcoin-a5550f877a2c-osx64.tar.gz
f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5  src/bitcoin-a5550f877a2c.tar.gz
9eb0221e962d2839770963bd03c6c9e98e8bf3078566bee2ae42f06233a710fa  bitcoin-core-osx-22-res.yml
Done.

jonasschnelli · 2020-08-27T09:45:39Z

should someone verify if the right sources have been taken from upstream (unchanged)?
https://github.com/tpoechtrager/apple-libtapi
https://github.com/apple/llvm-project/commits/apple/stable/20190104

Why is the snv suffix in the library name required?

fanquake · 2020-08-27T14:24:58Z

Why is the snv suffix in the library name required?

The .8svn suffix comes from LLVMs dylib naming, which is the LLVM major version number + the LLVM version suffix (svn). After building you can grep and find:

build/tools/llvm-config/BuildVariables.inc
35:#define LLVM_DYLIB_VERSION "8svn"

build/include/llvm/Config/config.h
330:#define PACKAGE_STRING "LLVM 8.0.0svn"
333:#define PACKAGE_VERSION "8.0.0svn"

build/include/llvm/Config/llvm-config.h
78:#define LLVM_VERSION_STRING "8.0.0svn"

etc.

should someone verify if the right sources have been taken from upstream (unchanged)?

I have done some diffs and can post them shortly.

fanquake · 2020-08-28T14:27:29Z

should someone verify if the right sources have been taken from upstream (unchanged)?

I've posted a diff of the Clang and libtapi sources here: https://gist.github.com/fanquake/1512109cc69d0a61f352e326f34bb90a.

You can recreate the using the following steps:
Clang:

git clone https://github.com/apple/llvm-project
git clone https://github.com/tpoechtrager/apple-libtapi

cd llvm-project
git checkout 729748d085a90bd2a4af36efbfb2dc33b4704de3

rm -rf clang/test
rm -rf clang/unittests/
rm -rf clang/www
cd ..

# compare clang
diffoscope --exclude-directory-metadata=yes llvm-project/clang apple-libtapi/src/llvm/projects/clang/

libtapi:

wget https://opensource.apple.com/tarballs/tapi/tapi-1100.0.11.tar.gz
tar xf tapi-1100.0.11.tar.gz
cd tapi-1100.0.11.tar.gz
rm -rf test/
rm -rf unittests/
cd ..

# compare libtapi
diffoscope --exclude-directory-metadata=yes tapi-1100.0.11 apple-libtapi/src/libtapi

If anyone wants an intro to .tbd stubs, I wrote something here using libconsensus: https://github.com/fanquake/core-review/blob/master/tbd-stubs.md.

promag · 2020-08-28T14:37:22Z

If anyone wants an intro to .tbd stubs, I wrote something here using libconsensus: https://github.com/fanquake/core-review/blob/master/tbd-stubs.md.

TIL, very nice! 👏

maflcko · 2020-08-29T17:46:55Z

make: Entering directory '/home/ubuntu/build/bitcoin/depends'
Extracting native_cctools...
/home/ubuntu/cache/common/55562e4073dea0fbfd0b20e0bf69ffe6390c7f97.tar.gz: OK
/home/ubuntu/cache/common/clang-llvm-8.0.0-x86_64-linux-gnu-ubuntu-14.04.tar.xz: OK
sha256sum: /home/ubuntu/cache/common/86f43cdb62a3ceb39f3ee6e4568eded67a4912e8.tar.gz: No such file or directory
/home/ubuntu/cache/common/86f43cdb62a3ceb39f3ee6e4568eded67a4912e8.tar.gz: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read
funcs.mk:260: recipe for target '/home/ubuntu/build/bitcoin/depends/work/build/x86_64-apple-darwin16/native_cctools/55562e4073dea0fbfd0b20e0bf69ffe6390c7f97-cd4b9b6d54b/.stamp_extracted' failed
make: *** [/home/ubuntu/build/bitcoin/depends/work/build/x86_64-apple-darwin16/native_cctools/55562e4073dea0fbfd0b20e0bf69ffe6390c7f97-cd4b9b6d54b/.stamp_extracted] Error 1
make: Leaving directory '/home/ubuntu/build/bitcoin/depends'

maflcko · 2020-08-29T19:03:40Z

Steps to reproduce:

   git checkout bitcoin-core/master 
   cd depends/
   make download HOST=x86_64-apple-darwin16 
   git cherry-pick 826d42903da5a927f296be1fa86096bce53d5f1b
   make HOST=x86_64-apple-darwin16

laanwj · 2020-09-03T10:22:24Z

If anyone wants an intro to .tbd stubs, I wrote something here using libconsensus: https://github.com/fanquake/core-review/blob/master/tbd-stubs.md.

I've wanted this kind of thing forever. I wish this was cross-platform.

fanquake · 2020-09-15T08:44:59Z

This needs updating for a newer libtapi update, and one of either @dongcarl's or @hebasto's "fix building dependant libs" commits pulled in to fix the rebuilding.

jonasschnelli · 2020-12-04T10:01:57Z

@MarcoFalke: you need to clear the depends/sources/ dir (a make clean) is not sufficient.

Tested ACK 1a1fc87 - tested with Xcode-12.2-12B45b

maflcko · 2020-12-04T10:09:39Z

depends should ideally detect missing or outdated sources automatically and download them. This is how it used to work, am I wrong?

fanquake · 2020-12-09T13:15:42Z

Seeing as this is now useful for a reason other than mentioned in the OP, I'll address outstanding issues and take this out of draft.

depends should ideally detect missing or outdated sources automatically and download them. This is how it used to work, am I wrong?

IIRC this has never been the case for the native_cctools package. However it's always been masked by the fact that we'd always be bumping Clang when changing libtapi, so you'd always be rebuilding anyways.

DrahtBot · 2021-03-17T16:07:48Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

build, qt: No longer need to patch translation.pro #21420 (build, qt: No longer need to patch translation.pro by hebasto)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

fanquake · 2021-03-18T02:47:45Z

I've modified this to be a macOS toolchain bump, based on #21457, rather than updating libtapi individually.

merge bitcoin#19817, bitcoin#20470, bitcoin#22993: macOS C++17 bump

update build system for darwin cross-compile to match bitcoin upstream: - bitcoin/bitcoin#21457 - split libtapi and clang out of native_cctools - bitcoin/bitcoin#19817 - macOS toolchain bump

…ge)_download_file` assignments d644c45 build, refactor: Drop redundant `$(package)_download_file` assignments (Hennadii Stepanov) Pull request description: No need to specify `$(package)_download_file` when it is equal to `$(package)_file_name`. Historically, before bitcoin/bitcoin#19817, distinct `$(package)_download_file` and `$(package)_file_name` were used for better portability (I guess) by removing `+` characters from a file name. The only package which still use file renaming is `native_capnp`: https://github.com/bitcoin/bitcoin/blob/eca694a4e78d54ce4e29b388b3e81b06e55c2293/depends/packages/native_capnp.mk#L3-L5 ACKs for top commit: shaavan: ACK d644c45 fanquake: ACK d644c45 Tree-SHA512: 488dd0f55cea077174e78a75d8385bacb1a5463883cadeb5fd7c9426865ea5f3a8bad0bd6e8e9d530bce6f0c1715349b3fbabb4e22634348cdd68f5fc8a3c53b

…load_file` assignments d644c45 build, refactor: Drop redundant `$(package)_download_file` assignments (Hennadii Stepanov) Pull request description: No need to specify `$(package)_download_file` when it is equal to `$(package)_file_name`. Historically, before bitcoin#19817, distinct `$(package)_download_file` and `$(package)_file_name` were used for better portability (I guess) by removing `+` characters from a file name. The only package which still use file renaming is `native_capnp`: https://github.com/bitcoin/bitcoin/blob/eca694a4e78d54ce4e29b388b3e81b06e55c2293/depends/packages/native_capnp.mk#L3-L5 ACKs for top commit: shaavan: ACK d644c45 fanquake: ACK d644c45 Tree-SHA512: 488dd0f55cea077174e78a75d8385bacb1a5463883cadeb5fd7c9426865ea5f3a8bad0bd6e8e9d530bce6f0c1715349b3fbabb4e22634348cdd68f5fc8a3c53b

fanquake added the Build system label Aug 27, 2020

maflcko added Needs gitian build DrahtBot Guix build requested labels Aug 27, 2020

laanwj added the macOS label Aug 27, 2020

DrahtBot removed DrahtBot Guix build requested Needs gitian build labels Aug 29, 2020

fanquake marked this pull request as draft September 15, 2020 08:44

fanquake force-pushed the libtapi_1100011 branch from 826d429 to 1a1fc87 Compare September 19, 2020 12:46

fanquake force-pushed the libtapi_1100011 branch from 1a1fc87 to 061e155 Compare January 29, 2021 08:36

fanquake mentioned this pull request Mar 17, 2021

build: split libtapi and clang out of native_cctools #21457

Merged

fanquake changed the title ~~build: libtapi 1100.0.11~~ build: macOS toolchain bump Mar 18, 2021

fanquake force-pushed the libtapi_1100011 branch from 061e155 to a09d6db Compare March 18, 2021 02:46

fanquake mentioned this pull request Mar 18, 2021

build: Bump clang version to fix non-determinism #20454

Closed

fanquake added this to the 22.0 milestone Mar 18, 2021

bitcoin deleted a comment from DrahtBot Mar 18, 2021

DrahtBot mentioned this pull request Mar 18, 2021

guix: Add guix-{attest,verify} scripts #21462

Merged

kwvg pushed a commit to kwvg/dash that referenced this pull request Aug 27, 2021

partial merge bitcoin#19817: clang 10.0.1

cd9be9a

kwvg pushed a commit to kwvg/dash that referenced this pull request Aug 30, 2021

partial merge bitcoin#19817: clang 10.0.1

50060a3

kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021

partial merge bitcoin#19817: clang 10.0.1

19e01b5

kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021

partial merge bitcoin#19817: clang 10.0.1

eebfef9

kwvg added a commit to kwvg/dash that referenced this pull request Sep 1, 2021

partial merge bitcoin#19817: clang 10.0.1

2cbbbd6

kwvg added a commit to kwvg/dash that referenced this pull request Sep 2, 2021

partial merge bitcoin#19817: clang 10.0.1

0b31f21

kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021

partial merge bitcoin#19817: clang 10.0.1

1b5f20e

kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021

partial merge bitcoin#19817: clang 10.0.1

0681143

kwvg added a commit to kwvg/dash that referenced this pull request Sep 3, 2021

partial merge bitcoin#19817: clang 10.0.1

89b7b32

kwvg added a commit to kwvg/dash that referenced this pull request Sep 4, 2021

partial merge bitcoin#19817: clang 10.0.1

b20e601

UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Sep 5, 2021

partial merge bitcoin#19817: build: use -stdlib++-isystem with Clang 10

2bc2607

kwvg added a commit to kwvg/dash that referenced this pull request Oct 2, 2021

merge bitcoin#19817: macOS toolchain bump

6ed6716

kwvg added a commit to kwvg/dash that referenced this pull request Oct 2, 2021

merge bitcoin#19817: macOS toolchain bump

f022a01

kwvg added a commit to kwvg/dash that referenced this pull request Oct 5, 2021

merge bitcoin#19817: macOS toolchain bump

091477d

PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Oct 5, 2021

Merge pull request #4477 from kittywhiskers/dependsbp

07b1d52

merge bitcoin#19817, bitcoin#20470, bitcoin#22993: macOS C++17 bump

kwvg added a commit to kwvg/dash that referenced this pull request Oct 12, 2021

merge bitcoin#19817: macOS toolchain bump

752e52e

pravblockc pushed a commit to pravblockc/dash that referenced this pull request Nov 18, 2021

merge bitcoin#19817: macOS toolchain bump

7488507

hebasto mentioned this pull request Feb 8, 2022

build, refactor: Drop redundant $(package)_download_file assignments #24288

Merged

str4d mentioned this pull request Apr 15, 2022

Backport upstream macOS build changes from 2021 zcash/zcash#5884

Open

gades pushed a commit to cosanta/cosanta-core that referenced this pull request May 4, 2022

merge bitcoin#19817: macOS toolchain bump

740d6f5

gades pushed a commit to cosanta/cosanta-core that referenced this pull request May 8, 2022

partial merge bitcoin#19817: clang 10.0.1

406d9b5

gwillen pushed a commit to ElementsProject/elements that referenced this pull request Jun 1, 2022

Merge 77d569c into merged_master (Bitcoin PR bitcoin/bitcoin#19817)

bf708b8

bitcoin locked as resolved and limited conversation to collaborators Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

build: macOS toolchain bump #19817

build: macOS toolchain bump #19817

Uh oh!

fanquake commented Aug 27, 2020 •

edited

Loading

Uh oh!

jonasschnelli commented Aug 27, 2020

Uh oh!

fanquake commented Aug 27, 2020

Uh oh!

fanquake commented Aug 28, 2020

Uh oh!

promag commented Aug 28, 2020

Uh oh!

maflcko commented Aug 29, 2020

Uh oh!

maflcko commented Aug 29, 2020

Uh oh!

laanwj commented Sep 3, 2020

Uh oh!

fanquake commented Sep 15, 2020

Uh oh!

jonasschnelli commented Dec 4, 2020

Uh oh!

maflcko commented Dec 4, 2020

Uh oh!

fanquake commented Dec 9, 2020

Uh oh!

DrahtBot commented Mar 17, 2021 •

edited

Loading

Uh oh!

fanquake commented Mar 18, 2021

Uh oh!

Uh oh!

build: macOS toolchain bump #19817

build: macOS toolchain bump #19817

Uh oh!

Conversation

fanquake commented Aug 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonasschnelli commented Aug 27, 2020

Uh oh!

fanquake commented Aug 27, 2020

Uh oh!

fanquake commented Aug 28, 2020

Uh oh!

promag commented Aug 28, 2020

Uh oh!

maflcko commented Aug 29, 2020

Uh oh!

maflcko commented Aug 29, 2020

Uh oh!

laanwj commented Sep 3, 2020

Uh oh!

fanquake commented Sep 15, 2020

Uh oh!

jonasschnelli commented Dec 4, 2020

Uh oh!

maflcko commented Dec 4, 2020

Uh oh!

fanquake commented Dec 9, 2020

Uh oh!

DrahtBot commented Mar 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

fanquake commented Mar 18, 2021

Uh oh!

Uh oh!

fanquake commented Aug 27, 2020 •

edited

Loading

DrahtBot commented Mar 17, 2021 •

edited

Loading