contrib: macdeploy: make gen-sdk deterministic #24534

prusnak · 2022-03-11T15:23:46Z

This PR attempts to make contrib/macdeploy/gen-sdk deterministic

Can anyone with the Xcode_12.2.xip confirm that gen-sdk produces the same hash? => e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae

fanquake · 2022-03-11T15:55:08Z

Testing your branch @ 13a4a09:

shasum -a 256 Xcode_12.2.xip
28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0  Xcode_12.2.xip
....
contrib/macdeploy/gen-sdk Xcode.app
Found Xcode (version: 12.2, build id: 12B45b)
Found MacOSX SDK (version: 11.0, build id: 20A2408)
Creating output .tar.gz file...
Adding MacOSX SDK 11.0 files...
Adding libc++ headers...
Done! Find the resulting gzipped tarball at:
Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
...
shasum -a 256 Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 
a396dd24f61fb55a6d3ec98b8b58fc0b04cdb6b2695039869d04105f885e0867  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

prusnak · 2022-03-11T16:23:35Z

Testing your branch @ 13a4a09:

How did you extract the xip file? Via xip -x or apple-sdk-tools/extract_xcode.py ?

Edit: I tried to unpack the xip file using both methods and despite the extracted folders have different timestamps (xip -x keeps the timestamps from the archive, the cpio method ruins them), the updated gen-sdk script produced the same hash for me.

Will investigate further ...

prusnak · 2022-03-11T17:12:09Z

It seems the default TAR format has changed from GNU to PAX in Python 3.8.

This is addressed by change in 140f8e7 which sets the GNU format explicitly.

prusnak · 2022-03-11T18:46:48Z

Argh, found another culprit - https://bugs.python.org/issue18819 (fixed via python/cpython#18080)

Python 3.8 and older use ASCII string 0000000\x00 for devmajor/devminor to indicate the entry is not a device. Python 3.9 and newer use the binary string \x00\x00\x00\x00\x00\x00\x00\x00 (to match the original TAR behaviour) 😭

This might be fixed by monkey-patching the tarfile.TarInfo._create_header function.

on different Python versions (there was a change in TAR handling between Python 3.8 and Python 3.9)

prusnak · 2022-03-12T11:38:25Z

This might be fixed by monkey-patching the tarfile.TarInfo._create_header function.

I monkey-patched the Python bug in ba30a54.

The new deterministic hash should be e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae,

I was able to reproduce this hash on:

Linux with Python 3.9 where the SDK was extracted via apple-sdk-tools/extract_xcode.py
macOS with Python 3.8 where the SDK was extracted via xip -x

hebasto · 2022-03-12T13:32:04Z

Concept ACK.

fanquake · 2022-03-12T20:33:03Z

The new deterministic hash should be e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae,

Nice. I'm now getting a matching hash.

➜  SDK ./contrib/macdeploy/gen-sdk Xcode.app                                     
Found Xcode (version: 12.2, build id: 12B45b)
Found MacOSX SDK (version: 11.0, build id: 20A2408)
Creating output .tar.gz file...
Adding MacOSX SDK 11.0 files...
Adding libc++ headers...
Done! Find the resulting gzipped tarball at:
Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
➜  SDK shasum -a 256 Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

prusnak · 2022-03-12T21:45:25Z

Nice. I'm now getting a matching hash.

Thanks. Switching from Draft to Ready for review.

DrahtBot · 2022-03-12T21:58:13Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#21778 (build: LLVM 14 & LLD based macOS toolchain by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

laanwj · 2022-04-06T12:13:54Z

Concept ACK. I think having a deterministic MacOS SDK input is very useful, so that we can be sure to start from the same point based on hashes. I think we should at least get this in before the next SDK bump.

jarolrod · 2022-04-13T05:25:13Z

Concept ACK, would love to see this be deterministic but I have a hash mismatch with the resulting archive file.

I get the same hash for the downloaded xip file:

$ sha256sum ./Xcode_12.2.xip 

28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0

But, I get a different hash for the resulting archive file:

$ ./contrib/macdeploy/gen-sdk ./Xcode.app

Found Xcode (version: 12.2, build id: 12B45b)
Found MacOSX SDK (version: 11.0, build id: 20A2408)
Creating output .tar.gz file...
Adding MacOSX SDK 11.0 files...
Adding libc++ headers...
Done! Find the resulting gzipped tarball at:
/home/xyz/Code/Bitcoin/review/bitcoin-24534/bitcoin/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

$ sha256sum ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 

501625bd401d7f228b3bae18f264fd3739da5788464f0897e5c455451d2128c2  ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

prusnak · 2022-04-15T19:30:03Z

Concept ACK, would love to see this be deterministic but I have a hash mismatch with the resulting archive file.

OS version and Python version? Also what filesystem are you using?

laanwj · 2022-04-19T11:18:09Z

Tested ACK ba30a54

Output matches the hash in the OP (tried on Ubuntu 22.04, x86_64, Python 3.10.4):

$ sha256sum ~/Downloads/Xcode_12.2.xip
28d352f8c14a43d9b8a082ac6338dc173cb153f964c6e8fb6ba389e5be528bd0  /home/orion/Downloads/Xcode_12.2.xip
$ …/apple-sdk-tools/extract_xcode.py -f ~/Downloads/Xcode_12.2.xip | cpio -d -i
$ …/bitcoin/contrib/macdeploy/gen-sdk $PWD/Xcode.app/
Found Xcode (version: 12.2, build id: 12B45b)
Found MacOSX SDK (version: 11.0, build id: 20A2408)
Creating output .tar.gz file...
Adding MacOSX SDK 11.0 files...
Adding libc++ headers...
Done! Find the resulting gzipped tarball at:
…/apple-sdk-tools/Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
$ sha256sum Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz
e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

jarolrod · 2022-04-19T18:21:56Z

@prusnak That run was Ubuntu 22.04, python 3.10, on arm64 cpu. Will rerun on x86 ubuntu

reserved for run on x86 ubuntu

prusnak · 2022-04-19T21:33:57Z

@jarolrod Hm, weird. can you please share the tarball with the 501625bd401d7f228b3bae18f264fd3739da5788464f0897e5c455451d2128c2 hash? I could compare locally and try to figure out what went wrong.

jarolrod

Tested ACK ba30a54

I don't know what had happened in my original run, its possible that on accident I hadn't successfully changed to the PR branch. I re-ran everything on my setup and now I get a matching hash :)

sha256sum ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz 
e7ca56bc8804d16624fad68be2e71647747d6629cacaaa3de5fbfa7f444e9eae  ./Xcode-12.2-12B45b-extracted-SDK-with-libcxx-headers.tar.gz

laanwj · 2022-04-22T19:08:33Z

I think this causes an error while building from the SDK: #24947

fanquake added macOS Build system Scripts and tools and removed Build system labels Mar 11, 2022

prusnak force-pushed the deterministic-macos-sdk branch 2 times, most recently from c719824 to 13a4a09 Compare March 11, 2022 15:52

prusnak force-pushed the deterministic-macos-sdk branch from 13a4a09 to 140f8e7 Compare March 11, 2022 17:11

prusnak added 2 commits March 12, 2022 12:35

contrib: macdeploy: make gen-sdk deterministic

1868a17

contrib: macdeploy: monkey-patch gen-sdk to be deterministic

ba30a54

on different Python versions (there was a change in TAR handling between Python 3.8 and Python 3.9)

prusnak force-pushed the deterministic-macos-sdk branch from 140f8e7 to ba30a54 Compare March 12, 2022 11:37

prusnak marked this pull request as ready for review March 12, 2022 21:45

DrahtBot mentioned this pull request Mar 13, 2022

build: LLD based macOS toolchain #21778

Merged

prusnak mentioned this pull request Mar 18, 2022

build: Use Xcode .xip as Guix input #18674

Open

jarolrod approved these changes Apr 21, 2022

View reviewed changes

laanwj merged commit 6f55ab5 into bitcoin:master Apr 21, 2022

prusnak deleted the deterministic-macos-sdk branch April 21, 2022 08:11

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 22, 2022

Merge bitcoin#24534: contrib: macdeploy: make gen-sdk deterministic

26f4505

laanwj mentioned this pull request Apr 22, 2022

Permissions corrupted in gen-sdk macosx headers #24947

Closed

bitcoin locked and limited conversation to collaborators Apr 22, 2023

contrib: macdeploy: make gen-sdk deterministic #24534

contrib: macdeploy: make gen-sdk deterministic #24534

Uh oh!

Conversation

prusnak commented Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanquake commented Mar 11, 2022

Uh oh!

prusnak commented Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prusnak commented Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prusnak commented Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prusnak commented Mar 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hebasto commented Mar 12, 2022

Uh oh!

fanquake commented Mar 12, 2022

Uh oh!

prusnak commented Mar 12, 2022

Uh oh!

DrahtBot commented Mar 12, 2022

Conflicts

Uh oh!

laanwj commented Apr 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jarolrod commented Apr 13, 2022

Uh oh!

prusnak commented Apr 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laanwj commented Apr 19, 2022

Uh oh!

jarolrod commented Apr 19, 2022

Uh oh!

prusnak commented Apr 19, 2022

Uh oh!

jarolrod left a comment

Choose a reason for hiding this comment

Uh oh!

laanwj commented Apr 22, 2022

Uh oh!

Uh oh!

prusnak commented Mar 11, 2022 •

edited

Loading

prusnak commented Mar 11, 2022 •

edited

Loading

prusnak commented Mar 11, 2022 •

edited

Loading

prusnak commented Mar 11, 2022 •

edited

Loading

prusnak commented Mar 12, 2022 •

edited

Loading

laanwj commented Apr 6, 2022 •

edited

Loading

prusnak commented Apr 15, 2022 •

edited

Loading