Skip to content

Conversation

dongcarl
Copy link
Contributor

@dongcarl dongcarl commented Nov 24, 2020

This is a redo of fanquake's #18151, which, aside from switching us from the deprecated genisoimage to the maintained xorriso, is also necessary for Guix to achieve determinism without using faketime.

xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.

In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.

From wiki.debian.org/genisoimage?action=recall&rev=11 :

The classical command line interface for production of ISO 9660
filesystem images is the option set established by program mkisofs.
For reasons of licensing and other problems with its author, Debian
ships a fork of mkisofs, called genisoimage, which was split off in
2006 and then developed independently.

Meanwhile, genisoimage gets no new features and not even bug fixes. It
is first choice only if its options -udf or -hfs are needed.

Replacement in most uses cases, especially for bootable ISO 9660
filesystems, archiving, and backup, is xorrisofs which starts the -as
mkisofs emulation mode of program xorriso.

@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 24, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@laanwj
Copy link
Member

laanwj commented Nov 24, 2020

Concept ACK

@practicalswift
Copy link
Contributor

Concept ACK: maintained is better than unmaintained :)

@hebasto
Copy link
Member

hebasto commented Nov 25, 2020

Concept ACK.

@DrahtBot
Copy link
Contributor

Gitian builds

File commit e2ff5e7
(master)
commit 4cd306c
(master and this pull)
bitcoin-core-linux-22-res.yml 44e52af36236879f... 9e35f5cf2a311cd8...
bitcoin-core-osx-22-res.yml 71d5fb0e6642d6b0... 39c3b4d8606449d9...
bitcoin-core-win-22-res.yml 0148a2f5e7295184... c44f40f485521b1d...
*-aarch64-linux-gnu-debug.tar.gz 383e84aaa96f342a... c2373d9b17ef64aa...
*-aarch64-linux-gnu.tar.gz 3558211891c0484a... bbe6f3a350eee12e...
*-arm-linux-gnueabihf-debug.tar.gz 5ebbb475a590480a... b8620629a4943471...
*-arm-linux-gnueabihf.tar.gz 60951d893e2688b6... 6a29c12f830614d5...
*-osx-unsigned.dmg 7cba6edda598c96c... 7746491c14d8e536...
*-osx64.tar.gz ea2ab5c893780a2e... 40b977f268fd9103...
*-riscv64-linux-gnu-debug.tar.gz 0c20a43923efcd5f... e0c2d9f76e8f7bdd...
*-riscv64-linux-gnu.tar.gz 8d3632d81c77eff1... 1d6d0f4afed9e98d...
*-win64-debug.zip 72c35e133cf9a0db... 44bb9249ea02fb4c...
*-win64-setup-unsigned.exe 55c17bfff54273e5... 6bb09951569edd6b...
*-win64.zip f7bf6737c7f741c3... 22a04573095f31ee...
*-x86_64-linux-gnu-debug.tar.gz abf91baef5c44686... 2100bfc3e7ad0630...
*-x86_64-linux-gnu.tar.gz 75fda490174df335... 17ce53433f37c554...
*.tar.gz 793a2c7af39452d5... 42b4a292b2236755...
linux-build.log 9c5461a914e1f4c0... 58c11f1d8d0b6a9e...
osx-build.log 5adaf67afbaf00fc... c91d144bb6cd0bcc...
win-build.log 756101ecccffabf0... 3488da316ac6068e...
bitcoin-core-linux-22-res.yml.diff 872de92646873957...
bitcoin-core-osx-22-res.yml.diff c2beeb988be1ec6f...
bitcoin-core-win-22-res.yml.diff 70161d6cfad01933...
linux-build.log.diff 1e0eb4be52ea6b97...
osx-build.log.diff 15491d7a0b331268...
win-build.log.diff d6882ad0a0026167...

@jonasschnelli
Copy link
Contributor

Tested ACK f2cfb5c - Did a gitian build and verified the OSX disk image https://bitcoin.jonasschnelli.ch/gitian/build/347

@dongcarl
Copy link
Contributor Author

dongcarl commented Dec 9, 2020

Pushed b9d60dd -> 75bfa82, specify that we need to use the faketime-wrapped xorrisofs to ./configure in Gitian

@fanquake
Copy link
Member

Looks good. I tested with 48e1b68 on top of master (86f2007).

One thing I noticed is that the naming of the DMGs is now inconsistent. i.e a make deploy on macOS still produces Bitcoin-Core.dmg, whereas when cross-compiling you'll end up with bitcoin-21.99.0.dmg (along with the .iso). We should probably make this consistent. #20422 recently changed the name of the DMG produced on macOS (to Bitcoin-Core.dmg over Bitcoin-Qt.dmg`). Probably worth rebasing this branch so that you're working on top of those changes.

Will be nice if xorrisofs can do DMG compression in the future.

Note: we should also add a *.iso entry to .gitignore as part of this change.

@dongcarl
Copy link
Contributor Author

dongcarl commented Dec 11, 2020

Addressed fanquake's concerns. Here are the Gitian hashes for fdb76e6

ee16ae845d4ea95842abd59e81b91c3a84924149c94c526c2f68d46f08b40cf3  bitcoin-fdb76e62ce8d-osx-unsigned.dmg
a96e1af388a4c1616489d4d52076d567c1ab33557bd0e56ea21fbd766739d33c  bitcoin-fdb76e62ce8d-osx-unsigned.tar.gz
c2691b955bf1d781f19c41908f9152264502a299b827b30d40049b8f567d35e6  bitcoin-fdb76e62ce8d-osx-unsigned.temp.iso
fc298e3f2f592398a386d9cf6eabaed8c41e7bf949636014ef78120430ef1351  bitcoin-fdb76e62ce8d-osx64.tar.gz
2e3e88d07f07bd42457ec22bf43fae6cfe51582faae50c0e0b0d41aae2c6d809  src/bitcoin-fdb76e62ce8d.tar.gz

Copy link
Member

@theuni theuni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. I never had much luck figuring out who the true upstreams were for these tools. Much better to use something supported.

Previously, the compression of the .iso file to a .dmg file was done
outside of `make deploy' in order to use the faketime-wrapped version of
libdmg-hfsplus's DMG tool.

Specifying the faketime-wrapped version of the DMG tool to ./configure
fixes this and simplifies build scripts.
@dongcarl
Copy link
Contributor Author

dongcarl commented Dec 11, 2020

Pushed fdb76e6 -> 25d39b6: addressed #20470 (comment)

616680767fe519b013f31a8c915719d15aeb45647929a2185e9d7d4b34ed057a  bitcoin-25d39b61095d-osx-unsigned.dmg
19e44ee5454510a7ce77b61ffdfd988f55ec1da8497b0c63803605601ef45ef0  bitcoin-25d39b61095d-osx-unsigned.tar.gz
ab639cd94bcb1ca168312927e1708de84a844be94e9672951d71de1446297f39  bitcoin-25d39b61095d-osx64.tar.gz
574fac8995ab9cb900aeb263e5b920d7ae3d1385523a4dd9cbe58c56184e1761  src/bitcoin-25d39b61095d.tar.gz

@DrahtBot
Copy link
Contributor

Gitian builds

File commit 736eb4d
(master)
commit 33d40f2
(master and this pull)
*-aarch64-linux-gnu-debug.tar.gz 2be2a11a222f573c... 74c1a539c73cf28d...
*-aarch64-linux-gnu.tar.gz aa6be788b43c77ab... cce35d5a5cc98494...
*-arm-linux-gnueabihf-debug.tar.gz bc084de532268d02... e020fafd3eefd2d2...
*-arm-linux-gnueabihf.tar.gz 8bf8847238bbb772... 776c5a95b8aef35c...
*-osx-unsigned.dmg 0e0b5353ad614bc4... 91daf88d0bac46dc...
*-osx64.tar.gz 138e5e00bfec397f... 6ce3ca2f85f19001...
*-riscv64-linux-gnu-debug.tar.gz f77b9fc97e1691c7... cc61ad54b008cfed...
*-riscv64-linux-gnu.tar.gz 45248a9a8de4982e... 5204a540dbf32e48...
*-win64-debug.zip 09d982b554816f6b... b4052a10e53c7604...
*-win64-setup-unsigned.exe 3c905f96f0f756e0... 4844a665c8c9ae50...
*-win64.zip 8c7fc6e3a8ea93fe... b7772ae95be473df...
*-x86_64-linux-gnu-debug.tar.gz e47b5bfa0a774d6b... d25b8e240357d68c...
*-x86_64-linux-gnu.tar.gz fe3f8b7e4e601768... 83a4306e193a8adc...
*.tar.gz 12158696afd16c3a... c04d253bd527c8e0...
bitcoin-core-linux-22-res.yml 70d8be4a51bc67f7... bd865fc816add89a...
bitcoin-core-osx-22-res.yml 97d889ba103beac9... e6a03a2e9e8fabee...
bitcoin-core-win-22-res.yml 169a1d1fafcb841e... 7f3603516013ac23...
linux-build.log 46b0db1f12f1fed8... 5fcf65342eac3280...
osx-build.log 42dd0c03e04b575f... e19160b8ae60c902...
win-build.log 9a4cfdc2af2389a0... 70fa1e1642d1fc3a...
bitcoin-core-linux-22-res.yml.diff 13b5676969e41e52...
bitcoin-core-osx-22-res.yml.diff 9b78fa0489fca201...
bitcoin-core-win-22-res.yml.diff 0a70271891fb1fc9...
linux-build.log.diff fe013e611195940f...
osx-build.log.diff db6c123c606f097d...
win-build.log.diff 227eb1413cdf08a3...

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 25d39b6, tested cross-compiling on Linux Mint 20 (x86_64), and made the gitian build:

616680767fe519b013f31a8c915719d15aeb45647929a2185e9d7d4b34ed057a  bitcoin-25d39b61095d-osx-unsigned.dmg
19e44ee5454510a7ce77b61ffdfd988f55ec1da8497b0c63803605601ef45ef0  bitcoin-25d39b61095d-osx-unsigned.tar.gz
ab639cd94bcb1ca168312927e1708de84a844be94e9672951d71de1446297f39  bitcoin-25d39b61095d-osx64.tar.gz
574fac8995ab9cb900aeb263e5b920d7ae3d1385523a4dd9cbe58c56184e1761  src/bitcoin-25d39b61095d.tar.gz

xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.

In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.

From https://wiki.debian.org/genisoimage?action=recall&rev=11 :

> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.
@dongcarl
Copy link
Contributor Author

Pushed 25d39b6 -> 7587d11: addressed @hebasto's comments, thanks!

6853d97647d9a77e9ddb7e7fdb117917fc64eff35b3b745c99c7f8a9af21dc91  bitcoin-7587d11ec959-osx-unsigned.dmg
e1546285a738ae3a7efea5f743e0e3493c00d3132fd65ff34938783c5ec7ca15  bitcoin-7587d11ec959-osx-unsigned.tar.gz
ad7074ef60772c80330a4fcaeada2ba56e2ce0e66ad29070d8463b3228da5b9e  bitcoin-7587d11ec959-osx64.tar.gz
5fea71337d11f362b086d4424a9254df7d2ec841c7fbac123ac68b71f49ea6f9  src/bitcoin-7587d11ec959.tar.gz

Maintainers: I think this is good to merge.

@sipa
Copy link
Member

sipa commented Dec 16, 2020

Concept ACK

@laanwj laanwj merged commit 4acbcfa into bitcoin:master Dec 16, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Dec 17, 2020
@@ -135,8 +136,12 @@ $(APP_DIST_DIR)/Applications:

$(APP_DIST_EXTRAS): $(APP_DIST_DIR)/$(OSX_APP)/Contents/MacOS/Bitcoin-Qt

$(OSX_DMG): $(APP_DIST_EXTRAS)
$(GENISOIMAGE) -no-cache-inodes -D -l -probe -V "$(OSX_VOLNAME)" -no-pad -r -dir-mode 0755 -apple -o $@ dist
.INTERMEDIATE: $(OSX_TEMP_ISO)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This apparently fires a warning:

$ ./autogen.sh 
...
Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
Makefile.am:139: ... '.INTERMEDIATE' previously defined here
...

fanquake added a commit that referenced this pull request Dec 28, 2020
5e0dedb build: Define .INTERMEDIATE target once only (Hennadii Stepanov)

Pull request description:

  A new warning was introduced in 22437fc (#20470):
  ```
  $ ./autogen.sh
  ...
  Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
  Makefile.am:139: ... '.INTERMEDIATE' previously defined here
  ...
  ```

  Fixed in this PR.

ACKs for top commit:
  jonatack:
    Tested and very light review ACK 5e0dedb

Tree-SHA512: ecf8de79ba394c36ee84e0b8d3ba78587e0f856259e9731e6bbb38d0baebfd083eb44d7ef6a386dd9e4508dd64fec1c2b9a007e175fbd4d986e845b1c300a649
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Dec 28, 2020
5e0dedb build: Define .INTERMEDIATE target once only (Hennadii Stepanov)

Pull request description:

  A new warning was introduced in bitcoin@22437fc (bitcoin#20470):
  ```
  $ ./autogen.sh
  ...
  Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
  Makefile.am:139: ... '.INTERMEDIATE' previously defined here
  ...
  ```

  Fixed in this PR.

ACKs for top commit:
  jonatack:
    Tested and very light review ACK 5e0dedb

Tree-SHA512: ecf8de79ba394c36ee84e0b8d3ba78587e0f856259e9731e6bbb38d0baebfd083eb44d7ef6a386dd9e4508dd64fec1c2b9a007e175fbd4d986e845b1c300a649
fanquake added a commit to fanquake/bitcoin that referenced this pull request Jan 22, 2021
fanquake added a commit that referenced this pull request Jan 22, 2021
5b41d84 doc: add xorriso to macOS depends packages (fanquake)

Pull request description:

  This was missed in #20470.

ACKs for top commit:
  hebasto:
    ACK 5b41d84, tested on Linux Mint 20.1 (x86_64).

Tree-SHA512: bcfd8468a099c69175f8a9d295c1466764ab25d6a61121b28675a09c3e96f45b6309e1523d341f4cb21d0ddee4945f00ba060ba02da835f2f0db7e694fd6c44b
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jan 22, 2021
5b41d84 doc: add xorriso to macOS depends packages (fanquake)

Pull request description:

  This was missed in bitcoin#20470.

ACKs for top commit:
  hebasto:
    ACK 5b41d84, tested on Linux Mint 20.1 (x86_64).

Tree-SHA512: bcfd8468a099c69175f8a9d295c1466764ab25d6a61121b28675a09c3e96f45b6309e1523d341f4cb21d0ddee4945f00ba060ba02da835f2f0db7e694fd6c44b
remyers pushed a commit to remyers/bitcoin that referenced this pull request Jan 26, 2021
danben pushed a commit to danben/bitcoin that referenced this pull request Feb 1, 2021
laanwj added a commit that referenced this pull request Mar 4, 2021
c967fb7 guix: Remove libcap from manifest (Hennadii Stepanov)
7bbb409 guix: Update darwin native packages dependencies (Hennadii Stepanov)

Pull request description:

  It is a #20470 follow up.

ACKs for top commit:
  fanquake:
    ACK c967fb7

Tree-SHA512: 66ce05770f578ba61a44c58747c5a2669f425a989ed987838058bd86e3b49e342ac5a4f8852fc49f2b3a86b58fb6a340fdf3e34c1fc19bdab910729febba4bc7
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Mar 4, 2021
c967fb7 guix: Remove libcap from manifest (Hennadii Stepanov)
7bbb409 guix: Update darwin native packages dependencies (Hennadii Stepanov)

Pull request description:

  It is a bitcoin#20470 follow up.

ACKs for top commit:
  fanquake:
    ACK c967fb7

Tree-SHA512: 66ce05770f578ba61a44c58747c5a2669f425a989ed987838058bd86e3b49e342ac5a4f8852fc49f2b3a86b58fb6a340fdf3e34c1fc19bdab910729febba4bc7
kwvg pushed a commit to kwvg/dash that referenced this pull request Oct 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 2, 2021
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 2, 2021
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 2, 2021
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 2, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 5, 2021
PastaPastaPasta added a commit to dashpay/dash that referenced this pull request Oct 5, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Oct 12, 2021
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 20, 2021
5e0dedb build: Define .INTERMEDIATE target once only (Hennadii Stepanov)

Pull request description:

  A new warning was introduced in bitcoin@22437fc (bitcoin#20470):
  ```
  $ ./autogen.sh
  ...
  Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
  Makefile.am:139: ... '.INTERMEDIATE' previously defined here
  ...
  ```

  Fixed in this PR.

ACKs for top commit:
  jonatack:
    Tested and very light review ACK 5e0dedb

Tree-SHA512: ecf8de79ba394c36ee84e0b8d3ba78587e0f856259e9731e6bbb38d0baebfd083eb44d7ef6a386dd9e4508dd64fec1c2b9a007e175fbd4d986e845b1c300a649
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Oct 21, 2021
5e0dedb build: Define .INTERMEDIATE target once only (Hennadii Stepanov)

Pull request description:

  A new warning was introduced in bitcoin@22437fc (bitcoin#20470):
  ```
  $ ./autogen.sh
  ...
  Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
  Makefile.am:139: ... '.INTERMEDIATE' previously defined here
  ...
  ```

  Fixed in this PR.

ACKs for top commit:
  jonatack:
    Tested and very light review ACK 5e0dedb

Tree-SHA512: ecf8de79ba394c36ee84e0b8d3ba78587e0f856259e9731e6bbb38d0baebfd083eb44d7ef6a386dd9e4508dd64fec1c2b9a007e175fbd4d986e845b1c300a649
pravblockc pushed a commit to pravblockc/dash that referenced this pull request Nov 18, 2021
pravblockc pushed a commit to pravblockc/dash that referenced this pull request Nov 18, 2021
5e0dedb build: Define .INTERMEDIATE target once only (Hennadii Stepanov)

Pull request description:

  A new warning was introduced in bitcoin@22437fc (bitcoin#20470):
  ```
  $ ./autogen.sh
  ...
  Makefile.am:335: warning: .INTERMEDIATE was already defined in condition !BUILD_DARWIN, which is included in condition TRUE ...
  Makefile.am:139: ... '.INTERMEDIATE' previously defined here
  ...
  ```

  Fixed in this PR.

ACKs for top commit:
  jonatack:
    Tested and very light review ACK 5e0dedb

Tree-SHA512: ecf8de79ba394c36ee84e0b8d3ba78587e0f856259e9731e6bbb38d0baebfd083eb44d7ef6a386dd9e4508dd64fec1c2b9a007e175fbd4d986e845b1c300a649
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants