build: don't compress macOS DMG #24031

fanquake · 2022-01-11T04:09:13Z

Skip compressing the macOS DMG, and drop related build steps and dependencies. Uncompressed the DMG increases from ~16mb to ~30mb, which compared to other software a user may download, (Firefox 125mb, VLC 52mb, Open Office 176mb), is still relatively small. When contrasted against the 100's of GB of blockchain data a node will download, an additional 15mb to get the release binary, isn't much additional overhead. Note that if / when we build with LTO enabled for releases, this size will shrink back down significantly again.

native_libdmg-hfsplus is not maintained, and I doubt the DMG creation feature will ever be fixed. If at some point xorrisofs supports compressing dmgs, we could enable that.

Guix Build on x86_64:

25b7c8bb7bc8ea014d43cebb844a842d2ac8d5a343039a820d24b649c9e6bc8a  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/SHA256SUMS.part
16beb5c52c9bf51b5ce9ef5a0d17c0038238a833383586a1b14acbca78533e4b  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.dmg
d8f89a61a7448d6334dbb3639386a7b6340542393933f35421a9e6dfc724e455  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.tar.gz
11617dc261ef602433f5bb29956a40a9085dbc783f519f75fbe06e80970148d0  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin.tar.gz
aa8550d4a394d3161d14ec5e6012ed07354135afb022e905a1946785b4665664  guix-build-1dd8cbfbc631/output/dist-archive/bitcoin-1dd8cbfbc631.tar.gz
2b837f2f971a9738d0b7b8497f7ded740ef5e67c8baa7f30ca33e6b7d826eec8  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/SHA256SUMS.part
db972b2c06dbde5525a3f9e6ceb9c20a8120bc9a6f15e1d852a4bfac09d88569  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.dmg
50fe990c3f9923ee92195125faf6517396e7c1b017a8f4f7d52e991ebce52f0c  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.tar.gz
1d9022b0ae46ead41046c40f82291ce363760660a3cd6e6ef6a5b1128b90faef  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin.tar.gz

Guix Build on arm64:

DrahtBot · 2022-01-11T06:21:05Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#21778 (build: LLVM 14 & LLD based macOS toolchain by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Sjors · 2022-01-13T13:23:08Z

Concept ACK

When I build locally on macOS 12.1 and without using depends, the difference in size between this commit and its parent is negligible. Both are 20 MB. Am I doing it wrong or is there something in the Guix build that lends itself more to compression?

Update: oh wait, when building on macOS it uses completely different tools (see if BUILD_DARWIN in Makefile)

monstrito

@maldito-dot

fanquake · 2022-01-31T10:59:42Z

An additional reason for merging this is that is will fix Guix building for arm64 darwin on arm64 darwin (#24211).

fanquake · 2022-03-16T16:20:48Z

Rebased past #23565.

Sjors · 2022-03-18T11:28:15Z

The code in 6c73e2a looks reasonable, but what's the process for testing this? I assume we need detached signatures to have guix bake the DMG? If that can be done with something self-signed, what's the incantation?

jarolrod · 2022-04-11T21:43:59Z

Concept ACK

Guix hashes on x86

25b7c8bb7bc8ea014d43cebb844a842d2ac8d5a343039a820d24b649c9e6bc8a  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/SHA256SUMS.part
16beb5c52c9bf51b5ce9ef5a0d17c0038238a833383586a1b14acbca78533e4b  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.dmg
d8f89a61a7448d6334dbb3639386a7b6340542393933f35421a9e6dfc724e455  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.tar.gz
11617dc261ef602433f5bb29956a40a9085dbc783f519f75fbe06e80970148d0  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin.tar.gz
aa8550d4a394d3161d14ec5e6012ed07354135afb022e905a1946785b4665664  guix-build-1dd8cbfbc631/output/dist-archive/bitcoin-1dd8cbfbc631.tar.gz
2b837f2f971a9738d0b7b8497f7ded740ef5e67c8baa7f30ca33e6b7d826eec8  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/SHA256SUMS.part
db972b2c06dbde5525a3f9e6ceb9c20a8120bc9a6f15e1d852a4bfac09d88569  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.dmg
50fe990c3f9923ee92195125faf6517396e7c1b017a8f4f7d52e991ebce52f0c  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.tar.gz
1d9022b0ae46ead41046c40f82291ce363760660a3cd6e6ef6a5b1128b90faef  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin.tar.gz

Guix hashes on arm64

e5c954168658b4e8347b96753b78cddb84f5fd8758260e26769b35e00f15ee02  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/SHA256SUMS.part
23bf8ddde7c6850843a79843fa4c42f5d09f1dd11ed3492ed00462fd87d32583  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.dmg
f441b8695d26af1040ed5dfe7feb217561edc56bf693639774711502d27172c7  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.tar.gz
d7a3ccb47907db23f263d2f9c3ca548bb60e357d80f765a45bfdc38652030d68  guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin.tar.gz
aa8550d4a394d3161d14ec5e6012ed07354135afb022e905a1946785b4665664  guix-build-1dd8cbfbc631/output/dist-archive/bitcoin-1dd8cbfbc631.tar.gz
2b837f2f971a9738d0b7b8497f7ded740ef5e67c8baa7f30ca33e6b7d826eec8  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/SHA256SUMS.part
db972b2c06dbde5525a3f9e6ceb9c20a8120bc9a6f15e1d852a4bfac09d88569  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.dmg
50fe990c3f9923ee92195125faf6517396e7c1b017a8f4f7d52e991ebce52f0c  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.tar.gz
1d9022b0ae46ead41046c40f82291ce363760660a3cd6e6ef6a5b1128b90faef  guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin.tar.gz

Sjors · 2022-04-13T18:22:34Z

re-tACK 1dd8cbf on Intel macOS

File is about 31 MB.

Only changes compared to 6c73e2a are in Makefile.am. Guix hashes match what @jarolrod found.

laanwj · 2022-04-14T07:23:34Z

Build system changes code review ACK 1dd8cbf, I don't know anything about MacOS application formats and their internals so do not have an opinion on the contents of this change.

hebasto · 2022-04-17T19:47:23Z

I think the reason this may no-longer pop up is because there isn't a hfsplus volume, which would have a specific bit set in it's header, to tell the OS/Finder to open a window on mount. Adding -hfsplus to the xorrisofs invocation doesn't seem to fix this, although maybe we should be adding that anyway.

Looks like a bug in the xorrisofs. Has it already been known/reported by someone?

If "yes", should we modify our macOS install instructions?

Which install instructions?

I mean, the change in user's steps to install Bitcoin Core on macOS should be documented somehow. Maybe, in release notes?

While touching this code, could the deploydir target be added to .PHONY:?

jarolrod · 2022-04-19T08:09:19Z

Approach ACK. I have reviewed the code and done some testing, but still plan to do more testing before my final ack.

Confirming that the quix built dmg does not automatically open in a file window when double clicked, but does mount as expected. The DMG deployed through make deploy on a non-depends native compile build does open up automatically in a file window when double clicked.

This behavior change is fine with me, as we are getting rid of things (yay less things) and achieving cross-architecture reproducibility on a platform. If we need documentation on the behavior change, that can be picked up in a follow-up. Additionally, it could be possible to address the behavior change in a follow-up down the road.

Sjors · 2022-04-19T09:21:32Z

Confirming that the quix built dmg does not automatically open in a file window when double clicked, but does mount as expected.

We should explain to the user that they have to do "open Bitcoin-Core.dmg". Could be fixed in a followup; I'd rather not have to run Guix again :-)

fanquake · 2022-04-20T08:16:10Z

Looks like a bug in the xorrisofs. Has it already been known/reported by someone?

I'm not sure it is a bug. Aside from my not being sure if my assumption is correct, if it was, the same header bit that means something on macOS, probably doesn't mean anything on Linux for hfsplus volumes, which would likely be why the developers of the tool wouldn't have cared about this behaviour. I can look at following up.

I mean, the change in user's steps to install Bitcoin Core on macOS should be documented somehow. Maybe, in release notes?

Sure, we could add something to the release notes. I don't think it's a big deal either way.

While touching this code, could the deploydir target be added to .PHONY:?

If I touch this branch again I could do.

We should explain to the user that they have to do "open Bitcoin-Core.dmg".

They don't have to do open Bitcoin-Core.dmg. They can also just double click on it as usual, and pick the volume in Finder.

jarolrod

ACK 1dd8cbf

I've tested guix builds on x86 and arm64 to ensure that I can build and open the resulting dmg. This change looks good to me, While technically there is a behavior change in that the dmg doesn't automatically open up in the finder, I don't think this will reasonably cause any disturbance; I also don't think there will be any need for any documentation changes. One can reasonably be expected to know how to open up the dmg. The need for documentation can be revisited if there is ever an issue opened up on this topic.

No-longer required after bitcoin#24031.

fanquake · 2022-09-15T15:25:02Z

I'm going to remove "Needs release note" here (this will be in 24.0), as I don't actually think it warrants one.

jarolrod · 2022-09-26T17:48:02Z

I'm going to remove "Needs release note" here (this will be in 24.0), as I don't actually think it warrants one.

In regards to #26176, perhaps a release note is needed.

1dd8cbf build: don't compress macOS DMG (fanquake) Pull request description: Skip compressing the macOS DMG, and drop related build steps and dependencies. Uncompressed the DMG increases from ~16mb to ~30mb, which compared to other software a user may download, (Firefox 125mb, VLC 52mb, Open Office 176mb), is still relatively small. When contrasted against the 100's of GB of blockchain data a node will download, an additional 15mb to get the release binary, isn't much additional overhead. Note that if / when we build with LTO enabled for releases, this size will shrink back down significantly again. `native_libdmg-hfsplus` is not maintained, and I doubt the DMG creation feature will ever be fixed. If at some point `xorrisofs` supports compressing dmgs, we could enable that. Guix Build on x86_64: ```bash 25b7c8bb7bc8ea014d43cebb844a842d2ac8d5a343039a820d24b649c9e6bc8a guix-build-1dd8cbfbc631/output/arm64-apple-darwin/SHA256SUMS.part 16beb5c52c9bf51b5ce9ef5a0d17c0038238a833383586a1b14acbca78533e4b guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.dmg d8f89a61a7448d6334dbb3639386a7b6340542393933f35421a9e6dfc724e455 guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin-unsigned.tar.gz 11617dc261ef602433f5bb29956a40a9085dbc783f519f75fbe06e80970148d0 guix-build-1dd8cbfbc631/output/arm64-apple-darwin/bitcoin-1dd8cbfbc631-arm64-apple-darwin.tar.gz aa8550d4a394d3161d14ec5e6012ed07354135afb022e905a1946785b4665664 guix-build-1dd8cbfbc631/output/dist-archive/bitcoin-1dd8cbfbc631.tar.gz 2b837f2f971a9738d0b7b8497f7ded740ef5e67c8baa7f30ca33e6b7d826eec8 guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/SHA256SUMS.part db972b2c06dbde5525a3f9e6ceb9c20a8120bc9a6f15e1d852a4bfac09d88569 guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.dmg 50fe990c3f9923ee92195125faf6517396e7c1b017a8f4f7d52e991ebce52f0c guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin-unsigned.tar.gz 1d9022b0ae46ead41046c40f82291ce363760660a3cd6e6ef6a5b1128b90faef guix-build-1dd8cbfbc631/output/x86_64-apple-darwin/bitcoin-1dd8cbfbc631-x86_64-apple-darwin.tar.gz ``` Guix Build on arm64: ```bash ``` ACKs for top commit: Sjors: re-tACK 1dd8cbf on Intel macOS laanwj: Build system changes code review ACK 1dd8cbf, I don't know anything about MacOS application formats and their internals so do not have an opinion on the contents of this change. jarolrod: ACK bitcoin@1dd8cbf Tree-SHA512: 04c5bf78f26a9877777093ec4c50c457107bef59d720839ea5e7d7e4f7961dfee9f86b40cf791524a9e60e9e77403a797e9fcdae3849b60b759f9f66cc31b6ab

Summary: Skip compressing the macOS DMG, and drop related build steps and dependencies. Uncompressed the DMG increases from ~16mb to ~30mb, which compared to other software a user may download, (Firefox 125mb, VLC 52mb, Open Office 176mb), is still relatively small.Note that if / when we build with LTO enabled for releases, this size will shrink back down significantly again. native_libdmg-hfsplus is not maintained, and I doubt the DMG creation feature will ever be fixed. If at some point xorrisofs supports compressing dmgs, we could enable that. This is a backport of [[bitcoin/bitcoin#24031 | core#24031]] This is one step needed to make the MacOS build work with the current guix script. Test Plan: gitian-osx `HOSTS="x86_64-apple-darwin" contrib/guix/guix-build` This still fails with an exotic clang related error `ld: unrecognised emulation mode: acosx_version_min`, but it no longer fails with a `cannot find lz` error. Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D13950

fanquake mentioned this pull request Jan 11, 2022

build: Use zip instead of dmg for macOS #18128

Closed

DrahtBot added Build system Scripts and tools labels Jan 11, 2022

This was referenced Jan 11, 2022

doc: format markdown files #24019

Closed

build: Fix make apk for Android w/ non-default SOURCES_PATH in depends #22555

Closed

build: Improve depends build system robustness #22552

Closed

build: LLD based macOS toolchain #21778

Merged

monstrito reviewed Jan 18, 2022

View reviewed changes

fanquake linked an issue Jan 31, 2022 that may be closed by this pull request

guix: building for arm64-apple-darwin fails on arm64-apple (M1) #24211

Closed

fanquake force-pushed the dont_compress_macos_dmg branch from b38ef12 to abda6d4 Compare February 2, 2022 12:38

fanquake marked this pull request as ready for review February 2, 2022 13:26

DrahtBot mentioned this pull request Feb 12, 2022

doc: rewrite dependencies.md #23565

Merged

fanquake force-pushed the dont_compress_macos_dmg branch from abda6d4 to da1cc40 Compare February 22, 2022 10:40

fanquake force-pushed the dont_compress_macos_dmg branch from da1cc40 to 36493fc Compare March 9, 2022 10:27

DrahtBot mentioned this pull request Mar 14, 2022

guix: Use $HOST instead of generic osx{64} for macOS artifacts #24549

Merged

DrahtBot added the Needs rebase label Mar 16, 2022

fanquake force-pushed the dont_compress_macos_dmg branch from 36493fc to 3a0aa73 Compare March 16, 2022 16:20

DrahtBot added Needs rebase and removed Needs rebase labels Mar 16, 2022

fanquake force-pushed the dont_compress_macos_dmg branch from 3a0aa73 to 2fdbafc Compare March 16, 2022 21:01

DrahtBot removed the Needs rebase label Mar 16, 2022

This was referenced Mar 17, 2022

Add *.xip to .gitignore #24598

Closed

macdeploy: remove unused detached-sig-apply.sh #24603

Merged

DrahtBot added the Needs rebase label Mar 18, 2022

fanquake force-pushed the dont_compress_macos_dmg branch from 2fdbafc to 6c73e2a Compare March 18, 2022 11:05

hebasto mentioned this pull request Apr 17, 2022

build: No longer need to hack the PATH variable in config.site #24866

Merged

jarolrod approved these changes Apr 21, 2022

View reviewed changes

fanquake merged commit bfbce6c into bitcoin:master Apr 21, 2022

fanquake deleted the dont_compress_macos_dmg branch April 21, 2022 07:54

fanquake added the Needs release note label Apr 21, 2022

fanquake added a commit to fanquake/bitcoin that referenced this pull request Apr 21, 2022

guix: remove cmake dependency from macOS build

0b22b0e

No-longer required after bitcoin#24031.

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 22, 2022

Merge bitcoin#24031: build: don't compress macOS DMG

1abd2cf

fanquake removed the Needs release note label Sep 15, 2022

fanquake mentioned this pull request Sep 24, 2022

Opening macOS DMG does not open Finder window #26176

Closed

lyricidal mentioned this pull request Apr 1, 2023

[Upstream] build: don't compress macOS DMG PRCYCoin/PRCYCoin#461

Merged

azuchi mentioned this pull request May 18, 2023

Upgrade depends libraries chaintope/tapyrus-core#259

Merged

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

549f96d

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

0858421

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

5365419

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

59ba060

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

a95d9db

kwvg added a commit to kwvg/dash that referenced this pull request Jun 20, 2023

merge bitcoin#24031: don't compress macOS DMG

29739eb

kwvg added a commit to kwvg/dash that referenced this pull request Jun 28, 2023

merge bitcoin#24031: don't compress macOS DMG

fa60cfb

bitcoin locked and limited conversation to collaborators Sep 26, 2023

build: don't compress macOS DMG #24031

build: don't compress macOS DMG #24031

Uh oh!

Conversation

fanquake commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

Sjors commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

monstrito left a comment

Choose a reason for hiding this comment

Uh oh!

fanquake commented Jan 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanquake commented Mar 16, 2022

Uh oh!

Sjors commented Mar 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jarolrod commented Apr 11, 2022

Uh oh!

Sjors commented Apr 13, 2022

Uh oh!

laanwj commented Apr 14, 2022

Uh oh!

hebasto commented Apr 17, 2022

Uh oh!

jarolrod commented Apr 19, 2022

Uh oh!

Sjors commented Apr 19, 2022

Uh oh!

fanquake commented Apr 20, 2022

Uh oh!

jarolrod left a comment

Choose a reason for hiding this comment

Uh oh!

fanquake commented Sep 15, 2022

Uh oh!

jarolrod commented Sep 26, 2022

Uh oh!

Uh oh!

fanquake commented Jan 11, 2022 •

edited

Loading

DrahtBot commented Jan 11, 2022 •

edited

Loading

Sjors commented Jan 13, 2022 •

edited

Loading

fanquake commented Jan 31, 2022 •

edited

Loading

Sjors commented Mar 18, 2022 •

edited

Loading