Skip to content

Add instructions for parallel gitian builds. #10968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 14, 2017
Merged

Add instructions for parallel gitian builds. #10968

merged 1 commit into from
Aug 14, 2017

Conversation

coblee
Copy link
Contributor

@coblee coblee commented Aug 1, 2017

This makes builds much faster if you have a multi-core machine.

@laanwj
Copy link
Member

laanwj commented Aug 1, 2017

utACK 36928b0

(that's awful, it only picks a single-core VM by default?)

@laanwj laanwj added the Docs label Aug 1, 2017
@coblee
Copy link
Contributor Author

coblee commented Aug 1, 2017

Yes by default, it's only a single-core VM. And gbuild defaults to "-j 2".
Now builds are 4 times faster on my machine.

laanwj
laanwj reviewed Aug 2, 2017
View reviewed changes
doc/release-process.md Outdated
@@ -111,16 +111,16 @@ The gbuild invocations below <b>DO NOT DO THIS</b> by default.
### Build and sign Bitcoin Core for Linux, Windows, and OS X:

pushd ./gitian-builder
./bin/gbuild --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
./bin/gbuild --num-make `nproc` --memory 3000 --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit dangerous - in many cases, number of concurrent C++ compiles is limited by memory, not number of cores

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.sh appears to use -j instead of --num-make. Is there a difference?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-j and --num-make are the same thing.

@laanwj what's a better way then? I do know that builds are much faster on my quad-core mbp when I run it with "--num-make 4"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could just provide the default of --num-make 2 here, which works with --memory 3000
Doubling one of the numbers means you have to double the other one too, at least for KVM, for LXC the memory limit is ignored which is why you didn't notice anything.

@coblee coblee force-pushed the coblee_multi_core branch from 36928b0 to 393e776 Compare August 3, 2017 06:36
@coblee
Copy link
Contributor Author

coblee commented Aug 3, 2017

@laanwj I made the fix you recommended. Change num-make to 2

@jonasschnelli
Copy link
Contributor

utACK 393e776

@fanquake
Copy link
Member

fanquake commented Aug 9, 2017

utACK 393e776

@maflcko
Copy link
Member

maflcko commented Aug 9, 2017

utACK 393e776, but I am not entirely happy about the binary files in our source code repo. At least it seems pngcrush can reduce the size by 30k... Would you mind fixing that up? (c.f. 7a482e4)

@fanquake
Copy link
Member

ping @coblee Could you make Marcos change so we can merge this?

@coblee coblee force-pushed the coblee_multi_core branch from 393e776 to 9483ffb Compare August 13, 2017 09:56
@coblee
Copy link
Contributor Author

coblee commented Aug 13, 2017

Done and pngcrush'd 2 other images.

@maflcko maflcko force-pushed the coblee_multi_core branch from 9483ffb to e93ff71 Compare August 13, 2017 10:31
@maflcko
Copy link
Member

maflcko commented Aug 13, 2017

Looks like my compression was slightly more efficient. (I used contrib/devtools/optimize-pngs.py)

Also, rewriting existing files in the git repo will keep the old versions around. Probably not worth trying to reduce the size...

I reverted the change to the existing files and force pushed the commit to the litecoin repo. I hope this is ok.

@maflcko maflcko added this to the 0.15.0 milestone Aug 13, 2017
@coblee
Copy link
Contributor Author

coblee commented Aug 13, 2017

Sure, np

@fanquake
Copy link
Member

ACK e93ff71

@laanwj laanwj merged commit e93ff71 into bitcoin:master Aug 14, 2017
laanwj added a commit that referenced this pull request Aug 14, 2017
@laanwj
Merge #10968: Add instructions for parallel gitian builds.
98aa3f6 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
thrasher- added a commit to thrasher-/litecoin that referenced this pull request Aug 17, 2017
@thrasher-
Litecoin: Use 2 instead of nproc for gitian builds
d991b65 
Ref: bitcoin#10968
@thrasher- thrasher- mentioned this pull request Aug 17, 2017
Merged
thrasher- added a commit to litecoin-project/litecoin that referenced this pull request Aug 17, 2017
@thrasher-
Litecoin: Use 2 instead of nproc for gitian builds (#374)
27287cd 
Ref: bitcoin#10968
@thrasher- thrasher- deleted the coblee_multi_core branch August 17, 2017 21:34
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 6, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
d0d2d3f 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 6, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
4b67772 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 6, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
4329905 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 7, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
fb2774d 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 8, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
c3446d6 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
PastaPastaPasta pushed a commit to PastaPastaPasta/dash that referenced this pull request Aug 12, 2019
@laanwj @PastaPastaPasta
Merge bitcoin#10968: Add instructions for parallel gitian builds.
19a17a5 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
barrystyle pushed a commit to PACGlobalOfficial/PAC that referenced this pull request Jan 22, 2020
@laanwj
Merge bitcoin#10968: Add instructions for parallel gitian builds.
014c94c 
e93ff71 Add instructions for multi-processor gitian builds (Charlie Lee)

Pull request description:

  This makes builds much faster if you have a multi-core machine.

Tree-SHA512: edb64c691a31a8a2e6c45f87886912d429a3fd600ebc0de2b16d44efd9d38bb92fea33c5fed207ad684b75d3247218912cab67661466a5538d534845e0808c9f
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@laanwj laanwj laanwj left review comments

+1 more reviewer

@PRabahy PRabahy PRabahy left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
Docs
Projects
None yet
Milestone
0.15.0
Development

Successfully merging this pull request may close these issues.
6 participants
@coblee @laanwj @jonasschnelli @fanquake @maflcko @PRabahy