Skip to content

Allow building multiple build targets in parallel #545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

carlhoerberg
Copy link

Pass -j X or --jobs=X to start X build process in parallel to speed up
building applications with multiple targets (with the cost of more
memory).

Pass -j X or --jobs=X to start X build process in parallel to speed up
building applications with multiple targets (with the cost of more
memory).
@Blacksmoke16
Copy link
Member

Might be better to contribute to the discussion/progress of #539 instead of introducing a new approach?

@asterite
Copy link
Member

I was thinking about that... but I think this is about building things in parallel, while the other one is fetching things in parallel... which of course will be very confusing if we have two flags to control these.

@Blacksmoke16
Copy link
Member

Oh sorry, I didn't notice this was for building. But yea, that's a good point. Maybe the flag could be consolidated/generalized to support both cases?

@carlhoerberg
Copy link
Author

Maybe, but having a default > 1 might cause memory issues as each build process uses quite a lot of memory. Hence you might want different defaults for --jobs and --parallel-fetch..

@straight-shoota
Copy link
Member

I don't like this because I don't want shards to become more of a build tool. It should focus on managing dependencies, and IMO does not need advanced build features.

And there are many alternatives. If you use an actual build system (which I would recommend for any project), it probably already has an option for parallel building (make -j for example).
This is also easy to do with the parallel command: parallel shards build ::: target1 target2.

@m-o-e
Copy link
Contributor

m-o-e commented Feb 25, 2022

might cause memory issues as each build process uses quite a lot of memory
Hence you might want different defaults for --jobs and --parallel-fetch

Agree. If both get added they definitely need to have two separate flags.

In bundler --jobs means: The maximum number of parallel download and install jobs. (man-page)
In make --jobs means: Specifies the number of jobs (commands) to run simultaneously. (man-page)

Our own --jobs flag can only be compatible with one of them, not both at the same time.
(as the two tend to have different optimal values, esp. on low-memory machines)

Since shards is conceptually closer to bundler than make, I would vote for
adopting the bundler meaning of --jobs (as already done in #539).

If this one should also be added then I'd suggest to call the flag here --parallel-build (or similar).

I don't want shards to become more of a build tool.

I'd be inclined to agree with this as well. The default for this will probably always have to be 1.
Meaning anyone who wants to use it will have to change their invocation.

And at that point they could as well replace it with something involving parallel.

I suppose the argument for having this functionality baked in could be
to make it more discoverable/easier, tho. 🤔

@hugopl
Copy link

hugopl commented Oct 31, 2023

This is that kind of feature that at first looks very nice, but then you notice that we are crossing the borders between a dependency manager and a building tool... years later you notice that you just wrote a bad version of make.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants