FWIW I've tested this on two of my projects: smaller one and one that's a little bigger and it worked well. Of course the text was a bit interleaved (when subcommands used "progress-bars") but it was perfectly readable. When one of the sub-recipes failed I saw immediately at the end what was it.

It's really cool that one can just slap a [parallel] and it's all magically faster. 🚀

Thanks!

This is working great, thanks!

I was over-zealous and made a [parallel] recipe that had 70 dependencies...just did what it was supposed to do...but my pc couldn't handle it.

I think it would make sense to be able to control how many dependencies are spawned at a time for a particular [parallel] recipe, maybe something like this:

[parallel: 10]
my-job: dep01 dep02 .. dep70

The just my-job will execute at most 10 of its own dependencies concurrently at a time, until all its dependencies have executed, or at least one of its dependencies has failed. In essence, this recipe will set the available job slots to 10, which affects only the dependencies of my-job. If dep01 is also a [parallel] recipe, then the 10 doesn't affect how many dependencies of dep01 can execute concurrently, one would have to specify a limit independently for it. Alternatively, the 10 limit could apply to the entire recipe chain, i.e. my-job and all its dependents share the limit of 10.

@W1M0R I think this is reasonable. The implementation will be a little rough. This is what the thread spawning code looks like now:

      thread::scope::<_, RunResult>(|thread_scope| {
        let mut handles = Vec::new();
        for (recipe, arguments) in evaluated {
          handles.push(thread_scope.spawn(move || {
            Self::run_recipe(
              &arguments, config, dotenv, true, ran, recipe, scopes, search,
            )
          }));
        }
        for handle in handles {
          handle
            .join()
            .map_err(|_| Error::internal("parallel dependency thread panicked"))??;
        }
        Ok(())
      })?;

In order to implement this, I think threads will have to send Results back in an std::sync::mpsc::channel, which the spawning thread can recv on, and use to keep track of how many threads are outstanding, as well as recv on when it wants to block until a thread completes.

Maybe the parallelism should depend on something on the user's machine, eg the number of cores? 10 may be a good number on your computer but others may have worse or better machines... 🤔