Skip to content

Add a global lock around the out/ folder to allow a single mill command to run at a time #3519

@lihaoyi

Description

@lihaoyi

Follow up to #3397

Mill does not have a proper concurrency model, so running multiple mill commands in parallel can result in all sorts of crashes and misbehaviors. This is especially true because IDEs tend to run Mill on their own when trying to import the project.

Here are some issues I suspect are related:

#3243 was added to address parallel evaluations within the same process via BSP, but parallel evaluations across multiple processes remains a problem.

As a first pass, we should put all Mill evaluations between a global filesystem lock. This is the approach Bazel takes, and while not perfect, it works acceptably even for very large codebases and very large engineering organizations. We can add an optional --no-global-lock flag to disable this on an opt-in basis, but given the ongoing issues we need to go for correctness first by default, which means a global lock. The opt-out flag should be enough to handle edge cases where people really need the parallelism.

Implementing #3144 and moving the IDE work to a separate folder .bsp/out will help mitigate the most common concurrency issue of someone running 1 terminal and 1 IDE, at the cost of a bit of wasted computation (hopefully not too much)

We can figure out a finer-grained concurrency story later to try and optimize this, but I expect that will take a lot more time with no guarantee of success, so we should go with the global lock first

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions