Skip to content

Use spawn instead of fork for multiprocessing #956

@tiran

Description

@tiran

instructlab is using threads, which makes it incompatible with multiprocessing's default fork start method on Linux. Fork and threads don't mix well. They can lead to all sorts of trouble from crashes to deadlocks. The unit test warn about the problem:

tests/test_lab_generate.py: 40 warnings
  /usr/lib64/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=382568) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

Some bindings for hardware AI accelerators do not like fork, either. For example HabanaLabs documentation for Intel Gaudi HPUs states that:

Default start method is fork which may result in undefined behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions