Skip to content

Re-investigate parallel_map default number of CPU strategy #10868

@mtreinish

Description

@mtreinish

What should we add?

Right now the default number of CPUs used with parallel_map is the number of physical CPUs available:

CPU_COUNT = CONFIG.get("num_process", local_hardware_info()["cpus"])
which is pulled from psutil here:
"cpus": psutil.cpu_count(logical=False) or 1,

This was done at the time because benchmarking showed that there was minimal if any improvement from running parallel processes on logical cores (i.e. SMT or hyperthreading cores). However, this code was introduced ~5 years ago (see: #701), we should check that this assumption is still true on more modern hardware. We should check this on various CPU types and also different OSes (also remembering multiprocessing is disabled by default on macOS and windows, so it will need to be explicitly re-enabled to test this).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions