Skip to content

[Core] Too many threads in ray worker  #36936

@messense

Description

@messense

What happened + What you expected to happen

On a 8 core Linux server, in local mode after calling ray.init(), ray creates 8 (idle) workers (looks reasonable), but in each of them it also creates many threads, in this case total of 33 threads in each worker:

  • 3 worker.io
  • 4 grpc_global_tim
  • 1 task_event_buff
  • 4 client.poll0
  • 1 server.poll0
  • 1 grpcpp_sync_ser
  • 1 grpc_health_che
  • 2 resolver-execut
  • 2 default-execut
  • 4 ray :: IDLE
  • 1 timer_manager
  • 8 event_engine

On a 128 core Linux server:

  • 33 worker.io
  • 4 grpc_global_tim
  • 1 task_event_buff
  • 4 client.poll0
  • 1 server.poll0
  • 1 grpcpp_sync_ser
  • 1 grpc_health_che
  • 2 resolver-execut
  • 2 default-execut
  • 4 ray :: IDLE
  • 1 timer_manager
  • 32 event_engine

total of 86 threads in one ray IDLE process. And there are 128 ray IDLE processes, a total of over 10000 threads spawned just by calling ray.init().

See also https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881

Versions / Dependencies

$ ray --version
ray, version 2.4.0

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Reproduction script

import ray

ray.init()

Issue Severity

Medium: It is a significant difficulty but I can work around it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Issue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray Coreperformance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions