-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Closed
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray Coreperformance
Description
What happened + What you expected to happen
On a 8 core Linux server, in local mode after calling ray.init()
, ray creates 8 (idle) workers (looks reasonable), but in each of them it also creates many threads, in this case total of 33 threads in each worker:
- 3 worker.io
- 4 grpc_global_tim
- 1 task_event_buff
- 4 client.poll0
- 1 server.poll0
- 1 grpcpp_sync_ser
- 1 grpc_health_che
- 2 resolver-execut
- 2 default-execut
- 4 ray :: IDLE
- 1 timer_manager
- 8 event_engine
On a 128 core Linux server:
- 33 worker.io
- 4 grpc_global_tim
- 1 task_event_buff
- 4 client.poll0
- 1 server.poll0
- 1 grpcpp_sync_ser
- 1 grpc_health_che
- 2 resolver-execut
- 2 default-execut
- 4 ray :: IDLE
- 1 timer_manager
- 32 event_engine
total of 86 threads in one ray IDLE process. And there are 128 ray IDLE processes, a total of over 10000 threads spawned just by calling ray.init()
.
See also https://discuss.ray.io/t/too-many-threads-in-ray-worker/10881
Versions / Dependencies
$ ray --version
ray, version 2.4.0
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
Reproduction script
import ray
ray.init()
Issue Severity
Medium: It is a significant difficulty but I can work around it.
Metadata
Metadata
Assignees
Labels
P1Issue that should be fixed within a few weeksIssue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray CoreIssues that should be addressed in Ray Coreperformance