[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count #54988

ZacAttack · 2025-07-28T22:13:32Z

Why are these changes needed?

Workers spawn extraneous threads that aren't necessary. It's been identified that a major source of thread bloat is coming from the use of grpc in the worker processes. As an incremental step towards a full resolution, this patch adds "RAY_worker_num_grpc_internal_threads" as an environment variable which is honored by the raylet when spawning workers. Setting this environment variable in your base image or in your code will limit the number of threads spawned by grpc in the worker.

An example usage of this api would look like:

import ray
from ray.util.queue import Queue as RayQueue
from ray.runtime_env import RuntimeEnv

# restrict the number of threads for grpc to 1 per worker (this is probably what you want to do)
# be sure this is set before the any call to ray.init or ray.remote
os.environ['RAY_worker_num_grpc_internal_threads'] = '1'

class SimpleActor:
    def __init__(self):
        ....

Related issue number

Closes #36936
Closes #54225

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

gemini-code-assist

Summary of Changes

Hello @ZacAttack, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've implemented a core change to address thread bloat in Ray worker processes, specifically targeting gRPC's thread usage. This change allows users to explicitly control the number of gRPC threads by setting a new environment variable, RAY_num_grpc_threads, via Ray's RuntimeEnv. This provides an incremental step towards optimizing resource utilization and reducing unnecessary thread overhead in worker processes.

Highlights

gRPC Thread Control: I've introduced a new mechanism to control the number of gRPC threads spawned by Ray workers. This is achieved by applying a custom patch to the gRPC library during the build process.
RAY_num_grpc_threads Environment Variable: This pull request enables the use of the RAY_num_grpc_threads environment variable. When this variable is set in the RuntimeEnv for a Ray worker, gRPC will use the specified value as its thread pool size. This allows for fine-grained control over thread resource consumption.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist · 2025-07-28T22:20:13Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

bazel/ray_deps_setup.bzl

thirdparty/patches/grpc-thread-reduction.patch

python/ray/tests/test_basic.py

aslonnie · 2025-08-04T22:45:22Z

(does not seem to require ray-ci to review?)

src/ray/common/ray_config_def.h

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: Zac Policzer <zacattackftw@gmail.com>

edoakes

nit: let's name the patch something like grpc-configurable-thread-count

src/ray/common/ray_config_def.h

edoakes

One nit, a merge conflict, then LGTM

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

edoakes · 2025-08-06T21:09:39Z

@ZacAttack I fixed merge conflict but please fix DCO (sign off commits)

Signed-off-by: zac <zac@anyscale.com>

israbbani

🚢 nicely done!

python/ray/tests/test_basic.py

…unt (#54988) Signed-off-by: Zac Policzer <zacattackftw@gmail.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: zac <zac@anyscale.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: sampan <sampan@anyscale.com>

…unt (ray-project#54988) Signed-off-by: Zac Policzer <zacattackftw@gmail.com> Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: zac <zac@anyscale.com> Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Andrew Grosser <dioptre@gmail.com>

ZacAttack added 2 commits July 28, 2025 21:57

[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count

6438cfc

Merge branch 'master' into grpc_patch

0cbb386

ZacAttack added the go add ONLY when ready to merge, run all tests label Jul 28, 2025

gemini-code-assist bot reviewed Jul 28, 2025

View reviewed changes

ZacAttack requested a review from a team July 28, 2025 22:22

ZacAttack added the core Issues that should be addressed in Ray Core label Jul 28, 2025

ZacAttack commented Jul 28, 2025

View reviewed changes

bazel/ray_deps_setup.bzl Outdated Show resolved Hide resolved

ZacAttack added 6 commits July 31, 2025 22:12

tweak of interface based on feedback

07f0979

fix whitespace

fde7288

fix NPE in debug

4873680

add test for the thread count reduction

f2fbf57

fix misspelling in comment

248f69a

run formatter

78978f5

jjyao reviewed Aug 1, 2025

View reviewed changes

thirdparty/patches/grpc-thread-reduction.patch Outdated Show resolved Hide resolved

dayshah reviewed Aug 1, 2025

View reviewed changes

thirdparty/patches/grpc-thread-reduction.patch Outdated Show resolved Hide resolved

dayshah reviewed Aug 1, 2025

View reviewed changes

thirdparty/patches/grpc-thread-reduction.patch Outdated Show resolved Hide resolved

ZacAttack added 4 commits August 4, 2025 15:52

address comments

2102b32

make test more lax to avoid newly introduced flakiness

4355252

Merge branch 'master' into grpc_patch

15cc551

fix test flakiness

aa60138

edoakes reviewed Aug 4, 2025

View reviewed changes

ZacAttack mentioned this pull request Aug 4, 2025

[Core] Make third party library thread count static in worker #55215

Open

ZacAttack added 4 commits August 4, 2025 20:23

address comments and add todo with ticket

f055386

tweak name

250293f

temporary commit for test run

291c7a1

also adding temporarily to run full suite

2f7d649

ZacAttack requested a review from a team as a code owner August 4, 2025 21:13

ZacAttack added 2 commits August 4, 2025 21:15

test run changes

93e0ca1

remove testing changes for nightly run

2f19087

remove release test flag

d90aaf1

aslonnie removed the request for review from a team August 4, 2025 22:44

ZacAttack added 4 commits August 5, 2025 11:06

Merge branch 'master' into grpc_patch

e3b41af

move import to make CI happy

f6a7944

Merge branch 'grpc_patch' of github.com:ZacAttack/ray into grpc_patch

b67a839

align test env variables to what's used in docker CI

64f7f4d

jjyao reviewed Aug 5, 2025

View reviewed changes

src/ray/common/ray_config_def.h Outdated Show resolved Hide resolved

Update src/ray/common/ray_config_def.h

a7cff06

Co-authored-by: Jiajun Yao <jeromeyjj@gmail.com> Signed-off-by: Zac Policzer <zacattackftw@gmail.com>

edoakes reviewed Aug 6, 2025

View reviewed changes

src/ray/common/ray_config_def.h Outdated Show resolved Hide resolved

edoakes approved these changes Aug 6, 2025

View reviewed changes

Merge branch 'master' into grpc_patch

29ba96d

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

ZacAttack added 2 commits August 6, 2025 21:50

Merge branch 'master' into grpc_patch

c5b3cd0

fix newline and sign off

fe17f9b

Signed-off-by: zac <zac@anyscale.com>

edoakes enabled auto-merge (squash) August 6, 2025 22:30

israbbani approved these changes Aug 6, 2025

View reviewed changes

python/ray/tests/test_basic.py Outdated Show resolved Hide resolved

address comments

abdb456

github-actions bot disabled auto-merge August 6, 2025 23:55

jjyao merged commit a4910fb into ray-project:master Aug 7, 2025
5 checks passed

This was referenced Aug 7, 2025

Multi thread caused pthread_create failed: resource temporatily unavailable volcengine/verl#719

Open

[RFC] Improving Ray for Post-Training / RL for LLM Projects #54021

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count #54988

[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count #54988

Uh oh!

ZacAttack commented Jul 28, 2025 •

edited by jjyao

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot commented Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aslonnie commented Aug 4, 2025

Uh oh!

Uh oh!

edoakes left a comment

Uh oh!

Uh oh!

edoakes left a comment

Uh oh!

edoakes commented Aug 6, 2025

Uh oh!

israbbani left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count #54988

[core] Patch grpc with RAY_num_grpc_threads to control grpc thread count #54988

Uh oh!

Conversation

ZacAttack commented Jul 28, 2025 • edited by jjyao Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot commented Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aslonnie commented Aug 4, 2025

Uh oh!

Uh oh!

edoakes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edoakes left a comment

Choose a reason for hiding this comment

Uh oh!

edoakes commented Aug 6, 2025

Uh oh!

israbbani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZacAttack commented Jul 28, 2025 •

edited by jjyao

Loading