Skip to content

feat: parametrize GPUS_PER_NODE and CPUS_PER_WORKER in ray.sub #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 23, 2025
Merged

feat: parametrize GPUS_PER_NODE and CPUS_PER_WORKER in ray.sub #410

merged 6 commits into from
May 23, 2025

Conversation

terrykong
Copy link
Contributor

@terrykong terrykong commented May 18, 2025 •
edited
Loading

image

Closes #309

@terrykong
feat: parametrize GPUS_PER_NODE and CPUS_PER_WORKER in ray.sub
39d9f9b 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 18, 2025
@terrykong terrykong mentioned this pull request May 18, 2025
Closed
SahilJain314
SahilJain314 reviewed May 19, 2025
View reviewed changes
Copy link
Contributor

@SahilJain314 SahilJain314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're getting to a lot of 'knobs' in ray.sub. Maybe it's time to add a doc for it? It didn't seem obvious to me that envvars like HF_HOME and WANDB_API_KEY would get plumbed through ray.sub and now we're adding GPUS_PER_NODE and CPUS_PER_WORKER too.

@terrykong
Copy link
Contributor Author

I'll address the UV_CACHE_DIR in a follow up PR #426

terrykong added 2 commits May 20, 2025 23:06
@terrykong
cluster.md
629fc14 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong
touchup
3779563 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong terrykong requested a review from jgerh May 21, 2025 07:03
@terrykong terrykong mentioned this pull request May 21, 2025
Merged
4 tasks
SahilJain314
SahilJain314 previously approved these changes May 21, 2025
View reviewed changes
jgerh
jgerh reviewed May 21, 2025
View reviewed changes
Copy link
Contributor

@jgerh jgerh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed the Tech Pubs review of docs/cluster.md and provided some copyedits and suggested text revisions. Comments added inline with the "add a suggestion" tool as well as line-by-line for read-only text.

@terrykong @jgerh
Update docs/cluster.md
a9f8170 
Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
@terrykong
edit
66c5d0b 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong terrykong enabled auto-merge May 22, 2025 20:38
SahilJain314
SahilJain314 reviewed May 22, 2025
View reviewed changes
@terrykong
restore kubernetes line
1387e0b 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong terrykong added this pull request to the merge queue May 22, 2025
@parthchadha parthchadha removed this pull request from the merge queue due to a manual request May 22, 2025
@SahilJain314 SahilJain314 added this pull request to the merge queue May 23, 2025
Merged via the queue into main with commit f9e45de May 23, 2025
13 of 14 checks passed
@SahilJain314 SahilJain314 deleted the tk/cpu-task branch May 23, 2025 04:43
YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jun 10, 2025
@terrykong @jgerh
feat: parametrize GPUS_PER_NODE and CPUS_PER_WORKER in ray.sub (NVIDI…
49f083b 
…A-NeMo#410)

Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terrycurtiskong@gmail.com>
Co-authored-by: jgerh <163925524+jgerh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SahilJain314 SahilJain314 SahilJain314 left review comments

@jgerh jgerh jgerh left review comments

@parthchadha parthchadha parthchadha approved these changes

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

init_ray's runtime_env (with full os.environ) causes Ray runtime_env_agent to fail
4 participants
@terrykong @parthchadha @SahilJain314 @jgerh