Skip to content

feat: using v1 runtime for async rollouts #482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Jun 10, 2025
Merged

feat: using v1 runtime for async rollouts #482

merged 14 commits into from
Jun 10, 2025

Conversation

parthchadha
Copy link
Contributor

@parthchadha parthchadha commented Jun 4, 2025 •
edited
Loading

What does this PR do ?

Screenshot 2025-06-05 at 7 37 15 AM

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

terrykong and others added 9 commits June 2, 2025 14:44
@terrykong
feat: head node in ray.sub becomes schedulable to simplify deployment
ef6f2a5 
Signed-off-by: Terry Kong <terryk@nvidia.com>
@parthchadha
fix rollouts to use async generate; fix the case we generate more tok…
3778481 
…ens than allowed

Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha
Preserve the output order for async
2099e28 
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha
Check if vllm cfg exists before checking async engine
33285b9 
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha
Merge remote-tracking branch 'origin/head-becomes-worker' into pchadh…
cdb8e46 
…a/async-perf
@parthchadha
Use vllm v1 runtime for async rollouts
717491a 
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha
Merge remote-tracking branch 'origin/main' into pchadha/async-v1
9ba6c91
@parthchadha
await for all async funcs
acf3e01 
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha
Merge remote-tracking branch 'origin/main' into pchadha/async-v1
e7653b6
terrykong
terrykong previously approved these changes Jun 5, 2025
View reviewed changes
SahilJain314
SahilJain314 previously approved these changes Jun 5, 2025
View reviewed changes
@parthchadha parthchadha added this pull request to the merge queue Jun 5, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 5, 2025
@parthchadha
Copy link
Contributor Author

CI failing due to vllm issue vllm-project/vllm#19123 (when tp > 1). This is with vllm 0.9.0, v1 runtime

@parthchadha parthchadha dismissed stale reviews from SahilJain314 and terrykong via 64742de June 9, 2025 18:30
@parthchadha
Patch vllm to provide the correct py_executable to be used
727eb01 
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@parthchadha parthchadha added the CI:L0 Run doctests and unit tests label Jun 9, 2025
terrykong
terrykong reviewed Jun 10, 2025
View reviewed changes
@parthchadha
Add capability to override v1 runtime to be disabled via NRL_VLLM_USE…
cfa3149 
…_V1=0

Signed-off-by: Parth Chadha <pchadha@nvidia.com>
@github-actions github-actions bot added the documentation Improvements or additions to documentation label Jun 10, 2025
@parthchadha parthchadha added this pull request to the merge queue Jun 10, 2025
Merged via the queue into main with commit a73dfa5 Jun 10, 2025
13 of 14 checks passed
@parthchadha parthchadha deleted the pchadha/async-v1 branch June 10, 2025 05:38
yfw pushed a commit that referenced this pull request Jun 11, 2025
@parthchadha @terrykong @yfw
feat: using v1 runtime for async rollouts (#482)
fd74d7b 
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Co-authored-by: Terry Kong <terryk@nvidia.com>
YzjiaoNvd pushed a commit to YzjiaoNvd/NeMo-RL that referenced this pull request Jul 14, 2025
@parthchadha @terrykong
feat: using v1 runtime for async rollouts (NVIDIA-NeMo#482)
87aa492 
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Co-authored-by: Terry Kong <terryk@nvidia.com>
KiddoZhu pushed a commit that referenced this pull request Jul 28, 2025
@parthchadha @terrykong @KiddoZhu
feat: using v1 runtime for async rollouts (#482)
a4fb4ea 
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Co-authored-by: Terry Kong <terryk@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@terrykong terrykong terrykong approved these changes

@SahilJain314 SahilJain314 SahilJain314 left review comments

Assignees
No one assigned
Labels
CI:L0 Run doctests and unit tests documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@parthchadha @terrykong @SahilJain314