NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL #2882
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kashif
approved these changes
Jun 3, 2025
lewtun
approved these changes
Jun 3, 2025
| @@ -6087,3 +6087,17 @@ | |||
| - nlp | |||
| - tools | |||
| - community | |||
|
|
|||
| - local: vllm-colocate | |||
| title: "NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" | |||
There was a problem hiding this comment.
Not sure if it was intentional, but I'd use this for "NO"
Suggested change
| title: "NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" | |
| title: "No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" |
| @@ -0,0 +1,384 @@ | |||
| --- | |||
| title: "NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" | |||
There was a problem hiding this comment.
Suggested change
| title: "NO GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" | |
| title: "No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL" |
|
|
||
| ## 🚀 Introduction | ||
|
|
||
| TRL supports training LLMs using GRPO, an online learning algorithm recently introduced in the *DeepSeekMath* paper. In GRPO, the model learns from its own outputs: it generates responses during training, receives feedback, and uses that feedback to improve itself over time. |
There was a problem hiding this comment.
Would be nice to have a link to the paper & perhaps share the diagram of the algorithm (either from the paper or @qgallouedec's example from the TRL docs)
|
|
||
| ## 🛠️ Implementation Notes | ||
|
|
||
| Instead of launching vLLM as a server, the training now launches vLLM **in-process** using the external launcher, as shown below: |
There was a problem hiding this comment.
Maybe point to the bits of code in TRL where this comes from?
| ) | ||
| ``` | ||
|
|
||
| > Note: Depending on the model size and the overall GPU memory requirements for training, you may need to adjust the vllm_gpu_memory_utilization parameter in `GRPOConfig` to avoid underutilization or out-of-memory errors. |
There was a problem hiding this comment.
Is there a rough heuristic we can recommend for common model sizes like 7B / 14B / 32B / 70B?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Congratulations! You've made it this far! Once merged, the article will appear at https://huggingface.co/blog. Official articles
require additional reviews. Alternatively, you can write a community article following the process here.
Preparing the Article
You're not quite done yet, though. Please make sure to follow this process (as documented here):
mdfile. You can also specifyguestororgfor the authors.Here is an example of a complete PR: #2382
Getting a Review
Please make sure to get a review from someone on your team or a co-author.
Once this is done and once all the steps above are completed, you should be able to merge.
There is no need for additional reviews if you and your co-authors are happy and meet all of the above.
Feel free to add @pcuenca as a reviewer if you want a final check. Keep in mind he'll be biased toward light reviews
(e.g., check for proper metadata) rather than content reviews unless explicitly asked.