Skip to content

[Core] Add update_config RPC method #20095

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Jul 14, 2025
Merged

[Core] Add update_config RPC method #20095

merged 18 commits into from
Jul 14, 2025

Conversation

22quinn
Copy link
Collaborator

@22quinn 22quinn commented Jun 25, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Part of #19886

Test Plan

# Create an LLM without loading real weights
llm = LLM(
    model="facebook/opt-125m",
    load_format="dummy",
    enforce_eager=True,
)

llm.collective_rpc("update_config", args=({"load_config": {"load_format": "auto"}},))
llm.collective_rpc("load_model")

# Check real weights are loaded
outputs = llm.generate(prompts, sampling_params)
print("\nGenerated Outputs:\n" + "-" * 60)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt:    {prompt!r}")
    print(f"Output:    {generated_text!r}")
    print("-" * 60)

Test Result

Generated Outputs:
------------------------------------------------------------
Prompt:    'Hello, my name is'
Output:    ' Joel, my dad is my friend and we are in a relationship. I am'
------------------------------------------------------------
Prompt:    'The president of the United States is'
Output:    ' reportedly being investigated by the FBI over the allegations that he broke campaign finance laws in'
------------------------------------------------------------
Prompt:    'The capital of France is'
Output:    ' all set to open a new gateway to the most notorious terror camps in Europe,'
------------------------------------------------------------
Prompt:    'The future of AI is'
Output:    ' in their hands. We cannot prevent the future.\nAgreed.  '
------------------------------------------------------------

(Optional) Documentation Update

22quinn added 3 commits June 19, 2025 17:57
@22quinn
Add update_load_config RPC
9b56a6f 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
Merge branch 'main' into rpc-update-load-config
4709e7a 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
update_config RPC
e17a929 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 25, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @22quinn, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new RPC method, update_config, designed to provide greater flexibility in managing model configurations within the vLLM framework. This enhancement allows for programmatic adjustments to various configuration objects on both GPU and TPU model runners, enabling dynamic changes such as altering model loading behavior after initialization but prior to the actual model weights being loaded.

Highlights

  • New RPC Method: I've added a new RPC method, update_config, to the core vLLM system. This method allows for dynamic modification of configuration objects (e.g., load_config) on ModelRunner instances.
  • Configuration Update Mechanism: The update_config method is implemented to iterate through provided overrides, retrieve the existing configuration object, and then use dataclasses.replace to create a new configuration with the specified changes. This ensures that configuration objects are updated immutably and correctly.
  • Worker Integration: The GPUWorker and TPUWorker classes now expose the update_config method, which simply delegates the call to their respective model_runner instances. This makes the new configuration update functionality accessible at the worker level, enabling scenarios like updating load_format before model loading.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@mergify mergify bot added v1 tpu Related to Google TPUs labels Jun 25, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jun 25, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request introduces an update_config RPC method to both GPU and TPU model runners, allowing for dynamic updates to configuration parameters at runtime. The code includes basic error handling for unknown configurations but lacks type validation for the override values, which could lead to runtime issues. Consider adding type validation to ensure the override values match the expected types.

@22quinn
unit test
7129f12 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn marked this pull request as ready for review June 25, 2025 21:40
aarnphm
aarnphm reviewed Jun 26, 2025
View reviewed changes
aarnphm
aarnphm reviewed Jun 26, 2025
View reviewed changes
aarnphm
aarnphm approved these changes Jun 26, 2025
View reviewed changes
Copy link
Collaborator

@aarnphm aarnphm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like we might want to merge recursively here. might be good to have a utils for it to implement deepmerge 🤔

aarnphm
aarnphm reviewed Jun 26, 2025
View reviewed changes
Copy link
Collaborator

@aarnphm aarnphm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments

houseroad
houseroad reviewed Jun 26, 2025
View reviewed changes
houseroad
houseroad approved these changes Jun 26, 2025
View reviewed changes
Copy link
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, looks good to me.

@houseroad houseroad added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 26, 2025
@ProExpertProg
Copy link
Collaborator

Do we have tests that this works correctly with quantization and torch compilation?

22quinn added 2 commits June 26, 2025 13:34
@22quinn
Merge branch 'main' into rpc-update-config
ee6b2bb 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
unit test
9a8d179 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn
tpu todo
3d1ef20 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@mergify Mergify
Copy link

mergify bot commented Jul 12, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @22quinn.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jul 12, 2025
@22quinn
rebase
ef7fc53 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@mergify mergify bot removed the needs-rebase label Jul 13, 2025
@22quinn
Copy link
Collaborator Author

22quinn commented Jul 13, 2025 •
edited
Loading

Do we have tests that this works correctly with quantization and torch compilation?

@ProExpertProg I've tested torch.compile works fine via: https://gist.github.com/22quinn/c2ace183a56d0801e160cb133b436bd1
However, quantization is not supported for now. There are some users asking for it. I'll dig a bit and see how to support it in a separate PR.
For now, I want to finish up the basic in-place update functionality. Once basic features are complete and APIs are stable, I'll add an E2E test and example.

@22quinn
rebase
7776bc2 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn requested a review from houseroad July 13, 2025 23:58
@houseroad houseroad enabled auto-merge (squash) July 14, 2025 00:01
@houseroad houseroad merged commit 8632e83 into vllm-project:main Jul 14, 2025
65 checks passed
b8zhong pushed a commit to bzhng-development/vllm that referenced this pull request Jul 14, 2025
@22quinn @b8zhong
[Core] Add update_config RPC method (vllm-project#20095)
b54d0a1 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Brayden Zhong <b8zhong@uwaterloo.ca>
b8zhong pushed a commit to bzhng-development/vllm that referenced this pull request Jul 14, 2025
@22quinn @b8zhong
[Core] Add update_config RPC method (vllm-project#20095)
c519c34 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn mentioned this pull request Jul 14, 2025
Closed
4 tasks
py-andy-c pushed a commit to py-andy-c/vllm that referenced this pull request Jul 14, 2025
@22quinn @py-andy-c
[Core] Add update_config RPC method (vllm-project#20095)
e545a3a 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
py-andy-c pushed a commit to py-andy-c/vllm that referenced this pull request Jul 14, 2025
@22quinn @py-andy-c
[Core] Add update_config RPC method (vllm-project#20095)
7308b25 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
patrickvonplaten pushed a commit to patrickvonplaten/vllm that referenced this pull request Jul 15, 2025
@22quinn @patrickvonplaten
[Core] Add update_config RPC method (vllm-project#20095)
13709db 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025
@22quinn @LyrisZhong
[Core] Add update_config RPC method (vllm-project#20095)
e09eb69 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@wangxiyuan wangxiyuan mentioned this pull request Jul 24, 2025
Open
avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025
@22quinn @avigny
[Core] Add update_config RPC method (vllm-project#20095)
ff8c04c 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
@22quinn @x22x22
[Core] Add update_config RPC method (vllm-project#20095)
70bf3f0 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@22quinn
[Core] Add update_config RPC method (vllm-project#20095)
0aff59a 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@22quinn @npanpaliya
[Core] Add update_config RPC method (vllm-project#20095)
c68123f 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@22quinn @jinzhen-lin
[Core] Add update_config RPC method (vllm-project#20095)
7d0db40 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@22quinn @paulpak58
[Core] Add update_config RPC method (vllm-project#20095)
193811f 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Paul Pak <paulpak58@gmail.com>
taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025
@22quinn @taneem-ibrahim
[Core] Add update_config RPC method (vllm-project#20095)
86117b4 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
@22quinn 22quinn added the rl Related to RL workflows label Aug 14, 2025
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@22quinn @diegocastanibm
[Core] Add update_config RPC method (vllm-project#20095)
0699c49 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Signed-off-by: Diego-Castan <diego.castan@ibm.com>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 27, 2025
@22quinn @epwalsh
[Core] Add update_config RPC method (vllm-project#20095)
236bdb8 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025
@22quinn @googlercolin
[Core] Add update_config RPC method (vllm-project#20095)
1926810 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yaochengji yaochengji yaochengji left review comments

@houseroad houseroad houseroad approved these changes

@jeejeelee jeejeelee Awaiting requested review from jeejeelee

@youkaichao youkaichao Awaiting requested review from youkaichao

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo

@mgoin mgoin Awaiting requested review from mgoin

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth

@hmellor hmellor Awaiting requested review from hmellor

@aarnphm aarnphm Awaiting requested review from aarnphm

+2 more reviewers

@yeqcharlotte yeqcharlotte yeqcharlotte left review comments

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed rl Related to RL workflows tpu Related to Google TPUs v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@22quinn @ProExpertProg @yeqcharlotte @yaochengji @aarnphm @houseroad