Skip to content

[Misc] Fix Unable to detect current VLLM config. Defaulting to NHD kv cache layout warning #20400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 3, 2025
Merged

[Misc] Fix Unable to detect current VLLM config. Defaulting to NHD kv cache layout warning #20400

merged 1 commit into from
Jul 3, 2025

Conversation

NickLucche
Copy link
Contributor

This warning was popping up in scenarios where it wasn't meant to be shown, as the original purpose
was to notify of the KV layout change in disagg PD setups.

INFO 07-02 17:57:24 [default_loader.py:272] Loading weights took 0.31 seconds
INFO 07-02 17:57:25 [gpu_model_runner.py:1782] Model loading took 1.1201 GiB and 0.721210 seconds
INFO 07-02 17:57:30 [backends.py:508] Using cache directory: /home/mgoin/.cache/vllm/torch_compile_cache/a9f88f5fe5/rank_0_0/backbone for vLLM's torch.compile
INFO 07-02 17:57:30 [backends.py:519] Dynamo bytecode transform time: 4.93 s
INFO 07-02 17:57:34 [backends.py:155] Directly load the compiled graph(s) for shape None from the cache, took 3.627 s
INFO 07-02 17:57:34 [monitor.py:34] torch.compile takes 4.93 s in total
INFO 07-02 17:57:35 [gpu_worker.py:232] Available KV cache memory: 64.43 GiB
INFO 07-02 17:57:35 [kv_cache_utils.py:716] GPU KV cache size: 603,248 tokens
INFO 07-02 17:57:35 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 14.73x
# Gone from non-PD deployments
=>WARNING 07-02 17:57:35 [utils.py:101] Unable to detect current VLLM config. Defaulting to NHD kv cache layout.

@NickLucche
fix warning
d988ea5 
Signed-off-by: NickLucche <nlucches@redhat.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 2, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @NickLucche, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where a specific warning related to KV cache layout was being displayed unnecessarily in certain configurations. My changes refine the conditions under which this warning is issued, ensuring it only appears when relevant to disaggregated setups. Additionally, I've updated a related log message to reference the correct environment variable name.

Highlights

  • Warning Suppression: I've updated the logic in vllm/distributed/kv_transfer/kv_connector/utils.py to prevent the Unable to detect current VLLM config. Defaulting to NHD kv cache layout warning from appearing in scenarios where it's not relevant. Specifically, the warning will now only be triggered if KV transfer configuration (kv_config) is present but the model configuration (model_config) is missing, aligning with its intended purpose for disaggregated PD setups.
  • Log Message Correction: I've corrected a log message in vllm/v1/attention/backends/utils.py to accurately reflect the environment variable name for KV cache layout. The message now refers to VLLM_KV_CACHE_LAYOUT instead of FLASHINFER_KV_CACHE_LAYOUT.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@mergify mergify bot added the v1 label Jul 2, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 2, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses an issue where a warning about VLLM config detection was being shown in incorrect scenarios. The changes correctly scope this warning to disaggregated prefill/decode setups by checking if KV transfer is configured before issuing the warning. Additionally, a log message has been updated to reference the correct environment variable, VLLM_KV_CACHE_LAYOUT. The changes are logical, well-contained, and improve the accuracy of the logging. I don't see any issues of medium or higher severity.

mgoin
mgoin approved these changes Jul 2, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this :)

@mgoin mgoin enabled auto-merge (squash) July 2, 2025 20:17
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 2, 2025
@simon-mo simon-mo disabled auto-merge July 3, 2025 21:56
@simon-mo simon-mo merged commit 8d775dd into vllm-project:main Jul 3, 2025
52 of 56 checks passed
sfeng33 pushed a commit to sfeng33/vllm that referenced this pull request Jul 6, 2025
@NickLucche @sfeng33
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
c7ef379 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
huydhn pushed a commit to huydhn/vllm that referenced this pull request Jul 8, 2025
@NickLucche @huydhn
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
75f8a54 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
Chen-zexi pushed a commit to Chen-zexi/vllm that referenced this pull request Jul 13, 2025
@NickLucche @Chen-zexi
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
3dc5a6b 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025
@NickLucche @LyrisZhong
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
4484395 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025
@NickLucche @avigny
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
d9fd0bf 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@NickLucche
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
c550c16 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@NickLucche @npanpaliya
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
8b7f53a 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@NickLucche @jinzhen-lin
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
9b35f4e 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025
@NickLucche @googlercolin
[Misc] Fix `Unable to detect current VLLM config. Defaulting to NHD k…
060a30a 
…v cache layout` warning (vllm-project#20400)

Signed-off-by: NickLucche <nlucches@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@mgoin mgoin mgoin approved these changes

@jeejeelee jeejeelee jeejeelee approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@NickLucche @tlrmchlsmth @mgoin @jeejeelee @simon-mo