[VLM] Add Nemotron-Nano-VL-8B-V1 support #20349

kylehh · 2025-07-02T04:41:25Z

Added support for Nemotron-Nano-VL-8B-V1 support
Added C-Radios embedding model support
- Current support is copying all cradio code from HF. But the code is actually downloaded under HF cache. Need help to use the downloaded code instead of adding code under models folder

Purpose

Test Plan

online and offline test
python examples/offline_inference/vision_language.py -m nemotron_vl
pytest tests/models/multimodal/processing/test_nemotron_vl.py

Test Result

(Optional) Documentation Update

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

github-actions · 2025-07-02T04:41:34Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

mergify · 2025-07-02T04:41:59Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kylehh.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

gemini-code-assist

Summary of Changes

Hello @kylehh, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances vLLM by incorporating support for the Nemotron-Nano-VL-8B-V1 model and C-Radios embedding models. It addresses image processing within the vLLM framework, focusing on integrating new models and managing dependencies, with a temporary workaround for accessing C-Radio code.

Highlights

Model Support: Adds support for the Nemotron-Nano-VL-8B-V1 model to vLLM.
C-Radios Embedding: Introduces support for C-Radios embedding models, including necessary adaptors and configurations.
Code Integration: Includes C-Radio code directly from Hugging Face due to challenges with accessing downloaded code from the HF cache; seeks assistance to resolve this.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request adds support for Nemotron-Nano-VL-8B-V1 and C-Radios embedding model. The C-Radios embedding model support copies code from Hugging Face, but there's a request for help to use the downloaded code instead. The changes include modifications to chat_utils.py and the addition of several new files related to cradios. The review comments identify areas for improvement in code clarity, documentation, and handling of a critical limitation.

vllm/model_executor/models/cradios/common.py

vllm/entrypoints/chat_utils.py

vllm/model_executor/models/cradios/README

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

DarkLight1337 · 2025-07-03T08:19:24Z

Please update the model according to #20355

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

mergify · 2025-07-03T20:05:14Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kylehh.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

vllm/model_executor/models/nemotron_vl.py

DarkLight1337

Otherwise LGTM, thanks for your patience!

vllm/model_executor/models/nemotron_vl.py

DarkLight1337 · 2025-07-16T09:21:38Z

Please fix the tests

Isotr0py

Overall LGTM, just some nits.

vllm/model_executor/models/registry.py

examples/offline_inference/vision_language.py

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Himanshu Jaju <hj@mistral.ai>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: x22x22 <wadeking@qq.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Paul Pak <paulpak58@gmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Diego-Castan <diego.castan@ibm.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

kylehh added 6 commits June 14, 2025 07:35

add

b6561d7

update

8c17ba4

update

7569f6a

remove bk files

be95696

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

udpate

ba0f5f4

ruff format

31c5178

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

kylehh requested a review from aarnphm as a code owner July 2, 2025 04:41

mergify bot added frontend needs-rebase labels Jul 2, 2025

gemini-code-assist bot reviewed Jul 2, 2025

View reviewed changes

kylehh changed the title ~~[VLM] Add Nemotron-Nano-VL-8B-V1 support~~ [VLM] Add Nemotron-Nano-VL-8B-V1 support (WIP) Jul 2, 2025

gemini-code-assist bot reviewed Jul 2, 2025

View reviewed changes

vllm/model_executor/models/cradios/common.py Outdated Show resolved Hide resolved

vllm/entrypoints/chat_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/models/cradios/README Outdated Show resolved Hide resolved

format

c9f2381

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

Merge branch 'main' into nemotron_vl

68e6a9d

mergify bot removed the needs-rebase label Jul 3, 2025

kylehh added 3 commits July 3, 2025 07:53

update the model according to vllm-project#20355

705a72e

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

use Automodel

56db335

format

fcbd48d

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

mergify bot added the needs-rebase label Jul 3, 2025

Merge branch 'main' into nemotron_vl

960042d

mergify bot removed the needs-rebase label Jul 3, 2025

kylehh added 2 commits July 3, 2025 13:51

format

c2171de

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

add offline example

97a2d7e

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

mergify bot added the documentation Improvements or additions to documentation label Jul 5, 2025

clean code

65d2291

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

kylehh changed the title ~~[VLM] Add Nemotron-Nano-VL-8B-V1 support (WIP)~~ [VLM] Add Nemotron-Nano-VL-8B-V1 support Jul 6, 2025

kylehh and others added 4 commits July 16, 2025 00:00

Update vllm/model_executor/models/nemotron_vl.py

70cc3b7

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Update vllm/model_executor/models/nemotron_vl.py

1e58cd2

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

derive NemotronVLProcessingInfo from BaseInternVLProcessingInfo

c2f115f

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

remove video in the offline infernce example

fa431d4

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

DarkLight1337 reviewed Jul 16, 2025

View reviewed changes

vllm/model_executor/models/nemotron_vl.py Outdated Show resolved Hide resolved

DarkLight1337 approved these changes Jul 16, 2025

View reviewed changes

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 16, 2025

jeejeelee reviewed Jul 16, 2025

View reviewed changes

vllm/model_executor/models/nemotron_vl.py Show resolved Hide resolved

Isotr0py approved these changes Jul 16, 2025

View reviewed changes

vllm/model_executor/models/registry.py Outdated Show resolved Hide resolved

examples/offline_inference/vision_language.py Outdated Show resolved Hide resolved

test dep update and nit fixes

28d022c

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

mergify bot added the ci/build label Jul 16, 2025

kylehh added 3 commits July 16, 2025 09:58

requirement/test.in update

5805b5f

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>

update test.txt according to CI

36ed357

only fix torch at the beginning

eb0e9dd

DarkLight1337 enabled auto-merge (squash) July 17, 2025 03:16

vllm-bot merged commit 4ef00b5 into vllm-project:main Jul 17, 2025
97 of 100 checks passed

kylehh deleted the nemotron_vl branch July 17, 2025 14:33

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

1749b4f

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

1f75b3a

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

687bd9f

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

9c2ddbc

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 27, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

a5d0966

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[VLM] Add Nemotron-Nano-VL-8B-V1 support (vllm-project#20349)

a5042f3

Signed-off-by: Kyle Huang <kylhuang@nvidia.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[VLM] Add Nemotron-Nano-VL-8B-V1 support #20349

[VLM] Add Nemotron-Nano-VL-8B-V1 support #20349

Uh oh!

kylehh commented Jul 2, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

mergify bot commented Jul 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Jul 3, 2025

Uh oh!

mergify bot commented Jul 3, 2025

Uh oh!

Uh oh!

DarkLight1337 left a comment

Uh oh!

Uh oh!

DarkLight1337 commented Jul 16, 2025

Uh oh!

Isotr0py left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[VLM] Add Nemotron-Nano-VL-8B-V1 support #20349

[VLM] Add Nemotron-Nano-VL-8B-V1 support #20349

Uh oh!

Conversation

kylehh commented Jul 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Jul 2, 2025

Uh oh!

mergify bot commented Jul 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Jul 3, 2025

Uh oh!

mergify bot commented Jul 3, 2025

Uh oh!

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DarkLight1337 commented Jul 16, 2025

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kylehh commented Jul 2, 2025 •

edited by github-actions bot

Loading