[fix] fix pd disagg error of vlms #8094
Conversation
There was a problem hiding this comment.
Summary of Changes
Hello @ccw1996, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses a critical bug that caused failures when using the qwen2.5-vl model in disaggregation mode, specifically during the decoding process. The change ensures that necessary multimodal input data is correctly propagated, thereby stabilizing the system's operation for multimodal models.
Highlights
- Bug Fix: Resolves a
NoneTypeerror that occurred in the_comput_mrope__positionsfunction during the decode phase when usingqwen2.5-vlin disaggregation mode. - Data Handling: Ensures that
multimodal_inputsare properly captured and made available within theScheduleBatchby adding them to theprepare_for_prebuilt_extendmethod.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request addresses a NoneType error that occurs during disaggregated decoding for multimodal models like qwen2.5-vl. The fix involves correctly initializing the multimodal_inputs attribute on the ScheduleBatch for prebuilt extend requests, which was previously uninitialized and caused the downstream error.
The change is a targeted one-line addition that appears correct and aligns with how multimodal inputs are handled in other code paths. The fix is simple and effective.
While the fix is correct, I note that the PR checklist for adding unit tests is not checked. Adding a regression test case to verify this fix would be valuable to prevent similar issues in the future.
|
The accuracy in qwen2.5-vl is: disaggregation mode in sglang: 0.515 |
|
@JustinTong0323 @hnyls2002 @ByronHsu what shall I do next? |
|
It has been done. Does someone review it? |
|
@ccw1996 Hello，is the accuracy of qwen2.5-vl tested by benchmark/mmmu/bench_sglang.py? |
yes, i tested by this script. |
Got it, thank you! |
|
@mickqian Hello, this PR is about Qwen2.5-VL PD disaggregation. Could you please review it? It hasn't received a response for a long time. |
Motivation
When using pd disaggregation mode in qwen2.5-vl, the decode part will raise error in _comput_mrope__positions: 'NoneType' object is not subscriptable, This PR fix this bug.
Modifications
add multimodal_inputs variable in prepare_for_prebuilt_extend
Checklist