Skip to content

Add think chunk #21333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Jul 24, 2025
Merged

Add think chunk #21333

merged 13 commits into from
Jul 24, 2025

Conversation

juliendenize
Copy link
Contributor

@juliendenize juliendenize requested a review from aarnphm as a code owner July 21, 2025 21:15
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added deepseek Related to DeepSeek models frontend labels Jul 21, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 21, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for a 'think' content chunk, which seems to be used for representing reasoning steps from models like Mistral. The changes involve updating chat utilities to recognize and parse this new content type, and modifying the DeepSeekR1ReasoningParser to handle special tokens from Mistral's tokenizer.

I have raised a high-severity concern regarding the modification of DeepSeekR1ReasoningParser. Adding Mistral-specific logic to a class named and registered for DeepSeek models is a design flaw that will impact maintainability and clarity. I've recommended creating a separate parser for Mistral to keep the components clean and decoupled.

juliendenize
juliendenize commented Jul 22, 2025
View reviewed changes
@juliendenize juliendenize force-pushed the add_think_chunk branch 2 times, most recently from 1ad0d70 to df19b71 Compare July 23, 2025 09:30
patrickvonplaten
patrickvonplaten reviewed Jul 23, 2025
View reviewed changes
patrickvonplaten
patrickvonplaten reviewed Jul 23, 2025
View reviewed changes
patrickvonplaten
patrickvonplaten reviewed Jul 23, 2025
View reviewed changes
patrickvonplaten
patrickvonplaten reviewed Jul 23, 2025
View reviewed changes
DarkLight1337
DarkLight1337 reviewed Jul 23, 2025
View reviewed changes
tests/reasoning/utils.py Outdated
@@ -56,29 +57,44 @@ def run_reasoning_extraction(

def run_reasoning_extraction_nonstreaming(
reasoning_parser: ReasoningParser,
model_output: list[str],
model_output: Union[list[str], list[int]],
Copy link
Member

@DarkLight1337 DarkLight1337 Jul 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like how we have combined paths when the inputs are actually not valid for all tokenizers. Perhaps @aarnphm has a better idea of how to refactor this. Since @aarnphm is OOO lately, I'm fine with this for now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added utils just for mistral so that it removes the Union, @aarnphm might have another idea but at least this avoids changing previous code.

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 23, 2025
juliendenize
juliendenize commented Jul 23, 2025
View reviewed changes
@mergify mergify bot added the ci/build label Jul 23, 2025
@mergify Mergify
Copy link

mergify bot commented Jul 23, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @juliendenize.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Jul 23, 2025
juliendenize and others added 7 commits July 23, 2025 23:01
@juliendenize
Add ThinkChunk
8d2a6d4 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
clean
f28d316 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
Add Mistral reasoning parser
ce2da5f 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
clean
5e07129 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
clean
d2511d0 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
typing
dab21e3 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
fix
95ec295 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
refactor
70cc547 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
nit
8a23934 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
Fix doc
6ecbb76 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
fix
6afdcfc 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
refactor
4750762 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@juliendenize
update requirements
e15343c 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
@mergify mergify bot removed the needs-rebase label Jul 23, 2025
@simon-mo simon-mo added this to the v0.10.0 milestone Jul 23, 2025
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) July 24, 2025 03:23
@simon-mo simon-mo disabled auto-merge July 24, 2025 04:51
@simon-mo simon-mo merged commit 6d8d0a2 into vllm-project:main Jul 24, 2025
97 of 98 checks passed
DW934 pushed a commit to DW934/vllm that referenced this pull request Jul 28, 2025
@juliendenize
Add think chunk (vllm-project#21333)
9b0d624 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: 董巍 <dongwei@U-5XFVDYCF-2058.local>
avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025
@juliendenize @avigny
Add think chunk (vllm-project#21333)
5cefc7f 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>
wenscarl pushed a commit to wenscarl/vllm that referenced this pull request Aug 4, 2025
@juliendenize @wenscarl
Add think chunk (vllm-project#21333)
3fd5153 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: shuw <shuw@nvidia.com>
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
@juliendenize @x22x22
Add think chunk (vllm-project#21333)
972af6b 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: x22x22 <wadeking@qq.com>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@juliendenize
Add think chunk (vllm-project#21333)
63ec849 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@juliendenize @npanpaliya
Add think chunk (vllm-project#21333)
9ba6f0f 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@juliendenize @jinzhen-lin
Add think chunk (vllm-project#21333)
452850c 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@juliendenize @paulpak58
Add think chunk (vllm-project#21333)
c5656e7 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Paul Pak <paulpak58@gmail.com>
taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025
@juliendenize @taneem-ibrahim
Add think chunk (vllm-project#21333)
3d7c1d5 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
BoyuanFeng pushed a commit to BoyuanFeng/vllm that referenced this pull request Aug 14, 2025
@juliendenize @BoyuanFeng
Add think chunk (vllm-project#21333)
e422597 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Boyuan Feng <boyuan@meta.com>
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@juliendenize @diegocastanibm
Add think chunk (vllm-project#21333)
6690ee5 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Diego-Castan <diego.castan@ibm.com>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 28, 2025
@juliendenize @epwalsh
Add think chunk (vllm-project#21333)
eff89bc 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025
@juliendenize @googlercolin
Add think chunk (vllm-project#21333)
6d94203 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrickvonplaten patrickvonplaten patrickvonplaten left review comments

@DarkLight1337 DarkLight1337 DarkLight1337 left review comments

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
ci/build deepseek Related to DeepSeek models frontend ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
v0.10.0
Development

Successfully merging this pull request may close these issues.
5 participants
@juliendenize @patrickvonplaten @DarkLight1337 @mgoin @simon-mo