[Bugfix] Fix include prompt in stream response when echo=true #15233

fyuan1316 · 2025-03-20T16:18:20Z

I'm currently trying to fix the issue of "Return prompt in stream response when echo is enabled." I'm not entirely sure if the changes are complete or correct, and I would greatly appreciate it if you could spare some time to review them and provide guidance. Thank you!

FIX #15119

github-actions · 2025-03-20T16:18:28Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

tjohnson31415 · 2025-05-14T16:25:58Z

We hit this issue in our testing as well. I confirmed that this PR resolves the error and the changes look good to me!

@fyuan1316 Could you add a test-case for this to ensure it doesn't break again in the future?

mergify · 2025-05-19T16:11:12Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fyuan1316.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

fyuan1316 · 2025-05-23T15:36:12Z

@tjohnson31415
Thanks for the review and confirmation!

I've already added the test case in this PR. This should prevent future regressions.

tjohnson31415

Thanks for adding the unit test!

The CI test failure with PIL.UnidentifiedImageError looks unrelated 🤔...

One little simplification suggestion, but LGTM otherwise

vllm/entrypoints/openai/serving_completion.py

aarnphm

Thanks. LGTM.

aarnphm · 2025-06-12T07:13:27Z

vllm/entrypoints/openai/serving_completion.py

@@ -347,7 +360,7 @@ async def completion_stream_generator(
                        out_logprobs = output.logprobs

                        if not delta_text and not delta_token_ids \
-                            and not previous_num_tokens[i]:
+                                and not previous_num_tokens[i]:


let's revert the whitespace change

Finally it's reverted 😊

aarnphm

LGTM. Thanks

fyuan1316 · 2025-06-13T05:04:47Z

/retest

tjohnson31415 · 2025-06-25T16:22:49Z

The test failure has an illegal memory access 😮. Hopefully another rebase will resolve that...

Thanks for working on this fix @fyuan1316 ! Looking forward to having this merged!

shuynh2017 · 2025-06-26T11:43:19Z

The test failure has an illegal memory access 😮. Hopefully another rebase will resolve that...

Thanks for working on this fix @fyuan1316 ! Looking forward to having this merged!

Agree. Thanks @fyuan1316

Signed-off-by: Yuan Fang <yuanfang@alauda.io>

fyuan1316 · 2025-06-30T05:08:13Z

@tjohnson31415 , @shuynh2017 :
I've rebased and pushed the changes. Thanks for the review! 😃

NickLucche · 2025-06-30T15:58:10Z

vllm/entrypoints/openai/serving_completion.py

+                    if is_text_tokens_prompt(request_prompt):
+                        prompt_text = request_prompt["prompt"]
+                    else:
+                        prompt_text = None


won't pre-commit complain about ternary operator here?

NickLucche

lgtm

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io>

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io>

mergify bot added the frontend label Mar 20, 2025

DarkLight1337 requested a review from njhill March 21, 2025 04:11

mergify bot added the needs-rebase label May 19, 2025

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch from 6ae5649 to 9de90da Compare May 22, 2025 10:25

mergify bot removed the needs-rebase label May 22, 2025

fyuan1316 requested review from DarkLight1337, robertgshaw2-redhat and simon-mo as code owners May 23, 2025 15:29

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch 3 times, most recently from 6fe2eff to 3f40b08 Compare May 24, 2025 03:28

tjohnson31415 reviewed Jun 10, 2025

View reviewed changes

vllm/entrypoints/openai/serving_completion.py Outdated Show resolved Hide resolved

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch from 3f40b08 to 967ec22 Compare June 11, 2025 14:18

fyuan1316 requested a review from aarnphm as a code owner June 11, 2025 14:18

fyuan1316 requested a review from tjohnson31415 June 11, 2025 14:29

aarnphm approved these changes Jun 12, 2025

View reviewed changes

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch 2 times, most recently from 3f40b08 to 4887a46 Compare June 12, 2025 10:40

tjohnson31415 approved these changes Jun 12, 2025

View reviewed changes

aarnphm approved these changes Jun 13, 2025

View reviewed changes

aarnphm added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 13, 2025

aarnphm enabled auto-merge (squash) June 13, 2025 00:04

aarnphm removed the ready ONLY add when PR is ready to merge/full CI is needed label Jun 15, 2025

auto-merge was automatically disabled June 18, 2025 14:00
Head branch was pushed to by a user without write access

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch from 4887a46 to 26c9901 Compare June 18, 2025 14:00

[Bugfix] Fix include prompt in stream response when echo=true

a26de83

Signed-off-by: Yuan Fang <yuanfang@alauda.io>

fyuan1316 force-pushed the fix/v1-completions-stream-with-echo branch from 26c9901 to a26de83 Compare June 30, 2025 05:03

NickLucche reviewed Jun 30, 2025

View reviewed changes

NickLucche approved these changes Jun 30, 2025

View reviewed changes

aarnphm approved these changes Jun 30, 2025

View reviewed changes

aarnphm enabled auto-merge (squash) June 30, 2025 20:28

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 30, 2025

aarnphm merged commit e28533a into vllm-project:main Jul 1, 2025
81 checks passed

CSWYF3634076 pushed a commit to CSWYF3634076/vllm that referenced this pull request Jul 2, 2025

[Bugfix] Fix include prompt in stream response when echo=true (vllm-p…

dd0b82a

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io>

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

[Bugfix] Fix include prompt in stream response when echo=true (vllm-p…

c5b4420

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025

[Bugfix] Fix include prompt in stream response when echo=true (vllm-p…

f953b33

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[Bugfix] Fix include prompt in stream response when echo=true (vllm-p…

b852da6

…roject#15233) Signed-off-by: Yuan Fang <yuanfang@alauda.io>

Uh oh!

[Bugfix] Fix include prompt in stream response when echo=true #15233

[Bugfix] Fix include prompt in stream response when echo=true #15233

Uh oh!

Conversation

fyuan1316 commented Mar 20, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2025

Uh oh!

tjohnson31415 commented May 14, 2025

Uh oh!

mergify bot commented May 19, 2025

Uh oh!

fyuan1316 commented May 23, 2025

Uh oh!

tjohnson31415 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aarnphm left a comment

Choose a reason for hiding this comment

Uh oh!

aarnphm Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

fyuan1316 Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

aarnphm left a comment

Choose a reason for hiding this comment

Uh oh!

fyuan1316 commented Jun 13, 2025

Uh oh!

tjohnson31415 commented Jun 25, 2025

Uh oh!

shuynh2017 commented Jun 26, 2025

Uh oh!

fyuan1316 commented Jun 30, 2025

Uh oh!

NickLucche Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fyuan1316 commented Mar 20, 2025 •

edited by github-actions bot

Loading