Skip to content

[BugFix] make utils.current_stream thread-safety (#21252) #21253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 21, 2025
Merged

[BugFix] make utils.current_stream thread-safety (#21252) #21253

merged 1 commit into from
Jul 21, 2025

Conversation

simpx
Copy link
Contributor

@simpx simpx commented Jul 20, 2025 •
edited by github-actions bot
Loading

Essential Elements of an Effective PR Description Checklist

  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Fix #21252 , make current_stream to be thread-safety

Test Plan

run python -m pytest tests/test_utils.py::test_current_stream_multithread

Test Result

PASS

(vllm) simpx@galaxy:~/workspace/vllm (main)$ python -m pytest tests/test_utils.py::test_current_stream_multithread -v
 -s
INFO 07-20 23:02:35 [__init__.py:235] Automatically detected platform cuda.
================================================ test session starts ================================================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- /home/simpx/workspace/vllm/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/simpx/workspace/vllm
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 1 item                                                                                                    

tests/test_utils.py::test_current_stream_multithread PASSED

(Optional) Documentation Update

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 20, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

The pull request addresses a critical thread-safety issue in utils.current_stream by replacing a global variable with threading.local() for stream management. The implementation correctly uses thread-local storage to ensure that each thread has its own CUDA stream context, preventing race conditions in multi-threaded environments. The changes are well-contained and effectively resolve the reported bug.

simon-mo
simon-mo approved these changes Jul 21, 2025
View reviewed changes
Copy link
Collaborator

@simon-mo simon-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you for the fix.

@simon-mo simon-mo enabled auto-merge (squash) July 21, 2025 03:12
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 21, 2025
@simpx
Copy link
Contributor Author

simpx commented Jul 21, 2025

the check failures are caused by non-exists branch, see


[2025-07-21T04:40:27Z] ============ 7 passed, 1 skipped, 13 warnings in 1167.98s (0:19:27) ============
--
  | [2025-07-21T04:40:29Z] + pip install -U git+https://github.com/robertgshaw2-neuralmagic/lm-evaluation-harness.git@streaming-api
  | [2025-07-21T04:40:29Z] Collecting git+https://github.com/robertgshaw2-neuralmagic/lm-evaluation-harness.git@streaming-api
  | [2025-07-21T04:40:29Z]   Cloning https://github.com/robertgshaw2-neuralmagic/lm-evaluation-harness.git (to revision streaming-api) to /tmp/pip-req-build-gmtylmfw
  | [2025-07-21T04:40:29Z]   Running command git clone --filter=blob:none --quiet https://github.com/robertgshaw2-neuralmagic/lm-evaluation-harness.git /tmp/pip-req-build-gmtylmfw
  | [2025-07-21T04:40:30Z]   WARNING: Did not find branch or tag 'streaming-api', assuming revision or ref.
  | [2025-07-21T04:40:30Z]   Running command git checkout -q streaming-api
  | [2025-07-21T04:40:30Z]   error: pathspec 'streaming-api' did not match any file(s) known to git
  | [2025-07-21T04:40:30Z]   error: subprocess-exited-with-error
  | [2025-07-21T04:40:30Z]
  | [2025-07-21T04:40:30Z]   × git checkout -q streaming-api did not run successfully.
  | [2025-07-21T04:40:30Z]   │ exit code: 1
  | [2025-07-21T04:40:30Z]   ╰─> See above for output.
  | [2025-07-21T04:40:30Z]
  | [2025-07-21T04:40:30Z]   note: This error originates from a subprocess, and is likely not a problem with pip.
  | [2025-07-21T04:40:30Z] error: subprocess-exited-with-error
  | [2025-07-21T04:40:30Z]
  | [2025-07-21T04:40:30Z] × git checkout -q streaming-api did not run successfully.
  | [2025-07-21T04:40:30Z] │ exit code: 1
  | [2025-07-21T04:40:30Z] ╰─> See above for output.
  | [2025-07-21T04:40:30Z]
  | [2025-07-21T04:40:30Z] note: This error originates from a subprocess, and is likely not a problem with pip.
  | [2025-07-21T04:40:31Z] 🚨 Error: The command exited with status 1
  | [2025-07-21T04:40:31Z] user command error: The plugin docker command hook exited with status 1

Its was trying to git checkout streaming-api on https://github.com/robertgshaw2-neuralmagic/lm-evaluation-harness which has no streaming-api branch

@simon-mo I'm not sure, does this mean to be need a force merge?

@vllm-bot vllm-bot merged commit a0e827e into vllm-project:main Jul 21, 2025
73 of 75 checks passed
LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025
@simpx @LyrisZhong
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
b8b6812 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025
@simpx @avigny
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
4ef1205 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>
x22x22 pushed a commit to x22x22/vllm that referenced this pull request Aug 5, 2025
@simpx @x22x22
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
2ccd243 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Signed-off-by: x22x22 <wadeking@qq.com>
Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025
@simpx
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
4617ab2 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025
@simpx @npanpaliya
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
c5beafc 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025
@simpx @jinzhen-lin
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
bc6ac1a 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025
@simpx @paulpak58
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
d5f318a 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Signed-off-by: Paul Pak <paulpak58@gmail.com>
taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025
@simpx @taneem-ibrahim
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
1544d52 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025
@simpx @diegocastanibm
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
c3748e2 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Signed-off-by: Diego-Castan <diego.castan@ibm.com>
epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 27, 2025
@simpx @epwalsh
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
1439671 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025
@simpx @googlercolin
[BugFix] make utils.current_stream thread-safety (vllm-project#21252) (…
6de8eec 
…vllm-project#21253)

Signed-off-by: simpx <simpxx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simon-mo simon-mo simon-mo approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: pynccl leads to incorrect data in multi-thread GPU-worker
3 participants
@simpx @simon-mo @vllm-bot