Skip to content

[PD Perf] replace Queue to FastQueue #6649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
May 28, 2025
Merged

[PD Perf] replace Queue to FastQueue #6649

merged 9 commits into from
May 28, 2025

Conversation

whybeyoung
Copy link
Collaborator

@whybeyoung whybeyoung commented May 27, 2025 •
edited
Loading

The original while True queue polling implementation caused some CPU overhead under high concurrency. To minimize CPU usage, a more efficient approach using a deque combined with a Condition mechanism was adopted.

-            while True:
-                try:
-                    kv_chunk: TransferKVChunk = self.transfer_queue.get(timeout=0.01)
-                    reqs_to_be_processed = (
-                        self.transfer_infos[kv_chunk.room].values()
-                        if kv_chunk.room in self.transfer_infos

See the commit ..

CC @ByronHsu @ShangmingCai @zhyncs @fzyzcjy

zhyncs
zhyncs reviewed May 27, 2025
View reviewed changes
@zhyncs
Copy link
Member

zhyncs commented May 27, 2025

@whybeyoung May you help fix the conflicts? QQ can you also share some performance results?

ShangmingCai
ShangmingCai reviewed May 27, 2025
View reviewed changes
ShangmingCai
ShangmingCai reviewed May 27, 2025
View reviewed changes
@ShangmingCai
Fix shard rule and remove inner loop thread pool
0f7548f 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
Merge branch 'main' into deque-conn
7b63211
@ShangmingCai
Merge branch 'main' into deque-conn
c5b5e85
@ShangmingCai
Add per queue thread pool back and add a queue size env var
a4aed03 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
Add env var assertion
bd5d39d 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@ShangmingCai
Merge branch 'main' into deque-conn
0046fca
@ShangmingCai
fix merge
7c16813 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
@zhyncs zhyncs merged commit 6b23132 into sgl-project:main May 28, 2025
30 of 40 checks passed
@Hongbosherlock Hongbosherlock mentioned this pull request May 28, 2025
Merged
6 tasks
ChangyiYang pushed a commit to ChangyiYang/sglang-changyi that referenced this pull request May 29, 2025
@whybeyoung @ShangmingCai @ChangyiYang
[PD Perf] replace Queue to FastQueue (sgl-project#6649)
bdab9a1 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
Layssy pushed a commit to Layssy/sglang-iaas that referenced this pull request Jun 9, 2025
@whybeyoung @ShangmingCai @Layssy
[PD Perf] replace Queue to FastQueue (sgl-project#6649)
e64a2ed 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
xwu-intel pushed a commit to xwu-intel/sglang that referenced this pull request Jun 17, 2025
@whybeyoung @ShangmingCai @xwu-intel
[PD Perf] replace Queue to FastQueue (sgl-project#6649)
da608b7 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ShangmingCai ShangmingCai ShangmingCai left review comments

@zhyncs zhyncs zhyncs approved these changes

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

Assignees

@hnyls2002 hnyls2002

@ShangmingCai ShangmingCai

Labels
high priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@whybeyoung @zhyncs @ShangmingCai @hnyls2002