Skip to content

[Feature] Hybrid EP and TP #8590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 57 commits into from
Jul 31, 2025
Merged

[Feature] Hybrid EP and TP #8590

merged 57 commits into from
Jul 31, 2025

Conversation

ch-wan
Copy link
Collaborator

@ch-wan ch-wan commented Jul 31, 2025 •
edited
Loading

Motivation

Dependency:

Modifications

Accuracy Test

python3 -m sglang.launch_server --model-path /dev/shm/GLM-4.5-Air-FP8 --trust-remote-code --tp 4 --enable-ep-moe --base-gpu-id 4 --ep-size 2
python3 few_shot_gsm8k.py

Accuracy: 93.5%

Benchmark & Profiling

Checklist

ch-wan and others added 30 commits July 27, 2025 23:53
@ch-wan
Support EPLB in FusedMoE
22684cd
@ch-wan
format
2a5f299
@ch-wan
Update layer.py
9cb6b17
@zhyncs
Merge branch 'main' into cheng/fusedmoe-eplb
257c4b7
@ch-wan
fix mixtral
f55d09b
@ch-wan
fix
da672c6
@ch-wan
fix: many models do not support expert location metadata
7616696
@ch-wan
Merge branch 'main' into cheng/fusedmoe-eplb
f5a2e4c
@ch-wan
pylint?
663ca36
@ch-wan
fix compatibility with shared experts fusion
91128f3
@ch-wan
fix glm4_moe
39bd6f9
@ch-wan
fix glm4_moe
f3bf4d6
@ch-wan
wip
319ce4e
@ch-wan
fix
4ffd4a6
@ch-wan
wip
e927ed3
@ch-wan
wip
409b33b
@ch-wan
update
cb5821c
@ch-wan
almost ready
9c77430
@ch-wan
update sgl-kernel for ep
bc73b8e
@ch-wan
fix
5576959
@ch-wan
Merge branch 'main' into cheng/dev/ep-kernel
8ad5493
@ch-wan
Merge commit '5576959bda2f323d57d325f8afe9f1a4930e26b1' into cheng/re…
83151e3 
…factor/deprecate-ep
@ch-wan
clean up
990955c
@ch-wan
cleanup
e56530c
@ch-wan
fix
0056583
@ch-wan
cleanup
0b8ecc3
@ch-wan
Merge branch 'main' into cheng/refactor/deprecate-ep
aed31b9
@ch-wan
kernel part
b521465
@ch-wan
sgl-kernel for ep: python part
91b3c47
@ch-wan
add comment
dc4aace
@zhyncs zhyncs merged commit 7a1f7fc into sgl-project:main Jul 31, 2025
2 of 54 checks passed
@trevor-m
Copy link
Collaborator

This is also causing accuracy issues for FP4 moe path

@trevor-m trevor-m mentioned this pull request Jul 31, 2025
Merged
6 tasks
huangzhilin-hzl pushed a commit to huangzhilin-hzl/sglang that referenced this pull request Aug 1, 2025
@ch-wan
[Feature] Hybrid EP and TP (sgl-project#8590)
791e827
MahmoudAshraf97 pushed a commit to MahmoudAshraf97/sglang that referenced this pull request Aug 1, 2025
@ch-wan @MahmoudAshraf97
[Feature] Hybrid EP and TP (sgl-project#8590)
01f81f9
TianQiLin666666 pushed a commit to TianQiLin666666/sglang that referenced this pull request Aug 1, 2025
@ch-wan @TianQiLin666666
[Feature] Hybrid EP and TP (sgl-project#8590)
0b3fc07
@hubertlu-tw hubertlu-tw mentioned this pull request Aug 1, 2025
Merged
6 tasks
lifuhuang pushed a commit that referenced this pull request Aug 3, 2025
@ch-wan @lifuhuang
[Feature] Hybrid EP and TP (#8590)
8013452
ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025
@ch-wan @ssssnow
[Feature] Hybrid EP and TP (#8590)
3c6c73c
ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025
@ch-wan @ssssnow
[Feature] Hybrid EP and TP (#8590)
815caac
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025
@ch-wan @narutolhy
[Feature] Hybrid EP and TP (sgl-project#8590)
201ff4c
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 18, 2025
@ch-wan @narutolhy
[Feature] Hybrid EP and TP (sgl-project#8590)
a521127
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HandH1998 HandH1998 Awaiting requested review from HandH1998

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@FlamingoPg FlamingoPg Awaiting requested review from FlamingoPg

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

@kushanam kushanam Awaiting requested review from kushanam kushanam is a code owner

@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner

@slin1237 slin1237 Awaiting requested review from slin1237 slin1237 is a code owner

@rkooo567 rkooo567 Awaiting requested review from rkooo567 rkooo567 is a code owner

@kssteven418 kssteven418 Awaiting requested review from kssteven418 kssteven418 is a code owner

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ch-wan @trevor-m @zhyncs