Skip to content

[Feature] Support Tensor Parallelism and Weight Slicing for Lora #4239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 264 commits into from
Closed

[Feature] Support Tensor Parallelism and Weight Slicing for Lora #4239

wants to merge 264 commits into from

Conversation

aoshen524
Copy link
Contributor

@aoshen524 aoshen524 commented Mar 9, 2025 •
edited by Fridge003
Loading

Motivation

#3414 reports issues regarding limited model support compared to test_generation_models.py. This PR introduces tensor parallelism and weight slicing for LoRA, alongside additional improvements to testing and functionality.

Modifications

  • Implemented tensor parallelism support in LoRA, allowing efficient distribution of computations across multiple devices.
  • Introduced LoRA weight slicing and refactor memory pool to facilitate distributed inference, optimizing memory usage and performance.

checklist:

  • Remove tensor.contiguous() used in GPU

zhaochenyang20 and others added 13 commits February 19, 2025 16:01
@Fridge003 Fridge003 changed the title Feature/lora [Feature] Support Tensor Parallelism and Weight Slicing for Lora Mar 9, 2025
This was referenced Mar 9, 2025
Closed
Open
merrymercy and others added 22 commits March 11, 2025 04:05
@merrymercy @ShenAo1111
Minor style fix for sgl-kernel (sgl-project#4243)
a43e55a
@merrymercy @ShenAo1111
Auto balance CI tests (sgl-project#4238)
98decee
@merrymercy @ShenAo1111
Clean up fp8 support (sgl-project#4230)
5730180
@merrymercy @ShenAo1111
Move activation.cu to sgl-kernel/elementwise (sgl-project#4250)
110b0ad
@laixinn @sleepcoo
@HandH1998 @shuaills @FlamingoPg @zhyncs @ShenAo1111
DeepGemm integrate to sgl-kernel (sgl-project#4165)
69583c9 
Co-authored-by: sleepcoo <sleepcoo@gmail.com>
Co-authored-by: HandH1998 <1335248067@qq.com>
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
Co-authored-by: yinfan98 <1106310035@qq.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
@DavidChan0519 @ShenAo1111
[Bug fixed] fixed the crash when enable the dp-attention on the singl…
122a4b3 
…e card (sgl-project#3958)
@simveit @zhaochenyang20 @ShenAo1111
Added example for multimodal embedding (sgl-project#4206)
d6666a6 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
@merrymercy @ShenAo1111
Simplify tests & Fix trtllm custom allreduce registration (sgl-projec…
bd58fd8 
…t#4252)
@Young1993 @ShenAo1111
fix the input_ids is None error (sgl-project#4144)
4bf7b08
@BBuf @ShenAo1111
fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0 (
5b7a02a 
sgl-project#4231)
@merrymercy @ShenAo1111
Release sgl-kernel v0.0.4.post1 (sgl-project#4255)
9413891
@merrymercy @ShenAo1111
Fix quantization and nightly tests (sgl-project#4258)
95145b6
@merrymercy @ShenAo1111
increase the timeout of nightly-test.yml (sgl-project#4262)
c3b8b97
@merrymercy @ShenAo1111
Optimize rope in sgl kernel (sgl-project#4267)
c2f851e
@merrymercy @ShenAo1111
Test no vllm custom allreduce (sgl-project#4256)
80ba9b3
@HandH1998 @ShenAo1111
Amd test fp8 (sgl-project#4261)
28e811d
@zhyncs @ShenAo1111
add THIRDPARTYNOTICES for DeepGEMM (sgl-project#4272)
2c5b08e
@ShenAo1111
fix(lora): improve handling of gate projections in LoRA weights.
6f1d7d5
@ShenAo1111
style(lora): format LoRA codebase with pre-commit tool.
911b334
@ShenAo1111
perf(sglang): adjust static memory fraction to original value in main…
15ba985 
… branch.
@ShenAo1111
test(srt): update LoRA test tolerances and configurations.
57bc3c7
@ShenAo1111
feat(lora): add support for tensor parallelism in LoRA
dc9ad0a
@Fridge003 Fridge003 closed this Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@HandH1998 HandH1998 Awaiting requested review from HandH1998

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu ByronHsu is a code owner

@rkooo567 rkooo567 Awaiting requested review from rkooo567 rkooo567 is a code owner

@kssteven418 kssteven418 Awaiting requested review from kssteven418 kssteven418 is a code owner

@zhaochenyang20 zhaochenyang20 Awaiting requested review from zhaochenyang20

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.