[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion #7181

Alcanderian · 2025-06-14T12:02:32Z

Motivation

Using weights = [w for w in weights_list if w[0] not in names_to_remove] to update weights will change it from async weight iterator to a list. Thats will make the tqdm looks very fast but the weight are still loading in the backround. Making the progress bar confusing.

acc

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:38<00:00, 34.70it/s]
Accuracy: 0.941
Invalid: 0.000
Latency: 39.257 s
Output throughput: 3658.625 token/s

after: from 11:54:08 to 11:54:59

[2025-06-14 11:54:08 TP0] Detected fp8 checkpoint.
[2025-06-14 11:54:08 TP2] Shared experts fusion optimization enabled.
[2025-06-14 11:54:08 TP6] Shared experts fusion optimization enabled.
[2025-06-14 11:54:08 TP7] Shared experts fusion optimization enabled.
[2025-06-14 11:54:08 TP1] Shared experts fusion optimization enabled.
[2025-06-14 11:54:08 TP0] Shared experts fusion optimization enabled.
Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]
[2025-06-14 11:54:08 TP3] Shared experts fusion optimization enabled.
[2025-06-14 11:54:08 TP5] Shared experts fusion optimization enabled.
Loading safetensors checkpoint shards:   2% Completed | 4/163 [00:00<00:11, 14.18it/s]
[2025-06-14 11:54:09 TP4] Shared experts fusion optimization enabled.
Loading safetensors checkpoint shards:   4% Completed | 6/163 [00:00<00:29,  5.30it/s]
Loading safetensors checkpoint shards:   4% Completed | 7/163 [00:01<00:34,  4.57it/s]
Loading safetensors checkpoint shards:   5% Completed | 8/163 [00:01<00:39,  3.97it/s]
Loading safetensors checkpoint shards:   6% Completed | 9/163 [00:02<00:45,  3.42it/s]
Loading safetensors checkpoint shards:   6% Completed | 10/163 [00:02<00:44,  3.40it/s]
Loading safetensors checkpoint shards:   7% Completed | 11/163 [00:02<00:46,  3.29it/s]
Loading safetensors checkpoint shards:   7% Completed | 12/163 [00:03<00:49,  3.06it/s]
Loading safetensors checkpoint shards:   8% Completed | 13/163 [00:03<00:47,  3.15it/s]
Loading safetensors checkpoint shards:   9% Completed | 14/163 [00:03<00:49,  2.99it/s]
Loading safetensors checkpoint shards:   9% Completed | 15/163 [00:04<00:48,  3.07it/s]
Loading safetensors checkpoint shards:  10% Completed | 16/163 [00:04<00:48,  3.02it/s]
Loading safetensors checkpoint shards:  10% Completed | 17/163 [00:04<00:50,  2.87it/s]
Loading safetensors checkpoint shards:  11% Completed | 18/163 [00:05<00:48,  3.00it/s]
Loading safetensors checkpoint shards:  12% Completed | 19/163 [00:05<00:48,  2.96it/s]
Loading safetensors checkpoint shards:  12% Completed | 20/163 [00:05<00:50,  2.82it/s]
Loading safetensors checkpoint shards:  13% Completed | 21/163 [00:06<00:47,  3.01it/s]
Loading safetensors checkpoint shards:  13% Completed | 22/163 [00:06<00:49,  2.85it/s]
Loading safetensors checkpoint shards:  14% Completed | 23/163 [00:06<00:42,  3.27it/s]
Loading safetensors checkpoint shards:  15% Completed | 24/163 [00:07<00:42,  3.28it/s]
Loading safetensors checkpoint shards:  15% Completed | 25/163 [00:07<00:44,  3.10it/s]
Loading safetensors checkpoint shards:  16% Completed | 26/163 [00:07<00:42,  3.22it/s]
Loading safetensors checkpoint shards:  17% Completed | 27/163 [00:07<00:42,  3.20it/s]
Loading safetensors checkpoint shards:  17% Completed | 28/163 [00:08<00:44,  3.03it/s]
Loading safetensors checkpoint shards:  18% Completed | 29/163 [00:08<00:42,  3.13it/s]
Loading safetensors checkpoint shards:  18% Completed | 30/163 [00:08<00:42,  3.14it/s]
Loading safetensors checkpoint shards:  19% Completed | 31/163 [00:09<00:44,  2.98it/s]
Loading safetensors checkpoint shards:  20% Completed | 32/163 [00:09<00:42,  3.10it/s]
Loading safetensors checkpoint shards:  20% Completed | 33/163 [00:09<00:43,  2.97it/s]
Loading safetensors checkpoint shards:  21% Completed | 34/163 [00:10<00:42,  3.06it/s]
Loading safetensors checkpoint shards:  21% Completed | 35/163 [00:10<00:42,  3.02it/s]
Loading safetensors checkpoint shards:  22% Completed | 36/163 [00:11<00:44,  2.88it/s]
Loading safetensors checkpoint shards:  23% Completed | 37/163 [00:11<00:41,  3.01it/s]
Loading safetensors checkpoint shards:  23% Completed | 38/163 [00:11<00:41,  3.00it/s]
Loading safetensors checkpoint shards:  24% Completed | 39/163 [00:12<00:43,  2.87it/s]
Loading safetensors checkpoint shards:  25% Completed | 40/163 [00:12<00:40,  3.03it/s]
Loading safetensors checkpoint shards:  25% Completed | 41/163 [00:12<00:41,  2.91it/s]
Loading safetensors checkpoint shards:  26% Completed | 42/163 [00:12<00:33,  3.60it/s]
Loading safetensors checkpoint shards:  26% Completed | 43/163 [00:13<00:34,  3.43it/s]
Loading safetensors checkpoint shards:  27% Completed | 44/163 [00:13<00:35,  3.31it/s]
Loading safetensors checkpoint shards:  28% Completed | 45/163 [00:13<00:39,  3.00it/s]
Loading safetensors checkpoint shards:  28% Completed | 46/163 [00:14<00:37,  3.15it/s]
Loading safetensors checkpoint shards:  29% Completed | 47/163 [00:14<00:39,  2.95it/s]
Loading safetensors checkpoint shards:  29% Completed | 48/163 [00:14<00:37,  3.03it/s]
Loading safetensors checkpoint shards:  30% Completed | 49/163 [00:15<00:37,  3.02it/s]
Loading safetensors checkpoint shards:  31% Completed | 50/163 [00:15<00:38,  2.90it/s]
Loading safetensors checkpoint shards:  31% Completed | 51/163 [00:15<00:37,  3.02it/s]
Loading safetensors checkpoint shards:  32% Completed | 52/163 [00:16<00:36,  3.02it/s]
Loading safetensors checkpoint shards:  33% Completed | 53/163 [00:16<00:37,  2.93it/s]
Loading safetensors checkpoint shards:  33% Completed | 54/163 [00:16<00:35,  3.07it/s]
Loading safetensors checkpoint shards:  34% Completed | 55/163 [00:17<00:35,  3.02it/s]
Loading safetensors checkpoint shards:  34% Completed | 56/163 [00:17<00:34,  3.11it/s]
Loading safetensors checkpoint shards:  35% Completed | 57/163 [00:17<00:34,  3.08it/s]
Loading safetensors checkpoint shards:  36% Completed | 58/163 [00:18<00:35,  2.96it/s]
Loading safetensors checkpoint shards:  36% Completed | 59/163 [00:18<00:33,  3.09it/s]
Loading safetensors checkpoint shards:  37% Completed | 60/163 [00:18<00:33,  3.09it/s]
Loading safetensors checkpoint shards:  37% Completed | 61/163 [00:19<00:34,  2.99it/s]
Loading safetensors checkpoint shards:  38% Completed | 62/163 [00:19<00:31,  3.18it/s]
Loading safetensors checkpoint shards:  39% Completed | 63/163 [00:19<00:32,  3.11it/s]
Loading safetensors checkpoint shards:  39% Completed | 64/163 [00:19<00:25,  3.82it/s]
Loading safetensors checkpoint shards:  40% Completed | 65/163 [00:20<00:27,  3.62it/s]
Loading safetensors checkpoint shards:  40% Completed | 66/163 [00:20<00:27,  3.50it/s]
Loading safetensors checkpoint shards:  41% Completed | 67/163 [00:20<00:29,  3.21it/s]
Loading safetensors checkpoint shards:  42% Completed | 68/163 [00:21<00:28,  3.36it/s]
Loading safetensors checkpoint shards:  42% Completed | 69/163 [00:21<00:29,  3.20it/s]
Loading safetensors checkpoint shards:  43% Completed | 70/163 [00:21<00:28,  3.26it/s]
Loading safetensors checkpoint shards:  44% Completed | 71/163 [00:22<00:28,  3.24it/s]
Loading safetensors checkpoint shards:  44% Completed | 72/163 [00:22<00:29,  3.10it/s]
Loading safetensors checkpoint shards:  45% Completed | 73/163 [00:22<00:28,  3.19it/s]
Loading safetensors checkpoint shards:  45% Completed | 74/163 [00:23<00:27,  3.19it/s]
Loading safetensors checkpoint shards:  46% Completed | 75/163 [00:23<00:28,  3.07it/s]
Loading safetensors checkpoint shards:  47% Completed | 76/163 [00:23<00:27,  3.17it/s]
Loading safetensors checkpoint shards:  47% Completed | 77/163 [00:24<00:28,  3.06it/s]
Loading safetensors checkpoint shards:  48% Completed | 78/163 [00:24<00:27,  3.13it/s]
Loading safetensors checkpoint shards:  48% Completed | 79/163 [00:24<00:27,  3.07it/s]
Loading safetensors checkpoint shards:  49% Completed | 80/163 [00:25<00:28,  2.95it/s]
Loading safetensors checkpoint shards:  50% Completed | 81/163 [00:25<00:26,  3.08it/s]
Loading safetensors checkpoint shards:  50% Completed | 82/163 [00:25<00:26,  3.07it/s]
Loading safetensors checkpoint shards:  51% Completed | 83/163 [00:26<00:27,  2.96it/s]
Loading safetensors checkpoint shards:  52% Completed | 84/163 [00:26<00:25,  3.12it/s]
Loading safetensors checkpoint shards:  52% Completed | 85/163 [00:26<00:25,  3.02it/s]
Loading safetensors checkpoint shards:  53% Completed | 86/163 [00:26<00:20,  3.72it/s]
Loading safetensors checkpoint shards:  53% Completed | 87/163 [00:27<00:21,  3.50it/s]
Loading safetensors checkpoint shards:  54% Completed | 88/163 [00:27<00:22,  3.36it/s]
Loading safetensors checkpoint shards:  55% Completed | 89/163 [00:27<00:23,  3.09it/s]
Loading safetensors checkpoint shards:  55% Completed | 90/163 [00:28<00:22,  3.25it/s]
Loading safetensors checkpoint shards:  56% Completed | 91/163 [00:28<00:23,  3.02it/s]
Loading safetensors checkpoint shards:  56% Completed | 92/163 [00:28<00:22,  3.09it/s]
Loading safetensors checkpoint shards:  57% Completed | 93/163 [00:29<00:22,  3.07it/s]
Loading safetensors checkpoint shards:  58% Completed | 94/163 [00:29<00:23,  2.93it/s]
Loading safetensors checkpoint shards:  58% Completed | 95/163 [00:29<00:22,  3.03it/s]
Loading safetensors checkpoint shards:  59% Completed | 96/163 [00:30<00:22,  3.03it/s]
Loading safetensors checkpoint shards:  60% Completed | 97/163 [00:30<00:22,  2.89it/s]
Loading safetensors checkpoint shards:  60% Completed | 98/163 [00:30<00:21,  2.98it/s]
Loading safetensors checkpoint shards:  61% Completed | 99/163 [00:31<00:22,  2.87it/s]
Loading safetensors checkpoint shards:  61% Completed | 100/163 [00:31<00:20,  3.03it/s]
Loading safetensors checkpoint shards:  62% Completed | 101/163 [00:31<00:20,  3.08it/s]
Loading safetensors checkpoint shards:  63% Completed | 102/163 [00:32<00:20,  3.00it/s]
Loading safetensors checkpoint shards:  63% Completed | 103/163 [00:32<00:19,  3.15it/s]
Loading safetensors checkpoint shards:  64% Completed | 104/163 [00:32<00:18,  3.18it/s]
Loading safetensors checkpoint shards:  64% Completed | 105/163 [00:33<00:18,  3.13it/s]
Loading safetensors checkpoint shards:  65% Completed | 106/163 [00:33<00:17,  3.32it/s]
Loading safetensors checkpoint shards:  66% Completed | 107/163 [00:33<00:17,  3.26it/s]
Loading safetensors checkpoint shards:  66% Completed | 108/163 [00:33<00:13,  3.94it/s]
Loading safetensors checkpoint shards:  67% Completed | 109/163 [00:34<00:14,  3.74it/s]
Loading safetensors checkpoint shards:  67% Completed | 110/163 [00:34<00:14,  3.65it/s]
Loading safetensors checkpoint shards:  68% Completed | 111/163 [00:34<00:15,  3.38it/s]
Loading safetensors checkpoint shards:  69% Completed | 112/163 [00:35<00:14,  3.54it/s]
Loading safetensors checkpoint shards:  69% Completed | 113/163 [00:35<00:14,  3.40it/s]
Loading safetensors checkpoint shards:  70% Completed | 114/163 [00:35<00:14,  3.44it/s]
Loading safetensors checkpoint shards:  71% Completed | 115/163 [00:35<00:13,  3.43it/s]
Loading safetensors checkpoint shards:  71% Completed | 116/163 [00:36<00:14,  3.34it/s]
Loading safetensors checkpoint shards:  72% Completed | 117/163 [00:36<00:13,  3.40it/s]
Loading safetensors checkpoint shards:  72% Completed | 118/163 [00:36<00:13,  3.42it/s]
Loading safetensors checkpoint shards:  73% Completed | 119/163 [00:37<00:13,  3.31it/s]
Loading safetensors checkpoint shards:  74% Completed | 120/163 [00:37<00:12,  3.40it/s]
Loading safetensors checkpoint shards:  74% Completed | 121/163 [00:37<00:12,  3.30it/s]
Loading safetensors checkpoint shards:  75% Completed | 122/163 [00:37<00:12,  3.37it/s]
Loading safetensors checkpoint shards:  75% Completed | 123/163 [00:38<00:12,  3.32it/s]
Loading safetensors checkpoint shards:  76% Completed | 124/163 [00:38<00:12,  3.19it/s]
Loading safetensors checkpoint shards:  77% Completed | 125/163 [00:38<00:11,  3.29it/s]
Loading safetensors checkpoint shards:  77% Completed | 126/163 [00:39<00:11,  3.28it/s]
Loading safetensors checkpoint shards:  78% Completed | 127/163 [00:39<00:11,  3.18it/s]
Loading safetensors checkpoint shards:  79% Completed | 128/163 [00:39<00:10,  3.37it/s]
Loading safetensors checkpoint shards:  79% Completed | 129/163 [00:40<00:10,  3.32it/s]
Loading safetensors checkpoint shards:  80% Completed | 130/163 [00:40<00:08,  4.07it/s]
Loading safetensors checkpoint shards:  80% Completed | 131/163 [00:40<00:08,  3.82it/s]
Loading safetensors checkpoint shards:  81% Completed | 132/163 [00:40<00:08,  3.71it/s]
Loading safetensors checkpoint shards:  82% Completed | 133/163 [00:41<00:08,  3.41it/s]
Loading safetensors checkpoint shards:  82% Completed | 134/163 [00:41<00:08,  3.57it/s]
Loading safetensors checkpoint shards:  83% Completed | 135/163 [00:41<00:08,  3.42it/s]
Loading safetensors checkpoint shards:  83% Completed | 136/163 [00:42<00:07,  3.46it/s]
Loading safetensors checkpoint shards:  84% Completed | 137/163 [00:42<00:07,  3.45it/s]
Loading safetensors checkpoint shards:  85% Completed | 138/163 [00:42<00:07,  3.29it/s]
Loading safetensors checkpoint shards:  85% Completed | 139/163 [00:42<00:07,  3.37it/s]
Loading safetensors checkpoint shards:  86% Completed | 140/163 [00:43<00:06,  3.38it/s]
Loading safetensors checkpoint shards:  87% Completed | 141/163 [00:43<00:06,  3.24it/s]
Loading safetensors checkpoint shards:  87% Completed | 142/163 [00:43<00:06,  3.35it/s]
Loading safetensors checkpoint shards:  88% Completed | 143/163 [00:44<00:06,  3.23it/s]
Loading safetensors checkpoint shards:  88% Completed | 144/163 [00:44<00:05,  3.40it/s]
Loading safetensors checkpoint shards:  89% Completed | 145/163 [00:44<00:05,  3.40it/s]
Loading safetensors checkpoint shards:  90% Completed | 146/163 [00:45<00:05,  3.21it/s]
Loading safetensors checkpoint shards:  90% Completed | 147/163 [00:45<00:04,  3.39it/s]
Loading safetensors checkpoint shards:  91% Completed | 148/163 [00:45<00:04,  3.40it/s]
Loading safetensors checkpoint shards:  91% Completed | 149/163 [00:46<00:04,  3.18it/s]
Loading safetensors checkpoint shards:  92% Completed | 150/163 [00:46<00:03,  3.29it/s]
Loading safetensors checkpoint shards:  93% Completed | 151/163 [00:46<00:03,  3.14it/s]
Loading safetensors checkpoint shards:  94% Completed | 153/163 [00:47<00:02,  3.85it/s]
Loading safetensors checkpoint shards:  94% Completed | 154/163 [00:47<00:02,  3.66it/s]
Loading safetensors checkpoint shards:  95% Completed | 155/163 [00:47<00:02,  3.34it/s]
Loading safetensors checkpoint shards:  96% Completed | 156/163 [00:47<00:02,  3.49it/s]
Loading safetensors checkpoint shards:  96% Completed | 157/163 [00:48<00:01,  3.43it/s]
Loading safetensors checkpoint shards:  97% Completed | 158/163 [00:48<00:01,  3.24it/s]
Loading safetensors checkpoint shards:  98% Completed | 159/163 [00:48<00:01,  3.41it/s]
Loading safetensors checkpoint shards:  98% Completed | 160/163 [00:49<00:00,  3.27it/s]
Loading safetensors checkpoint shards:  99% Completed | 161/163 [00:49<00:00,  3.42it/s]
Loading safetensors checkpoint shards:  99% Completed | 162/163 [00:49<00:00,  3.41it/s]
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:50<00:00,  3.36it/s]
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:50<00:00,  3.25it/s]

[2025-06-14 11:54:59 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=58.29 GB, mem usage=79.73 GB.

before: from 12:00:27 to 12:01:19

[2025-06-14 12:00:27 TP0] Detected fp8 checkpoint.
Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   7% Completed | 11/163 [00:00<00:01, 105.15it/s]
Loading safetensors checkpoint shards:  13% Completed | 22/163 [00:00<00:01, 101.08it/s]
Loading safetensors checkpoint shards:  20% Completed | 33/163 [00:00<00:01, 100.91it/s]
Loading safetensors checkpoint shards:  27% Completed | 44/163 [00:00<00:01, 101.92it/s]
Loading safetensors checkpoint shards:  34% Completed | 55/163 [00:00<00:01, 100.75it/s]
Loading safetensors checkpoint shards:  40% Completed | 66/163 [00:00<00:00, 101.91it/s]
Loading safetensors checkpoint shards:  47% Completed | 77/163 [00:01<00:01, 57.36it/s]
Loading safetensors checkpoint shards:  54% Completed | 88/163 [00:01<00:01, 67.12it/s]
Loading safetensors checkpoint shards:  60% Completed | 98/163 [00:01<00:00, 73.79it/s]
Loading safetensors checkpoint shards:  67% Completed | 109/163 [00:01<00:00, 81.88it/s]
Loading safetensors checkpoint shards:  74% Completed | 120/163 [00:01<00:00, 87.03it/s]
Loading safetensors checkpoint shards:  80% Completed | 131/163 [00:01<00:00, 92.18it/s]
Loading safetensors checkpoint shards:  87% Completed | 142/163 [00:01<00:00, 94.62it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 156644.97it/s]
Loading safetensors checkpoint shards:  94% Completed | 153/163 [00:01<00:00, 98.02it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 163707.69it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 157354.23it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 159835.50it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 162831.08it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 162940.14it/s]
Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 163049.35it/s]
Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:01<00:00, 88.56it/s]

Cloning 1 shared expert into MoE: 100%|███████████████████████████████████████████████████████████████████████████████| 58/58 [00:00<00:00, 158585.16it/s]
[2025-06-14 12:01:19 TP0] Load weight end. type=DeepseekV3ForCausalLM, dtype=torch.bfloat16, avail mem=58.29 GB, mem usage=79.73 GB.

Modifications

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

gemini-code-assist

Summary of Changes

Hello @Alcanderian, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue with the DeepseekV3 weight loading process where the progress bar was inaccurate due to the weight iterator being prematurely consumed. It also simplifies the internal logic for handling and fusing shared experts by processing them directly within the loading loop.

Highlights

Fix tqdm for weight loading: The progress bar for loading DeepseekV3 weights was misleading because the async iterator was converted to a list, causing the bar to complete before loading finished. This PR fixes that by processing weights while iterating.
Simplify shared experts fusion: The logic for fusing shared experts has been simplified. Instead of collecting all weights into a list and then manipulating it, shared expert weights are now renamed directly during the iteration over the weight stream.
Code Refactor: Removed a large block of code that handled the old list-based weight manipulation for shared experts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configureGemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively addresses the misleading tqdm progress bar issue during dsv3 weight loading by simplifying the shared expert fusion logic. The previous approach of converting the weight iterator to a list caused the progress bar to complete prematurely. The new method of renaming shared expert weights on-the-fly during iteration is much cleaner, more efficient, and directly resolves the tqdm problem.

The refactoring of deep_gemm_grouped_gemm_nt_f8f8bf16_masked to use deep_gemm_wrapper is a minor change and seems appropriate for centralizing JIT logic.

Overall, the changes improve code maintainability and correctness of the loading progress indication. I've made one minor suggestion regarding logging consistency.

BBuf · 2025-06-14T12:19:44Z

python/sglang/srt/models/deepseek_v2.py

@@ -2128,9 +2033,19 @@ def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]], is_nextn=Fal
                "hnorm",
            ]

+        if self.num_fused_shared_experts > 0:
+            assert self.num_fused_shared_experts == 1


DeepSeek-Coder-V2-Lite-Instruct has 2 shared experts, which we also support—this assertion is unnecessary.

DeepSeek-Coder-V2-Lite-Instruct has 2 shared experts, which we also support—this assertion is unnecessary.

DeepseekV2ForCausalLM will be disabled to use shared expert fusion. I will add or self.config.n_shared_expert != 1 at https://github.com/sgl-project/sglang/pull/7180/files#diff-5b9e34dd492bd8a14702a18b594721091092276fad1cf8736fba6ef1f33c1b04R1722

BBuf

Good job! Creating a new list is indeed unnecessary, and I believe this is partly why some users encounter hangs when cloning shared experts.

…gl-project#7181)

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion

3ee675f

Alcanderian requested review from merrymercy, Ying1123, hnyls2002, zhyncs, ispobock, ByronHsu and zhaochenyang20 as code owners June 14, 2025 12:02

Alcanderian assigned BBuf and ch-wan Jun 14, 2025

gemini-code-assist bot reviewed Jun 14, 2025

View reviewed changes

BBuf reviewed Jun 14, 2025

View reviewed changes

BBuf approved these changes Jun 14, 2025

View reviewed changes

zhyncs merged commit ed54bf9 into sgl-project:main Jun 14, 2025
91 of 102 checks passed

coco-alen pushed a commit to jinleic/sglang that referenced this pull request Jun 20, 2025

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion (s…

a12c1c1

…gl-project#7181)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion #7181

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion #7181

Uh oh!

Alcanderian commented Jun 14, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

BBuf Jun 14, 2025 •

edited

Loading

Uh oh!

Alcanderian Jun 14, 2025 •

edited

Loading

Uh oh!

BBuf left a comment

Uh oh!

Uh oh!

Uh oh!

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion #7181

[fix] fix dsv3 weight loader tqdm and simplify shared experts fusion #7181

Uh oh!

Conversation

Alcanderian commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Checklist

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

BBuf Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alcanderian Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BBuf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Alcanderian commented Jun 14, 2025 •

edited

Loading

BBuf Jun 14, 2025 •

edited

Loading

Alcanderian Jun 14, 2025 •

edited

Loading