Fix fused MoE when `routed_scaling_factor is None` #8709

hnyls2002 · 2025-08-02T17:36:51Z

Reproduction:

python3 -m sglang.compile_deep_gemm --host=0.0.0.0 --port=8000 --model=Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --tp=8 --trust-remote-code --enable-ep-moe --tool-call-parser=qwen25 --reasoning-parser=qwen3 --context-length=131072

Issue:

  File "/root/sglang/python/sglang/srt/layers/moe/ep_moe/layer.py", line 283, in forward_deepgemm
    return output * self.routed_scaling_factor
TypeError: unsupported operand type(s) for *: 'Tensor' and 'NoneType'

gemini-code-assist

Summary of Changes

Hello @hnyls2002, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a minor but important fix within the fused MoE layer's initialization. It enhances the robustness of the MoE implementation by preventing potential runtime issues when a specific configuration parameter, routed_scaling_factor, is not explicitly provided, ensuring it always has a valid numerical value.

Highlights

Bug Fix: Addressed an issue in the fused Mixture-of-Experts (MoE) layer where the routed_scaling_factor could be None, potentially leading to errors. The change ensures that if routed_scaling_factor is passed as None, it defaults to 1.0.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Summary of Changes

Hello @hnyls2002, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a potential issue in the fused Mixture of Experts (MoE) layer where the routed_scaling_factor could be initialized as None, leading to unexpected behavior or errors. The change ensures that a sensible default value of 1.0 is used in such cases, improving the robustness of the MoE implementation.

Highlights

Bug Fix: Corrected the initialization of self.routed_scaling_factor in the FusedMoELayer to default to 1.0 when the routed_scaling_factor parameter is None.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request correctly handles the case where routed_scaling_factor is None by providing a default value of 1.0. This prevents a potential TypeError in EPMoE.forward_deepgemm. The change is small, correct, and improves the robustness of the code. I've added one minor suggestion for conciseness.

python/sglang/srt/layers/moe/fused_moe_triton/layer.py

gemini-code-assist

Code Review

This pull request correctly fixes a potential TypeError that could occur in MoE layers if routed_scaling_factor is None. The change ensures a default value is used, preventing a runtime error. I've added one suggestion to improve the code's conciseness.

python/sglang/srt/layers/moe/fused_moe_triton/layer.py

fix

a8f45f2

hnyls2002 requested review from merrymercy, Ying1123, zhyncs, ispobock, HaiShaw, ch-wan, BBuf and kushanam as code owners August 2, 2025 17:36

gemini-code-assist bot reviewed Aug 2, 2025

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/layer.py Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Aug 2, 2025

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/layer.py Outdated Show resolved Hide resolved

ch-wan self-assigned this Aug 2, 2025

ch-wan reviewed Aug 2, 2025

View reviewed changes

python/sglang/srt/layers/moe/fused_moe_triton/layer.py Outdated Show resolved Hide resolved

hebiao064 mentioned this pull request Aug 3, 2025

[perf] Weight Sync Optimization in Colocate Mode THUDM/slime#132

Open

hebiao064 self-assigned this Aug 3, 2025

fix

720d996

hnyls2002 merged commit 9f47d68 into main Aug 3, 2025
3 of 54 checks passed

hnyls2002 deleted the lsyin/fix-moe-routed-scaling-factor branch August 3, 2025 04:42

lifuhuang pushed a commit that referenced this pull request Aug 3, 2025

Fix fused MoE when routed_scaling_factor is None (#8709)

ce3fe09

htiennv pushed a commit to htiennv/sglang that referenced this pull request Aug 5, 2025

Fix fused MoE when routed_scaling_factor is None (sgl-project#8709)

6ff51fb

ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025

Fix fused MoE when routed_scaling_factor is None (#8709)

6240157

ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025

Fix fused MoE when routed_scaling_factor is None (#8709)

91895c4

narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025

Fix fused MoE when routed_scaling_factor is None (sgl-project#8709)

50704be

narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 18, 2025

Fix fused MoE when routed_scaling_factor is None (sgl-project#8709)

1e14693

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix fused MoE when `routed_scaling_factor is None` #8709

Fix fused MoE when `routed_scaling_factor is None` #8709

Uh oh!

hnyls2002 commented Aug 2, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fix fused MoE when routed_scaling_factor is None #8709

Fix fused MoE when routed_scaling_factor is None #8709

Uh oh!

Conversation

hnyls2002 commented Aug 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fix fused MoE when `routed_scaling_factor is None` #8709

Fix fused MoE when `routed_scaling_factor is None` #8709

hnyls2002 commented Aug 2, 2025 •

edited

Loading