Skip to content

[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 #8688

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 2, 2025
Merged

[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 #8688

merged 1 commit into from
Aug 2, 2025

Conversation

trevor-m
Copy link
Collaborator

@trevor-m trevor-m commented Aug 2, 2025

Motivation

Similar to #8333 we weren't applyign routed scaling factor for cutlass_fused_experts_fp8 (SGLANG_CUTLASS_MOE=1) path.

#8364 will fuse this into select_experts, but let's get the accuracy fixed first since that requires an sgl-kernel change and won't be merged for a while.

Modifications

Multiply MOE output by factor.

Accuracy Test

``
SGL_ENABLE_JIT_DEEPGEMM=0 SGLANG_CUTLASS_MOE=1 SGLANG_ENABLE_FLASHINFER_GEMM=1 python3 -m sglang.launch_server --tp=8 --trust-remote-code --disable-radix-cache --model-path=deepseek-ai/DeepSeek-R1-0528 --attention-backend=cutlass_mla

SGL_CUTLASS_MOE=1 FP8 path was giving 0.832 accuracy, now it gives 0.965.

## Benchmark & Profiling

<!-- If this PR is expected to impact performance, please provide benchmark and profiling results. Ref: https://docs.sglang.ai/references/benchmark_and_profiling.html -->

## Checklist

- [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit).
- [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci).
- [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci).
- [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html).
- [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
- [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 2, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @trevor-m, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a critical bug fix to improve the accuracy of Mixture-of-Experts (MoE) models utilizing FP8 quantization with the cutlass_fused_experts_fp8 kernel. The core change involves correctly applying a previously missing scaling factor to the expert output, which was identified as the cause of accuracy degradation.

Highlights

  • Bug Fix for MoE Accuracy: I've addressed a bug where the cutlass_fused_experts_fp8 path, used when SGLANG_CUTLASS_MOE=1 and FP8 quantization is enabled, was not applying the necessary routed scaling factor to its output. This omission led to reduced accuracy.
  • Accuracy Improvement: By applying the routed_scaling_factor to the output of the cutlass_fused_experts_fp8 operation, I've observed a significant improvement in accuracy for the affected FP8 path, increasing from 0.832 to 0.965 in the provided test case.
  • Interim Solution: This change provides an immediate accuracy fix. A future PR (#8364) is planned to fuse this scaling into the select_experts operation, but this current PR ensures accuracy is restored sooner.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Aug 2, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request correctly applies the routed_scaling_factor to the cutlass_fused_experts_fp8 path, which resolves an accuracy issue. The change is minimal and targeted.

I've pointed out a high severity issue where a TypeError could occur if routed_scaling_factor is None. I've included a code suggestion to fix this. After addressing this, the PR should be in great shape.

@trevor-m trevor-m force-pushed the fp8-routed-factor branch from a6c2702 to 2afb82b Compare August 2, 2025 00:04
@trevor-m trevor-m mentioned this pull request Aug 2, 2025
Merged
6 tasks
@kushanam kushanam merged commit 89caf7a into sgl-project:main Aug 2, 2025
55 of 62 checks passed
@gemini-code-assist Gemini Code Assist
Copy link
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

lifuhuang pushed a commit that referenced this pull request Aug 3, 2025
@trevor-m @lifuhuang
[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (#8688
4fda637 
)
ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025
@trevor-m @ssssnow
[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (#8688
f9e6b8d 
)
ShangmingCai pushed a commit that referenced this pull request Aug 5, 2025
@trevor-m @ssssnow
[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (#8688
869ad62 
)
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 17, 2025
@trevor-m @narutolhy
[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (sg…
75bd777 
…l-project#8688)
narutolhy pushed a commit to narutolhy/sglang that referenced this pull request Aug 18, 2025
@trevor-m @narutolhy
[bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (sg…
96a0ab0 
…l-project#8688)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kushanam kushanam kushanam approved these changes

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@ch-wan ch-wan Awaiting requested review from ch-wan ch-wan is a code owner

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@trevor-m @kushanam