Fix num_token_padding support for static per-tensor scaled_fp8_quant #20188

mgoin · 2025-06-27T15:32:54Z

Purpose

FIX #20192

After #20076 fixed the assert to always check for per-tensor scales in the static scaled_fp8_quant case, this test tests/quantization/test_fp8.py::test_scaled_fp8_quant started failing
https://buildkite.com/vllm/ci/builds/22749#0197ad88-59ca-41d0-9692-2bb3f5c6dca3

It seems to me that there is no reason why padding wouldn't be supported in this static per-tensor quant case

Test Plan

Use the existing failing test at

vllm/tests/quantization/test_fp8.py

Lines 190 to 196 in aafabaa

    
           # Padding 
        
           y, _ = ops.scaled_fp8_quant(x, inv_scale, num_token_padding=17) 
        
           assert y.shape[0] == 17 
        
           torch.testing.assert_close( 
        
               ref_y, 
        
               per_tensor_dequantize(torch.narrow(y, 0, 0, x.shape[0]), inv_scale, 
        
                                     dtype))

Test Result

tests/quantization/test_fp8.py::test_scaled_fp8_quant[dtype0] PASSED
tests/quantization/test_fp8.py::test_scaled_fp8_quant[dtype1] PASSED

Signed-off-by: mgoin <mgoin64@gmail.com>

gemini-code-assist

Summary of Changes

Hello @mgoin, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a specific issue where num_token_padding was inadvertently restricted for static per-tensor scaled FP8 quantization, leading to a test failure. My change re-enables the intended support for num_token_padding in this context, ensuring the correct behavior of the quantization function.

Highlights

Bug Fix: I've addressed a regression introduced by a previous PR (/pull/20076) that caused test_scaled_fp8_quant to fail when num_token_padding was used with static_scaled_fp8_quant.
Quantization Logic Adjustment: The assertion in vllm/_custom_ops.py for the static_scaled_fp8_quant path was relaxed. Previously, it incorrectly enforced num_token_padding is None in addition to scale.numel() == 1. I've removed the num_token_padding is None check, allowing num_token_padding to be supported in this specific static per-tensor scaled FP8 quantization scenario.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This PR resolves a regression where num_token_padding was incorrectly disallowed for static per-tensor FP8 quantization. The change removes the unnecessary assertion, and I've suggested ensuring input contiguity for CUDA kernel compatibility.

vllm/_custom_ops.py

github-actions · 2025-06-27T15:34:19Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

minosfuture · 2025-06-28T03:54:58Z

vllm/_custom_ops.py

@@ -1274,8 +1274,7 @@ def scaled_fp8_quant(
            scale = torch.zeros(1, device=input.device, dtype=torch.float32)
            torch.ops._C.dynamic_scaled_fp8_quant(output, input, scale)
    else:
-        # num_token_padding not implemented for this case
-        assert (scale.numel() == 1 and num_token_padding is None)
+        assert scale.numel() == 1


this assertion depends on use_per_token_if_dynamic. if use_per_token_if_dynamic, scale has a shape (shape[0], 1).

tests/kernels/moe/test_cutlass_moe.py on main is currently failing due to #20076 as well.

…llm-project#20188) Signed-off-by: mgoin <mgoin64@gmail.com>

…llm-project#20188) Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

…llm-project#20188) Signed-off-by: mgoin <mgoin64@gmail.com>

Fix num_token_padding support for static per-tensor scaled_fp8_quant

e874333

Signed-off-by: mgoin <mgoin64@gmail.com>

gemini-code-assist bot reviewed Jun 27, 2025

View reviewed changes

vllm/_custom_ops.py Show resolved Hide resolved

tlrmchlsmth approved these changes Jun 27, 2025

View reviewed changes

mgoin added bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed labels Jun 27, 2025

aarnphm enabled auto-merge (squash) June 27, 2025 20:15

minosfuture reviewed Jun 28, 2025

View reviewed changes

vllm-bot merged commit a29e62e into vllm-project:main Jun 28, 2025
80 of 82 checks passed

CSWYF3634076 pushed a commit to CSWYF3634076/vllm that referenced this pull request Jul 2, 2025

Fix num_token_padding support for static per-tensor scaled_fp8_quant (v…

5b9762c

…llm-project#20188) Signed-off-by: mgoin <mgoin64@gmail.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

Fix num_token_padding support for static per-tensor scaled_fp8_quant (v…

714932c

…llm-project#20188) Signed-off-by: mgoin <mgoin64@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix num_token_padding support for static per-tensor scaled_fp8_quant #20188

Fix num_token_padding support for static per-tensor scaled_fp8_quant #20188

Uh oh!

mgoin commented Jun 27, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

minosfuture Jun 28, 2025

Uh oh!

Uh oh!

Uh oh!

	# Padding
	y, _ = ops.scaled_fp8_quant(x, inv_scale, num_token_padding=17)
	assert y.shape[0] == 17
	torch.testing.assert_close(
	ref_y,
	per_tensor_dequantize(torch.narrow(y, 0, 0, x.shape[0]), inv_scale,
	dtype))

Uh oh!

Fix num_token_padding support for static per-tensor scaled_fp8_quant #20188

Fix num_token_padding support for static per-tensor scaled_fp8_quant #20188

Uh oh!

Conversation

mgoin commented Jun 27, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

minosfuture Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mgoin commented Jun 27, 2025 •

edited by github-actions bot

Loading