[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels #20331

yewentao256 · 2025-07-01T19:27:19Z

Purpose

Fixes #20327

Test

Tested on H100:

Performance (3% E2E throughput gain):

vllm bench throughput --model RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 --load-format dummy --input-len 1000 --output-len 100 --max-model-len 32768
# Main
Throughput: 37.24 requests/s, 40917.63 total tokens/s, 3723.77 output tokens/s
Throughput: 37.42 requests/s, 41122.36 total tokens/s, 3742.40 output tokens/s
# Now
Throughput: 38.38 requests/s, 42177.65 total tokens/s, 3838.44 output tokens/s
Throughput: 38.55 requests/s, 42355.99 total tokens/s, 3854.67 output tokens/s

Accuracy:

lm_eval --model vllm --model_args pretrained=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8,max_model_len=32768 --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto

# Main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7733|±  |0.0115|
|     |       |strict-match    |     5|exact_match|↑  |0.7604|±  |0.0118|
# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.7733|±  |0.0115|
|     |       |strict-match    |     5|exact_match|↑  |0.7604|±  |0.0118|

Signed-off-by: yewentao256 <zhyanwentao@126.com>

github-actions · 2025-07-01T19:27:28Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Summary of Changes

Hello @yewentao256, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on improving the performance of int8 quantization kernels by enhancing and introducing new vectorization utilities. The core change involves refactoring the data access patterns within the quantization kernels to utilize more efficient, aligned memory reads, which has been validated to provide a throughput increase without impacting model accuracy.

Highlights

Performance Optimization for Int8 Quantization: I've optimized the dynamic_scaled_int8_quant_kernel and dynamic_scaled_int8_azp_quant_kernel in int8_quant_kernels.cu by replacing traditional scalar for loops with the newly introduced vectorize_read_with_alignment utility. This change aims to leverage vectorized memory access for improved performance during absmax and min/max calculations.
New Vectorization Utility for Read Operations: A new template function, vectorize_read_with_alignment, has been added to vectorization_utils.cuh. This utility provides an efficient, read-only mechanism for iterating over input data with alignment guarantees, handling both aligned and potentially unaligned data segments by processing prefixes and tails separately.
Optimization of Existing Vectorization Utility: The existing vectorize_with_alignment utility in vectorization_utils.cuh has been enhanced with a 'fast path'. This optimization allows for direct vectorized processing when the entire data region is already aligned, bypassing the more complex logic for handling misaligned prefixes and tails, leading to more efficient operations.
Validated Performance Gains: Performance benchmarks using vllm bench throughput on an H100 GPU show a noticeable improvement in throughput for int8 quantized models (from ~37 req/s to ~38.5 req/s). Crucially, accuracy tests using lm_eval confirm that these optimizations do not introduce any regression in model accuracy.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces performance optimizations for INT8 quantization kernels by leveraging vectorized memory reads. The changes include a new vectorize_read_with_alignment utility and a fast path for aligned data in vectorize_with_alignment. I've identified one potential correctness issue in the new fast-path logic that should be addressed for robustness.

csrc/quantization/vectorization_utils.cuh

Signed-off-by: yewentao256 <zhyanwentao@126.com>

mgoin

Clean work, thanks Wentao!

DarkLight1337 · 2025-07-02T12:53:18Z

Can you merge from main to fix the CI failures?

…s-int8

yewentao256 · 2025-07-02T13:28:33Z

Can you merge from main to fix the CI failures?

Done, thanks!

…s-int8

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

optimize vectorization utils for int 8

28caa54

Signed-off-by: yewentao256 <zhyanwentao@126.com>

yewentao256 changed the title ~~[Perf] Optimize vectorization utils for int 8~~ [Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels Jul 1, 2025

gemini-code-assist bot reviewed Jul 1, 2025

View reviewed changes

csrc/quantization/vectorization_utils.cuh Show resolved Hide resolved

add comments

16bcd2d

Signed-off-by: yewentao256 <zhyanwentao@126.com>

mgoin approved these changes Jul 1, 2025

View reviewed changes

mgoin added performance Performance-related issues ready ONLY add when PR is ready to merge/full CI is needed labels Jul 1, 2025

Merge branch 'vllm-project:main' into wye-optimize-vectorization-util…

b352724

…s-int8

Merge branch 'vllm-project:main' into wye-optimize-vectorization-util…

e2a4364

…s-int8

DarkLight1337 merged commit 783921d into vllm-project:main Jul 4, 2025
105 checks passed

yewentao256 deleted the wye-optimize-vectorization-utils-int8 branch July 4, 2025 13:13

sfeng33 pushed a commit to sfeng33/vllm that referenced this pull request Jul 6, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

5a80815

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

huydhn pushed a commit to huydhn/vllm that referenced this pull request Jul 8, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

64a3d23

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

Chen-zexi pushed a commit to Chen-zexi/vllm that referenced this pull request Jul 13, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

b99057e

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

1d44155

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

17ab9f3

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

03acdab

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels (v…

d3606a2

…llm-project#20331) Signed-off-by: yewentao256 <zhyanwentao@126.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels #20331

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels #20331

Uh oh!

yewentao256 commented Jul 1, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

mgoin left a comment

Uh oh!

DarkLight1337 commented Jul 2, 2025

Uh oh!

yewentao256 commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels #20331

[Perf] Optimize Vectorization Utils for Int 8 Quantization Kernels #20331

Uh oh!

Conversation

yewentao256 commented Jul 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test

Uh oh!

github-actions bot commented Jul 1, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

DarkLight1337 commented Jul 2, 2025

Uh oh!

yewentao256 commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

yewentao256 commented Jul 1, 2025 •

edited by github-actions bot

Loading