Co-authored with @qingquansong

Roadmap Issue is here

Motivation

Support FA3 as attention backend by using fa3's flash_attn_with_kvcache

Conclusion:

• Prefill Throughput are on par than current baseline
• Decode Throughput are slightly higher than current baseline
• Accuracy is slightly better than current baseline
• Flashinfer will OOM when batch size, input size is large while FA3 won't

What has been supported:

• MHA Models like Llama/QWen/Gemma
• Cuda Graph
• Sliding Window (tested with Gemma 2)

TODO in this PR:

• Remove the clang format change, now it's blocking our commit for some reason
• Add check to fail the launch server command if gpu is below Hopper.

Next Steps after this PR:

• Figure out how to build FA3 into SGLang: @hebiao064 @zhyncs
• Page Size > 1
• Support Multimodal
• Support Speculative Decoding
• Support MLA for Deepseek-like models
• Support FP8

Benchmark on Latency

Note: I benchmarked on various of input/output length, the graph is an aggregated view based on batch size, please checkout the sheet for details

Benchmark on Accuracy

GSM 8K: python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-q uestions 1319 --parallel 1319

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.