python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4 --trust-remote-code --quantization modelopt_fp4 --tp 8 --enable-flashinfer-cutlass-moe --enable-ep-moe --ep-size 8 --dp 8 --enable-dp-attention
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319 --port=30000
Accuracy: 0.959
Invalid: 0.000
Latency: 22.890 s
Output throughput: 6335.267 token/s

python3 -m sglang.launch_server --model-path nvidia/DeepSeek-R1-0528-FP4 --trust-remote-code --quantization modelopt_fp4 --tp 8 --enable-flashinfer-cutlass-moe --enable-ep-moe --ep-size 8 --dp 8 --enable-dp-attention
python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompt 1024 --random-input 1024 --random-output 1024 --random-range-ratio 1 --max-concurrency 1024

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1024
Successful requests:                     1024
Benchmark duration (s):                  76.79
Total input tokens:                      1048576
Total generated tokens:                  1048576
Total generated tokens (retokenized):    1046065
Request throughput (req/s):              13.33
Input token throughput (tok/s):          13655.01
Output token throughput (tok/s):         13655.01
Total token throughput (tok/s):          27310.02
Concurrency:                             1021.18
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   76579.19
Median E2E Latency (ms):                 76556.41
---------------Time to First Token----------------
Mean TTFT (ms):                          11978.29
Median TTFT (ms):                        11876.27
P99 TTFT (ms):                           21513.72
---------------Inter-Token Latency----------------
Mean ITL (ms):                           63.15
Median ITL (ms):                         53.14
P95 ITL (ms):                            69.91
P99 ITL (ms):                            77.50
Max ITL (ms):                            18880.99
==================================================

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1024
Successful requests:                     1024
Benchmark duration (s):                  73.99
Total input tokens:                      1048576
Total generated tokens:                  1048576
Total generated tokens (retokenized):    1044921
Request throughput (req/s):              13.84
Input token throughput (tok/s):          14172.69
Output token throughput (tok/s):         14172.69
Total token throughput (tok/s):          28345.37
Concurrency:                             1021.65
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   73816.06
Median E2E Latency (ms):                 73829.85
---------------Time to First Token----------------
Mean TTFT (ms):                          10411.04
Median TTFT (ms):                        10263.18
P99 TTFT (ms):                           19516.86
---------------Inter-Token Latency----------------
Mean ITL (ms):                           61.98
Median ITL (ms):                         52.59
P95 ITL (ms):                            67.60
P99 ITL (ms):                            74.52
Max ITL (ms):                            17244.45
==================================================