/sgl-workspace/sglang# RCCL_MSCCL_ENABLE=0 CK_MOE=1 python -m sglang.bench_one_batch --batch-size 64 --input 512 --output 32 --model deepseek-ai/DeepSeek-R1/ --tp 8 --trust-remote-code --quantization fp8
Benchmark ...
Prefill. latency: 2.02619 s, throughput:  16172.25 token/s
Decode.  latency: 0.03899 s, throughput:   1641.31 token/s
Decode.  latency: 0.03917 s, throughput:   1633.71 token/s
Decode.  latency: 0.03974 s, throughput:   1610.47 token/s
Decode.  latency: 0.04054 s, throughput:   1578.81 token/s
Decode.  latency: 0.04060 s, throughput:   1576.45 token/s
Decode.  median latency: 0.04177 s, median throughput:   1532.14 token/s
Total. latency:  3.311 s, throughput:  10514.95 token/s

/sgl-workspace/sglang# RCCL_MSCCL_ENABLE=0 CK_MOE=1 AITER_BLOCK_GEMM=1 python -m sglang.bench_one_batch --batch-size 64 --input 512 --output 32 --model deepseek-ai/DeepSeek-R1/ --tp 8 --trust-remote-code --quantization fp8
Benchmark ...
Prefill. latency: 2.35484 s, throughput:  13915.15 token/s
Decode.  latency: 0.03659 s, throughput:   1749.00 token/s
Decode.  latency: 0.03680 s, throughput:   1739.00 token/s
Decode.  latency: 0.03729 s, throughput:   1716.12 token/s
Decode.  latency: 0.03764 s, throughput:   1700.34 token/s
Decode.  latency: 0.03790 s, throughput:   1688.75 token/s
Decode.  median latency: 0.03876 s, median throughput:   1651.03 token/s
Total. latency:  3.549 s, throughput:   9810.84 token/s