Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

TP1:
meta-llama/Meta-Llama-3.1-8B-Instruct
mistralai/Mistral-7B-Instruct-v0.3
deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
google/gemma-2-27b-it

TP2:
meta-llama/Meta-Llama-3.1-70B-Instruct
mistralai/Mistral-7B-Instruct-v0.3
Qwen/Qwen2-72B-Instruct
Qwen/Qwen2-57B-A14B-Instruct

TP8(TBD):
meta-llama/Meta-Llama-3.1-405B-Instruct-FP8
mistralai/Mixtral-8x22B-Instruct-v0.1
deepseek-ai/DeepSeek-Coder-V2-Instruct
xai-org/grok-1
neuralmagic/DeepSeek-Coder-V2-Instruct-FP8

Related resources

No response