-
Notifications
You must be signed in to change notification settings - Fork 732
Closed
Description
Is your feature request related to a problem? Please describe
SGLang could be used for optimizing multi round chat.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Since sglang relies on vllm, for now, we can add a variable environment to enable sglang when vllm is enabled.
SGLang only supported limited models(https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models), we could support those only.