Skip to content

FEAT: integrate SGLang #1097

@qinxuye

Description

@qinxuye

Is your feature request related to a problem? Please describe

SGLang could be used for optimizing multi round chat.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Since sglang relies on vllm, for now, we can add a variable environment to enable sglang when vllm is enabled.

SGLang only supported limited models(https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models), we could support those only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions