Motivation

This PR aims to fix the issue #3545 by enhancing the engine with the support of vision language models such as Qwen2-VL for offline inference.

Modifications

First of all, it should be noted that the design of current code has some issues which make this PR an imperfect solution, as a result, I have not added any documentation or unit test yet and wish to look for discussion on how to improve the code as a whole.

The root causes of #3545 are that:

1. VLMs require the prompt to be pre-processed by the chat template. This is because that the processor will add in some image-related tokens in the prompt. For exmaple, Qwen2-VL adds the following tokens to the prompt: <|vision_start|><|image_pad|><|vision_end|>. Thus, VLMs are different from LLMs in the sense that the chat template is a must for VLM but not always for LLMs (LLMs can still generate sensible outputs even without the template).
2. sgl.Engine is not responsible for applying the chat template to the prompts
3. the current API for applying the chat template is only for online serving, i.e. v1_chat_generate_request

As a result, the current proposed workflow for running VLMs offline is shown below:

It is not so elegant because it counter-intuitively uses a API used for online serving in the offline inference scenario. I would suggest we extract the preprocessing logic to independent APIs or create a new API for preprocessing in offline cases.

Open for discussion.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.