Add tokenization_kwargs to encode for embedding model truncation #21033
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the tokenization_kwargs parameter to the encode methods in both the V0 and V1 async LLM engines. This is a valuable addition that enables passing tokenizer-specific arguments, like truncation settings, directly to the encode function, which is particularly useful for embedding models.
I've reviewed the changes in vllm/engine/async_llm_engine.py and vllm/v1/engine/async_llm.py. The implementation correctly adds the new parameter and passes it through the respective call stacks to the input preprocessor. The changes are consistent, well-scoped to the stated objective, and I did not find any issues of high or critical severity.
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
This pull request has merge conflicts that must be resolved before it can be |
5c6f52d to
12a3fc7
Compare
|
Can you fix pre-commit? |
|
emmm, I’m a bit confused, which one are you referring to? |
|
Oh, got it, I try to fix it. |
Pull Request is not mergeable
…m-project#21033) Signed-off-by: qizixi <qizixi@meta.com>
…m-project#21033) Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>
…m-project#21033) Signed-off-by: shuw <shuw@nvidia.com>
…m-project#21033) Signed-off-by: x22x22 <wadeking@qq.com>
…m-project#21033) Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>
…m-project#21033) Signed-off-by: Paul Pak <paulpak58@gmail.com>
…m-project#21033) Signed-off-by: Diego-Castan <diego.castan@ibm.com>
Currently, embedding api has supported prompt truncation, while the encode function of async engine (both V1 and V0 engine) does not support long prompt truncation. This PR adds tokenization_kwargs to encode function of both V1 and V0 engine, allowing direct use of engine.encode to truncate inputs for embedding models.