[CI] add kvcache-connector dependency definition and add into CI build #18193

panpan0000 · 2025-05-15T07:53:14Z

FIX #NA (link existing issues this PR will resolve)

As KVCache-Sharing/Offloading become more essential in production, but the current default image vllm/vllm-openai:v0.8.* doesn't include the kv-connector dependencies, for example , when run vllm image with --kv-transfer-config '{"kv_connector":"LMCacheConnector","kv_role":"kv_both","kv_rank":0,"kv_parallel_size":2}'

[engine.py:448]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/lmcache_connector.py", line 37, in __init__
[engine.py:448]     from lmcache.experimental.cache_engine import LMCacheEngineBuilder 
[engine.py:448] ModuleNotFoundError: No module named 'lmcache'

So introducing a new requirements.txt for kv-connector(put lmcache for now, mooncake & nixl later)

And I enable kv-connector into dockerfile in CI scripts, I'm not sure if it's proper or maybe we should do it in vllm-production-stack project? Please correct me.

since we don't have a up to date CI image for vllm+lmcache, the image lmcache/vllm-openai:latest seems still using old vllm(v0.7.0) and image not build from a trackable github CI. @KuntaiDu , any suggestion?

github-actions · 2025-05-15T07:53:23Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

panpan0000 · 2025-06-03T08:37:33Z

@YuhanLiu11 What's your suggestion ?

YuhanLiu11 · 2025-06-03T16:54:06Z

@panpan0000 this is a great idea! Also, now LMCache has bdist package and it can be easily installed without extra compilation.
cc @ApostaC

KuntaiDu · 2025-06-03T17:27:35Z

Yeah agree --- and we can contribute some perf CIs for CPU offloading and KV cache sharing. @youkaichao @simon-mo @DarkLight1337 what's your thoughts on this?

panpan0000 · 2025-06-26T07:44:53Z

So how to push thing forward?

KuntaiDu · 2025-06-27T21:36:41Z

So how to push thing forward?

I am pinging people for feedbacks. Could you please fix the merge conflict first so that we can start the full CI?

panpan0000 · 2025-06-30T02:01:19Z

So how to push thing forward?

I am pinging people for feedbacks. Could you please fix the merge conflict first so that we can start the full CI?

Thanks for reminder. done.

KuntaiDu

LGTM!

KuntaiDu · 2025-06-30T16:53:13Z

I am running some test now to make sure compatibility issues and other things, will get back to you soon.

simon-mo

nit
s/INSTALL_KV_CONNECTOR/INSTALL_KV_CONNECTORS

panpan0000 · 2025-07-02T12:42:49Z

nit s/INSTALL_KV_CONNECTOR/INSTALL_KV_CONNECTORS

done :-) Thanks @simon-mo

KuntaiDu · 2025-07-02T18:24:06Z

I tested this PR and it is functional.

KuntaiDu · 2025-07-02T18:24:48Z

@panpan0000 I enabled automerge, could you please fix the linting errors? Many thanks for your contribution!

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

mergify bot added the ci/build label May 15, 2025

panpan0000 force-pushed the kv-cache-in-dockerfile branch from 4f33c86 to 5a4aeab Compare May 15, 2025 07:55

panpan0000 force-pushed the kv-cache-in-dockerfile branch from 5a4aeab to 3bbaded Compare June 30, 2025 01:58

KuntaiDu approved these changes Jun 30, 2025

View reviewed changes

KuntaiDu added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 30, 2025

simon-mo approved these changes Jun 30, 2025

View reviewed changes

panpan0000 force-pushed the kv-cache-in-dockerfile branch from 3bbaded to 76d6761 Compare July 2, 2025 12:42

KuntaiDu enabled auto-merge (squash) July 2, 2025 18:24

panpan0000 added 2 commits July 3, 2025 11:16

[CI] add kvcache-connector dependency definition and add into CI build

e6299c5

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

parameter changed to INSTALL_KV_CONNECTORS

5cee0d5

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

auto-merge was automatically disabled July 3, 2025 03:16
Head branch was pushed to by a user without write access

panpan0000 force-pushed the kv-cache-in-dockerfile branch from 76d6761 to 5cee0d5 Compare July 3, 2025 03:16

vllm-bot merged commit 5561681 into vllm-project:main Jul 4, 2025
103 of 105 checks passed

sfeng33 pushed a commit to sfeng33/vllm that referenced this pull request Jul 6, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

9a62ef2

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

huydhn pushed a commit to huydhn/vllm that referenced this pull request Jul 8, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

9b92666

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

Chen-zexi pushed a commit to Chen-zexi/vllm that referenced this pull request Jul 13, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

d07c399

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

0576327

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

ApostaC mentioned this pull request Jul 29, 2025

LMCache is causing torch versioning issue in vLLM docker build pipeline LMCache/LMCache#1184

Closed

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

f8a8947

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

e4ba7fc

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[CI] add kvcache-connector dependency definition and add into CI build (

a6493d8

vllm-project#18193) Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[CI] add kvcache-connector dependency definition and add into CI build #18193

[CI] add kvcache-connector dependency definition and add into CI build #18193

Uh oh!

panpan0000 commented May 15, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

panpan0000 commented Jun 3, 2025 •

edited

Loading

Uh oh!

YuhanLiu11 commented Jun 3, 2025

Uh oh!

KuntaiDu commented Jun 3, 2025 •

edited

Loading

Uh oh!

panpan0000 commented Jun 26, 2025

Uh oh!

KuntaiDu commented Jun 27, 2025

Uh oh!

panpan0000 commented Jun 30, 2025

Uh oh!

KuntaiDu left a comment

Uh oh!

KuntaiDu commented Jun 30, 2025

Uh oh!

simon-mo left a comment

Uh oh!

panpan0000 commented Jul 2, 2025

Uh oh!

KuntaiDu commented Jul 2, 2025

Uh oh!

KuntaiDu commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[CI] add kvcache-connector dependency definition and add into CI build #18193

[CI] add kvcache-connector dependency definition and add into CI build #18193

Uh oh!

Conversation

panpan0000 commented May 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

panpan0000 commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YuhanLiu11 commented Jun 3, 2025

Uh oh!

KuntaiDu commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panpan0000 commented Jun 26, 2025

Uh oh!

KuntaiDu commented Jun 27, 2025

Uh oh!

panpan0000 commented Jun 30, 2025

Uh oh!

KuntaiDu left a comment

Choose a reason for hiding this comment

Uh oh!

KuntaiDu commented Jun 30, 2025

Uh oh!

simon-mo left a comment

Choose a reason for hiding this comment

Uh oh!

panpan0000 commented Jul 2, 2025

Uh oh!

KuntaiDu commented Jul 2, 2025

Uh oh!

KuntaiDu commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

panpan0000 commented May 15, 2025 •

edited by github-actions bot

Loading

panpan0000 commented Jun 3, 2025 •

edited

Loading

KuntaiDu commented Jun 3, 2025 •

edited

Loading