[Model]: Add `transformers` backend support #11330

ArthurZucker · 2024-12-19T09:55:40Z

Adds support for `transformers` as a backend

Following huggingface/transformers#35235, a bunch of models should already be supported, we are ramping up support for more models.

Thanks @Isotr0py for the TP support, and @hmellor for his help as well!
This includes:

trust_remote_code=True support: any model on the hub, if it implements attention the correct way can be natively supported!!
tensor parallel support

Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>

github-actions · 2024-12-19T09:55:53Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

ywang96 · 2024-12-19T11:02:06Z

Hello @ArthurZucker! This is very exciting!

I know this PR is still a draft, but could you provide some context on the scope of this effort? Is it to support any model on transformers?

ArthurZucker · 2024-12-19T16:05:12Z

Yep, overall this should support any model that is supported in transformers, were the cache is "simple" so for now, most of the decoder models and the encoder models for a single modularity!
For MultiModal models, we might need a little bit of extra work, but I thing LLAVA models should work out of the box!

We are refactor our models to make sure it's propagated to as many models as possible!

ArthurZucker · 2024-12-20T10:02:02Z

Might not have time to finish this week, will make it ready for next week 🎄
This should be minimal (no support fort Lora or at least I am not testing it ! This might. need to either call transformers's from pretrain, or replace modules similarly to TP)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…orted Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor · 2025-01-16T18:08:29Z

Benchmarks on A100 using the following command:

python benchmarks/benchmark_throughput.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --dataset ShareGPT_V3_unfiltered_cleaned_split.json

Results:

Class	Result
`LlamaForCausalLM`	Throughput: 12.88 requests/s, 5325.05 total tokens/s, 2554.02 output tokens/s
`TransformersModel`	Throughput: 11.38 requests/s, 4705.90 total tokens/s, 2257.06 output tokens/s

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Isotr0py <2037008807@qq.com>

DarkLight1337 · 2025-02-01T11:30:11Z

Please fix the failing tests

Signed-off-by: Isotr0py <2037008807@qq.com>

DarkLight1337 · 2025-02-01T16:06:47Z

Please also add the distributed transformers test to the distributed tests CI

Signed-off-by: Isotr0py <2037008807@qq.com>

vllm/model_executor/model_loader/utils.py

Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Isotr0py <2037008807@qq.com>

@Isotr0py

# Adds support for `transformers` as a backend Following huggingface/transformers#35235, a bunch of models should already be supported, we are ramping up support for more models. Thanks @Isotr0py for the TP support, and @hmellor for his help as well! This includes: - `trust_remote_code=True` support: any model on the hub, if it implements attention the correct way can be natively supported!! - tensor parallel support --------- Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

jgong5 · 2025-08-04T11:07:21Z

vllm/model_executor/models/transformers.py

+            attn_metadata=attn_metadata,
+            intermediate_tensors=intermediate_tensors,
+            attention_instances=self.attention_instances,
+            return_dict=False)[0][0, ...]  # we remove batch dimension for now


@ArthurZucker May I know why we want to remove the batch dim here?

vLLM doesn't expect a batch dimension in the hidden states returned by the model's forward() method

@ArthurZucker Just curious, do you have plan to support models with MLA such as DeepSeek? It seems this integration only supports normal attention right now?

Yes happy to work on every and all model compatibility!

ArthurZucker and others added 2 commits December 19, 2024 10:33

Merge

0bb5519

Co-authored-by: Isotr0py <41363108+Isotr0py@users.noreply.github.com>

Merge branch 'main' into transformers-backend

8e238f7

mergify bot added the ci/build label Dec 19, 2024

ArthurZucker added 3 commits December 19, 2024 11:01

revert some changes

6d8f1fd

changes are now merged with main of transformers

fb37617

revert more changes

2d0c128

DarkLight1337 requested review from mgoin and ywang96 December 19, 2024 10:05

Isotr0py mentioned this pull request Dec 19, 2024

[Roadmap] vLLM Roadmap Q4 2024 #9006

Closed

40 tasks

simon-mo mentioned this pull request Jan 9, 2025

[Roadmap] vLLM Roadmap Q1 2025 #11862

Closed

38 tasks

hmellor added 13 commits January 9, 2025 11:35

Merge remote-tracking branch 'upstream/main' into fix-history

31c16a1

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Undo whitespace changes

a49aa81

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Merge remote-tracking branch 'upstream/main' into fix-history

ae2e1cf

Update transformers pin

ff19ade

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Remove unreachable code

038604b

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Remove dead code

882ef81

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Update to latest attention interface

f254f2c

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Always try to load TransformersModel if model isn't explicitly supp…

5a1a833

…orted Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Temporarily remove Llama from registry

b7de34d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Deduplicate registry code slightly

49c4616

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix profiling of Attentions

071246d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Run ./format.sh on transformers.py

6190591

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Fix spelling

7ae8262

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

hmellor and others added 2 commits January 16, 2025 19:08

Undo changes to chat.py

988586d

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

tests + md

5313551

DarkLight1337 enabled auto-merge (squash) February 1, 2025 06:22

fix custom model test

95c1916

Signed-off-by: Isotr0py <2037008807@qq.com>

DarkLight1337 disabled auto-merge February 1, 2025 11:29

Isotr0py added 2 commits February 1, 2025 23:00

fix incorrect backend fallback

2906626

Signed-off-by: Isotr0py <2037008807@qq.com>

fix oot registration test

8c33bd6

Signed-off-by: Isotr0py <2037008807@qq.com>

add transformers tp test

ccbff79

Signed-off-by: Isotr0py <2037008807@qq.com>

hmellor reviewed Feb 1, 2025

View reviewed changes

vllm/model_executor/model_loader/utils.py Outdated Show resolved Hide resolved

hmellor reviewed Feb 1, 2025

View reviewed changes

vllm/model_executor/model_loader/utils.py Outdated Show resolved Hide resolved

jeejeelee approved these changes Feb 2, 2025

View reviewed changes

Isotr0py and others added 2 commits February 2, 2025 10:50

Update vllm/model_executor/model_loader/utils.py

3647766

Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

clean up

f68af01

Signed-off-by: Isotr0py <2037008807@qq.com>

Isotr0py enabled auto-merge (squash) February 2, 2025 02:55

youkaichao disabled auto-merge February 3, 2025 13:30

youkaichao merged commit a1a2aaa into vllm-project:main Feb 3, 2025
69 of 71 checks passed

DarkLight1337 mentioned this pull request Feb 3, 2025

Release v0.7.3 #12465

Closed

2 tasks

comaniac mentioned this pull request Feb 3, 2025

[Misc] Add missing license headers #12703

Closed

hmellor deleted the fix-history branch February 5, 2025 10:34

XuehaiPan mentioned this pull request Apr 16, 2025

[RFC][Feature][Model] Add templated fallback HF transformers model backend in SRT sgl-project/sglang#5471

Closed

6 tasks

jgong5 reviewed Aug 4, 2025

View reviewed changes

mergify bot added the new-model Requests to new models label Aug 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Model]: Add `transformers` backend support #11330

[Model]: Add `transformers` backend support #11330

Uh oh!

ArthurZucker commented Dec 19, 2024 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Dec 19, 2024

Uh oh!

ywang96 commented Dec 19, 2024

Uh oh!

ArthurZucker commented Dec 19, 2024

Uh oh!

ArthurZucker commented Dec 20, 2024

Uh oh!

hmellor commented Jan 16, 2025

Uh oh!

DarkLight1337 commented Feb 1, 2025

Uh oh!

DarkLight1337 commented Feb 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgong5 Aug 4, 2025

Uh oh!

hmellor Aug 4, 2025 •

edited

Loading

Uh oh!

jgong5 Aug 19, 2025

Uh oh!

ArthurZucker Aug 20, 2025

Uh oh!

Uh oh!

Uh oh!

[Model]: Add transformers backend support #11330

[Model]: Add transformers backend support #11330

Uh oh!

Conversation

ArthurZucker commented Dec 19, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Adds support for transformers as a backend

Uh oh!

github-actions bot commented Dec 19, 2024

Uh oh!

ywang96 commented Dec 19, 2024

Uh oh!

ArthurZucker commented Dec 19, 2024

Uh oh!

ArthurZucker commented Dec 20, 2024

Uh oh!

hmellor commented Jan 16, 2025

Uh oh!

DarkLight1337 commented Feb 1, 2025

Uh oh!

DarkLight1337 commented Feb 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgong5 Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

hmellor Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgong5 Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

ArthurZucker Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

[Model]: Add `transformers` backend support #11330

[Model]: Add `transformers` backend support #11330

ArthurZucker commented Dec 19, 2024 •

edited by github-actions bot

Loading

Adds support for `transformers` as a backend

hmellor Aug 4, 2025 •

edited

Loading