[Model][VLM] Support JinaVL Reranker #20260

shineran96 · 2025-06-30T13:19:27Z

Purpose

Support JinaVLForRanking architectures ✅
Support multimodal for LLM.score ✅
Support multimodal for v1/score ✅
Support multimodal for v1/rerank ✅

Usage

Offline Inference

from vllm import LLM

model_name = "jinaai/jina-reranker-m0"

def get_model() -> LLM:
    """Initializes and returns the LLM model for JinaVL Reranker."""
    return LLM(
        model=model_name,
        task="score",
        dtype="float16",
        mm_processor_kwargs=mm_processor_kwargs,
        limit_mm_per_prompt=limit_mm_per_prompt,
    )


def main() -> None:
    query = ["slm markdown"]
    documents = {
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
                },
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
                },
            },
        ]
    }

    model = get_model()
    outputs = model.score(query, documents)

    print("-" * 30)
    print([output.outputs.score for output in outputs])
    print("-" * 30)
if __name__ == "__main__":
    main()

Output as follows

------------------------------
[0.4957047998905182, 0.7898827195167542]
------------------------------

Online Serving

vllm serve jinaai/jina-reranker-m0

Test v1/score with curl

curl -X 'POST' \
  'http://127.0.0.1:8000/v1/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "jinaai/jina-reranker-m0",
  "data_1": "slm markdown",
  "data_2": {
    "content": [
      {
        "type": "image_url",
        "image_url": {
          "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
        }
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
        }
      }
    ]
  }
}'

Response as follows

{"id":"score-f577ae2d2de54e83b5a721975ac9ecf1","object":"list","created":1751472415,"model":"jinaai/jina-reranker-m0","data":[{"index":0,"object":"score","score":0.4957047998905182},{"index":1,"object":"score","score":0.7898827195167542}],"usage":{"prompt_tokens":1162,"total_tokens":1162,"completion_tokens":0,"prompt_tokens_details":null}}

Test v1/rerank with curl

curl -X 'POST' \
  'http://127.0.0.1:8000/v1/rerank' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "jinaai/jina-reranker-m0",
  "query": "slm markdown",
  "documents": {
    "content": [
      {
        "type": "image_url",
        "image_url": {
          "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
        }
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
        }
      }
    ]
  }
}'

Response as follows

{"id":"rerank-b2924bf3dafc4544be5a4e267eb09785","model":"jinaai/jina-reranker-m0","usage":{"total_tokens":1162},"results":[{"index":1,"document":{"data":{"image_url":{"url":"https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"},"type":"image_url"}},"relevance_score":0.7898827195167542},{"index":0,"document":{"data":{"image_url":{"url":"https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"},"type":"image_url"}},"relevance_score":0.4957047998905182}]}

Fix #18447

gemini-code-assist

Summary of Changes

Hello @shineran96, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces comprehensive support for the JinaVL Reranker model, significantly expanding the system's capabilities to handle multimodal reranking tasks. It modifies the LLM.score API to seamlessly integrate image inputs alongside text, enabling users to perform cross-encoding with both modalities. The changes include the core model implementation, specialized prompt formatting for multimodal inputs, and extensive testing to ensure accuracy and reliability.

Highlights

New Model Support: Introduced support for the JinaVLForRanking model, enabling multimodal (text and image) reranking capabilities within the system.
Multimodal score API Extension: The LLM.score API has been extended to accept image inputs alongside text for cross-encoding tasks, specifically tailored for the JinaVL model. This allows for flexible input combinations like text-image, image-text, and image-image pairs.
Core Model Implementation: Added the JinaVLForSequenceClassification model definition under vllm/model_executor/models/jina_vl.py. This implementation integrates with the existing Qwen2VLForConditionalGeneration and includes a custom JinaVLScorer for the ranking head.
Multimodal Prompt Formatting: A new formatting_prompts utility was added to vllm/entrypoints/score_utils.py. This function is crucial for correctly constructing prompts and handling multimodal data for JinaVL, including inserting image placeholders and appending the specific score token ID (100).
Comprehensive Testing: New tests were added in tests/models/language/pooling/test_jinavl_reranker.py to validate the JinaVL Reranker across various multimodal scenarios (text-image, text-text, image-text, image-image), ensuring functional parity with the original Hugging Face implementation.
Documentation and Examples: The docs/models/supported_models.md file was updated to list the newly supported JinaVL model, and a new offline inference example script (examples/offline_inference/jinavl_reranker.py) was provided to demonstrate its usage.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-06-30T13:20:32Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

The pull request introduces support for the JinaVL Reranker model within the vLLM framework, including necessary code modifications and tests. The changes primarily involve adding new model classes, updating configurations, and adjusting input processing to accommodate multi-modal data. The code appears well-structured and includes necessary components for integrating the JinaVL model, but some docstrings and error messages could be improved for clarity.

vllm/entrypoints/llm.py

vllm/entrypoints/score_utils.py

vllm/model_executor/models/jina_vl.py

mergify · 2025-07-03T14:06:43Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shineran96.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337

Thanks for adding this model! Some initial comments.

cc @maxdebayser @noooop @Isotr0py

tests/models/language/pooling/test_jinavl_reranker.py

docs/models/supported_models.md

tests/models/registry.py

vllm/entrypoints/openai/protocol.py

vllm/entrypoints/score_utils.py

vllm/model_executor/models/jina_vl.py

tests/models/language/pooling/test_jinavl_reranker.py

examples/offline_inference/jinavl_reranker.py

vllm/entrypoints/llm.py

maxdebayser

There are a few places that need changes or clarifications but its a nice addition overall. Thanks for contributing.

vllm/entrypoints/llm.py

vllm/entrypoints/score_utils.py

vllm/model_executor/models/jina_vl.py

noooop · 2025-07-04T04:15:44Z

This is the first supported VL Reranker, thank you for getting it working.

mergify · 2025-07-07T14:52:29Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shineran96.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: shineran96 <shinewang96@gmail.com>

tests/models/multimodal/pooling/test_jinavl_reranker.py

vllm/entrypoints/llm.py

shineran96 · 2025-07-10T13:13:22Z

Please fix pre-commit

There are some code updating involving this PR in upstream branch, so I have to rebase and do some code changes.
Please help to review again, thanks!

Signed-off-by: shineran96 <shinewang96@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

Signed-off-by: shineran96 <shinewang96@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

linchen111 · 2025-08-12T03:54:09Z

I encounter this: Following weights were not initialized from checkpoint: {'language_model.score.weight'}

I use vllm 0.10.0

DarkLight1337 · 2025-08-12T04:55:58Z

I think this problem should be fixed on main branch, you can try installing vLLM from source or use the nightly Docker image

linchen111 · 2025-08-12T06:48:15Z

I think this problem should be fixed on main branch, you can try installing vLLM from source or use the nightly Docker image

Thankyou ,fixed

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Paul Pak <paulpak58@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com>

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Diego-Castan <diego.castan@ibm.com>

Signed-off-by: shineran96 <shinewang96@gmail.com>

mergify bot added documentation Improvements or additions to documentation frontend labels Jun 30, 2025

gemini-code-assist bot reviewed Jun 30, 2025

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/score_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/models/jina_vl.py Outdated Show resolved Hide resolved

shineran96 force-pushed the jina-reranker-m0 branch from 00851cd to 4f1a1e0 Compare July 3, 2025 12:22

shineran96 marked this pull request as ready for review July 3, 2025 13:52

shineran96 requested review from hmellor, DarkLight1337, ywang96 and aarnphm as code owners July 3, 2025 13:52

mergify bot added needs-rebase new-model Requests to new models labels Jul 3, 2025

shineran96 force-pushed the jina-reranker-m0 branch from f484fb6 to f0cb47f Compare July 3, 2025 14:35

shineran96 changed the title ~~[WIP][Model][VLM] Support JinaVL Reranker~~ [Model][VLM] Support JinaVL Reranker Jul 3, 2025

mergify bot removed the needs-rebase label Jul 3, 2025

DarkLight1337 reviewed Jul 3, 2025

View reviewed changes

Isotr0py reviewed Jul 3, 2025

View reviewed changes

maxdebayser reviewed Jul 3, 2025

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

maxdebayser suggested changes Jul 3, 2025

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

vllm/entrypoints/score_utils.py Outdated Show resolved Hide resolved

vllm/model_executor/models/jina_vl.py Outdated Show resolved Hide resolved

mergify bot added the multi-modality Related to multi-modality (#4194) label Jul 4, 2025

shineran96 requested review from robertgshaw2-redhat and simon-mo as code owners July 5, 2025 00:28

shineran96 force-pushed the jina-reranker-m0 branch from e592e5d to 55779e0 Compare July 7, 2025 12:12

mergify bot added the needs-rebase label Jul 7, 2025

shineran96 force-pushed the jina-reranker-m0 branch from 55779e0 to fa12128 Compare July 7, 2025 15:49

mergify bot removed the needs-rebase label Jul 7, 2025

shineran96 added 6 commits July 10, 2025 08:27

fix v1 engine

2eee812

Signed-off-by: shineran96 <shinewang96@gmail.com>

some fix

a7d2440

Signed-off-by: shineran96 <shinewang96@gmail.com>

add SupportsScoreTemplate

3e31e03

Signed-off-by: shineran96 <shinewang96@gmail.com>

some fix

e6803e9

Signed-off-by: shineran96 <shinewang96@gmail.com>

ruff-format

83ec36b

Signed-off-by: shineran96 <shinewang96@gmail.com>

fix mypy

2a8adfe

Signed-off-by: shineran96 <shinewang96@gmail.com>

auto-merge was automatically disabled July 10, 2025 13:03
Head branch was pushed to by a user without write access

shineran96 force-pushed the jina-reranker-m0 branch from efa7843 to 2a8adfe Compare July 10, 2025 13:03

DarkLight1337 reviewed Jul 10, 2025

View reviewed changes

tests/models/multimodal/pooling/test_jinavl_reranker.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Jul 10, 2025

View reviewed changes

vllm/entrypoints/llm.py Outdated Show resolved Hide resolved

some fix

9dae5f0

Signed-off-by: shineran96 <shinewang96@gmail.com>

vllm-bot merged commit 4bed167 into vllm-project:main Jul 10, 2025
91 of 95 checks passed

DarkLight1337 mentioned this pull request Jul 10, 2025

[New Model]: jinaai/jina-embeddings-v4 #20463

Open

1 task

Liuchenlong mentioned this pull request Jul 11, 2025

[Bug]: JinaVLForRanking 400 BadRequest #20804

Closed

1 task

Chen-zexi pushed a commit to Chen-zexi/vllm that referenced this pull request Jul 13, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

14cc27e

Signed-off-by: shineran96 <shinewang96@gmail.com>

patrickvonplaten pushed a commit to patrickvonplaten/vllm that referenced this pull request Jul 15, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

853ad2b

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>

LyrisZhong pushed a commit to LyrisZhong/vllm that referenced this pull request Jul 23, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

d4e7dc9

Signed-off-by: shineran96 <shinewang96@gmail.com>

avigny pushed a commit to avigny/vllm that referenced this pull request Jul 31, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

dadfaa7

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: avigny <47987522+avigny@users.noreply.github.com>

Pradyun92 pushed a commit to Pradyun92/vllm that referenced this pull request Aug 6, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

647ceb0

Signed-off-by: shineran96 <shinewang96@gmail.com>

npanpaliya pushed a commit to odh-on-pz/vllm-upstream that referenced this pull request Aug 6, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

4896abe

Signed-off-by: shineran96 <shinewang96@gmail.com>

jinzhen-lin pushed a commit to jinzhen-lin/vllm that referenced this pull request Aug 9, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

62f5f0c

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Jinzhen Lin <linjinzhen@hotmail.com>

paulpak58 pushed a commit to paulpak58/vllm that referenced this pull request Aug 13, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

4750d8a

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Paul Pak <paulpak58@gmail.com>

taneem-ibrahim pushed a commit to taneem-ibrahim/vllm that referenced this pull request Aug 14, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

15c82af

Signed-off-by: shineran96 <shinewang96@gmail.com>

diegocastanibm pushed a commit to diegocastanibm/vllm that referenced this pull request Aug 15, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

454e743

Signed-off-by: shineran96 <shinewang96@gmail.com> Signed-off-by: Diego-Castan <diego.castan@ibm.com>

epwalsh pushed a commit to epwalsh/vllm that referenced this pull request Aug 27, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

2cea2b3

Signed-off-by: shineran96 <shinewang96@gmail.com>

googlercolin pushed a commit to googlercolin/vllm that referenced this pull request Aug 29, 2025

[Model][VLM] Support JinaVL Reranker (vllm-project#20260)

c0b7b51

Signed-off-by: shineran96 <shinewang96@gmail.com>

Uh oh!

[Model][VLM] Support JinaVL Reranker #20260

[Model][VLM] Support JinaVL Reranker #20260

Uh oh!

Conversation

shineran96 commented Jun 30, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Usage

Offline Inference

Online Serving

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions bot commented Jun 30, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Jul 3, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxdebayser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

noooop commented Jul 4, 2025

Uh oh!

mergify bot commented Jul 7, 2025

Uh oh!

Uh oh!

Uh oh!

shineran96 commented Jul 10, 2025

Uh oh!

Uh oh!

linchen111 commented Aug 12, 2025

Uh oh!

DarkLight1337 commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linchen111 commented Aug 12, 2025

Uh oh!

Uh oh!

shineran96 commented Jun 30, 2025 •

edited by github-actions bot

Loading

DarkLight1337 commented Aug 12, 2025 •

edited

Loading