Skip to content

model: add ListConRanker model #2742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 27, 2025

Conversation

tutuDoki
Copy link
Contributor

@tutuDoki tutuDoki commented May 30, 2025

Adding the ListConRanker model discussed in #2584

fixes #2584

Since ListConRanker is a model with a new architecture, it does not support direct loading using the transformers library. We recommend cloning the open-source model and then loading it locally. We have shown an example below and at the top of the code.

import mteb
import sys
model = mteb.get_model('ByteDance/ListConRanker', listconranker_local_path=YOUR_LOCAL_MODEL_PATH)
tasks = mteb.get_tasks(tasks=['T2Reranking], languages=['zho-Hans'])
evaluation = mteb.MTEB(tasks=tasks)
encode_kwargs = {'batch_size': sys.maxsize}
evaluation.run(model, encode_kwargs=encode_kwargs, previous_results=YOUR_LOCAL_PREVIOUS_RESULTS_PATH)

Checklist

  • I did not add a dataset, or if I did, I added the dataset checklist to the PR and completed it.
  • I did not add a model, or if I did, I added the model checklist to the PR and completed it.
    • I have filled out the ModelMeta object to the extent possible
    • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
    • I have tested the implementation works on a representative set of tasks.
    • The model is public, i.e. is available either as an API or the wieght are publicly avaiable to download

@Samoed Samoed changed the title add ListConRanker model model: add ListConRanker model Jun 6, 2025
@tutuDoki
Copy link
Contributor Author

@Samoed Hi! I've updated the implementation of ListConRanker. It can now automatically load models via AutoModelForSequenceClassification. Please review the code. Thanks for your contributions again!

@Samoed Samoed requested a review from KennethEnevoldsen June 23, 2025 18:58
@KennethEnevoldsen KennethEnevoldsen added the new model Questions related to adding a new model to the benchmark label Jun 24, 2025
@tutuDoki
Copy link
Contributor Author

tutuDoki commented Jun 24, 2025

@Samoed Hi, I changed the release date back to 2024-12-11. And I added information about the training datasets. However, some of the training datasets haven’t been released on Hugging Face or MTEB (i.e., cMedQAv1, cMedQAv2, MARC, and CSL). Is it acceptable to add them in this way?

Copy link
Member

@Samoed Samoed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can update datasets like this

@tutuDoki
Copy link
Contributor Author

@Samoed Done, thanks!

@Samoed
Copy link
Member

Samoed commented Jun 24, 2025

I've linted your code. Now CI should pass

@Samoed Samoed merged commit 9d40b7e into embeddings-benchmark:v2.0.0 Jun 27, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new model Questions related to adding a new model to the benchmark
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants