Skip to content

feat: first batch of results for the MTEB(Medical) benchmark #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Dec 3, 2024
Merged

feat: first batch of results for the MTEB(Medical) benchmark #55

merged 6 commits into from
Dec 3, 2024

Conversation

dbuades
Copy link
Contributor

@dbuades dbuades commented Nov 22, 2024 •
edited
Loading

As a follow up to embeddings-benchmark/mteb#1459, this PR contains the results for a list of 15 open source models in the new MTEB(Medical) benchmark.

The models included are:

  • name: "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
    revision: "bf3bf13ab40c3157080a7ab344c831b9ad18b5eb"

  • name: "BAAI/bge-small-en-v1.5"
    revision: "5c38ec7c405ec4b44b94cc5a9bb96e735b38267a"

  • name: "BAAI/bge-base-en-v1.5"
    revision: "a5beb1e3e68b9ab74eb54cfd186867f64f240e1a"

  • name: "BAAI/bge-large-en-v1.5"
    revision: "d4aa6901d3a41ba39fb536a557fa166f842b0e09"

  • name: "intfloat/multilingual-e5-small"
    revision: "fd1525a9fd15316a2d503bf26ab031a61d056e98"

  • name: "intfloat/multilingual-e5-base"
    revision: "d13f1b27baf31030b7fd040960d60d909913633f"

  • name: "intfloat/multilingual-e5-large"
    revision: "ab10c1a7f42e74530fe7ae5be82e6d4f11a719eb"

  • name: "Alibaba-NLP/gte-multilingual-base"
    revision: "7fc06782350c1a83f88b15dd4b38ef853d3b8503"

  • name: "jinaai/jina-embeddings-v3"
    revision: "215a6e121fa0183376388ac6b1ae230326bfeaed"

  • name: "Snowflake/snowflake-arctic-embed-m-v1.5"
    revision: "97eab2e17fcb7ccb8bb94d6e547898fa1a6a0f47"

  • name: "mixedbread-ai/mxbai-embed-large-v1"
    revision: "990580e27d329c7408b3741ecff85876e128e203"

  • name: "abhinand/MedEmbed-small-v0.1"
    revision: "40a5850d046cfdb56154e332b4d7099b63e8d50e"

  • name: "abhinand/MedEmbed-base-v0.1"
    revision: "7a90c50263f620dff743eb9794b89a42bfc5d765"

  • name: "abhinand/MedEmbed-large-v0.1"
    revision: "e621837c7904456dc37d689f97e654424de62318"

  • name: "nvidia/NV-Embed-v2". # Using the code in this PR
    revision: "7604d305b621f14095a1aa23d351674c2859553a"

We also plan to add the following models once the inconsistencies are solved since we also noticed strange results for them:

  • name: "Alibaba-NLP/gte-Qwen2-1.5B-instruct"
    revision: "3276994ba02b26841920728d1adcf115473c88e9"

  • name: "Alibaba-NLP/gte-Qwen2-7B-instruct"
    revision: "e26182b2122f4435e8b3ebecbf363990f409b45b"

Finally, we added a bm25s baseline for the retrieval tasks, although there is an issue with clustering and reranking tasks at the moment.

My colleague @olivierr42 will take it from here since I will not be available next week.

Feel free to suggest other interesting models and we'll happily run them too 💪

@dbuades dbuades marked this pull request as draft November 22, 2024 18:51
@dbuades
Copy link
Contributor Author

dbuades commented Nov 22, 2024

I haven't updated paths.json yet since when running results.py the paths.json that it is generated is very different to the one currently in main. Should we just manually add the results to the paths.json instead?

@KennethEnevoldsen
Copy link
Contributor

@Samoed I see that you are reviewing the related PR (embeddings-benchmark/mteb#1436), will you have the time to take this PR as well?

Samoed
Samoed approved these changes Nov 24, 2024
View reviewed changes
Copy link
Member

@Samoed Samoed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The results look good. If you want your model to appear on the leaderboard, you'll need to generate a paths.json. However, since a new version of the leaderboard is currently being developed, you might want to wait until it's finished

@dbuades dbuades marked this pull request as ready for review December 2, 2024 19:52
@dbuades
Copy link
Contributor Author

dbuades commented Dec 2, 2024 •
edited
Loading

The results look good. If you want your model to appear on the leaderboard, you'll need to generate a paths.json. However, since a new version of the leaderboard is currently being developed, you might want to wait until it's finished

I’m back this week! Since automatically generating the paths.json file is a bit messy at the moment and the new leaderboard is looking good, let’s go ahead and merge just the results for now, as you suggested. We can revisit and make adjustments if needed once the new leaderboard is fully ready.

@KennethEnevoldsen
Copy link
Contributor

I will merge this in for now and then we can resolve the inconsistencies in a separate issue

@KennethEnevoldsen KennethEnevoldsen merged commit 50ba3b9 into embeddings-benchmark:main Dec 3, 2024
2 checks passed
@dbuades dbuades deleted the feat/mteb-medical-results branch December 6, 2024 13:39
This was referenced Dec 7, 2024
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Samoed Samoed Samoed approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dbuades @KennethEnevoldsen @Samoed @oliviercl42