Skip to content

Conversation

ekolodin
Copy link
Contributor

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the results files checker make pre-push.

Adding a model checklist

Ref #

  • I have added model implementation to mteb/models/ directory. Instruction to add a model can be found here

@Samoed
Copy link
Member

Samoed commented Dec 20, 2024

Could you run MiraclReranking with the current MTEB version since its score format has changed? Also, could you add the implementation to the MTEB repository?

Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay on this

The result format is good.

Just to make you aware you will have to submit a model meta to the mteb repo for it to appear on the leaderboard.

The model meta file is also unfilled

@Samoed
Copy link
Member

Samoed commented Feb 5, 2025

@KennethEnevoldsen I was trying to add this model in embeddings-benchmark/mteb#1741

@KennethEnevoldsen
Copy link
Contributor

Ah yes I see - I could have seen that from the reference

@KennethEnevoldsen
Copy link
Contributor

Re-reviewed this PR. Seems like the revision is missing. I don't think we can merge it without it. However @Samoed's implementation has been merged so it should be easy to rerun.

Will close this PR for now, but do open a new one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants