Add AbsTaskInstructionRetrieval from the FollowIR paper #408
Conversation
add comment back in
…k/mteb into instructions
|
@KennethEnevoldsen and @Muennighoff, this is ready to be looked at when you get time. Thanks for your help! |
There was a problem hiding this comment.
Very nice job @orionw! Can you please run the proposed tasks and add the results? You just need to run mteb on the tasks with intfloat/multilingual-e5-small and sentence_transformers/paraphrase-multilingual-MiniLM-L12-v2 and push the JSON files 🙂
|
Thanks @imenelydiaker! I added the small model (e5-small) results I ran as a verification check. I plan on adding a lot more (all the ones in my paper) but since I'm making some minor updates to the dataset I was gonna wait to re-run them all. I also plan to make a PR to create the leaderboard to show these results and another PR to add rerankers to MTEB, so this is just the first of many :) |
mteb/tasks/InstructionRetrieval/eng/Core17InstructionRetrieval.py
Outdated
Show resolved
Hide resolved
mteb/tasks/InstructionRetrieval/eng/Core17InstructionRetrieval.py
Outdated
Show resolved
Hide resolved
mteb/tasks/InstructionRetrieval/eng/Core17InstructionRetrieval.py
Outdated
Show resolved
Hide resolved
|
Thanks @KennethEnevoldsen! Added the metadata and documentation 🙂 |
…nto instructions
|
Everything is looking good. I will merge this in! Very happy with this addition 🎉 |
Real PR for FollowIR (old draft for reference in #356).
Re-uses as much of the
AbsTaskRetrievalabstract task as possible, thanks to previous feedback. Dataset task files are up to date with the newer MTEB metadata. Adds a test to check that things load and that the metric works.I left the sample and n_characters metadata empty, I'm updating the dataset slightly (should be done in a few days) and will make a new PR when it's finished to update the dataset revisions and this metadata.