fix: Filling missing metadata for leaderboard release #1895
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -21,12 +21,12 @@ class BiossesSTS(AbsTaskSTS): | |||
| eval_langs=["eng-Latn"], | |||
| main_score="cosine_spearman", | |||
| date=None, | |||
There was a problem hiding this comment.
Do we not have any information on the dates?
There was a problem hiding this comment.
No, actually it's hard to infer the dates. I assumed you only needed domains so I filled them in pirority.
|
Also, remember to run linting :D |
7 tasks
|
Looks like tests are failing due to (old?) metadata - Pydantic validation: |
|
@x-tabdeveloping I tried to fill a maximum number of missing metadata for the tasks you listed, I used mostly the data we put in the paper. I don't have the |
4 tasks
|
Thanks for this fix @imenelydiaker - added annotations from #1910 |
KennethEnevoldsen
approved these changes
Jan 30, 2025
|
Thanks for the work @imenelydiaker :D |
isaac-chung
added a commit
that referenced
this pull request
Feb 3, 2025
* Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.4 Automatically generated by python-semantic-release * Update tasks table * fix: Limited plotly version to be less than 6.0.0 (#1902) Limited plotly version to be less than 6.0.0 * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * update stella/jasper metainfo (#1896) update stella meta * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.5 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Feat: Add FaMTEB (Farsi/Persian Text Embedding Benchmark) (#1843) * Add Summary Retrieval Task * Add FaMTEBClassification * Add FaMTEBClustering * Add FaMTEBPairClassification * Add FaMTEBRetrieval and BEIRFA and FaMTEBSTS * Add FaMTEBSummaryRetrieval * Add FaMTEB to benchmarks * fix benchmark names * temporary fix metadata * Fix dataset revisions * Update SummaryRetrievalEvaluator.py * Update task files * Update task files * add data domain and subtask description * Update AbsTaskSummaryRetrieval and FaMTEBSummaryRetrieval * Update AbsTaskSummaryRetrieval * Add mock task * Update AbsTaskSummaryRetrieval * Update AbsTaskSummaryRetrieval * make lint * Refactor SummaryRetrieval to subclass BitextMining * Add aggregated datasets --------- Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> * Update tasks table * Docs: update docs according to current state (#1870) * update docs * Apply suggestions from code review Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * update readme * Update README.md Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Adding a banner to the new MMTEB leaderboard (#1908) * Adding a banner to the new MMTEB leaderboard * linting * Update mteb/leaderboard/app.py Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * adding reference to mteb arena --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: Filling missing metadata for leaderboard release (#1895) * Update ArxivClusteringS2S.py * fill some metadat for retrieval * fill in the reste of missing metadata * fix metadata * fix climatefever metadata * fix: Added CQADupstack annotations * removed annotation for non-exisitant task * format * Added financial to other financial dataset * Moved ArguAna annotation to derivate datasets --------- Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.6 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: remove SummaryRetrieval as a type (#1915) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: revert rename and add to description (#1918) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * docs: Add sort to domains for task metadata (#1922) Tests currently go into an infinite loop. This should prevent that. * Update tasks table * 1.31.7 Automatically generated by python-semantic-release * docs: Updated citation for mteb(scandinavian) (#1914) fix: Updated citation for mteb(scandinavian) * fix: Add datasets in CodeRAG-Bench (#1595) * add three out of four datasets in CodeRAG-Bench * add verified CodeRAGStackoverflowPostsRetrieval dataset * clean up code and make some comments * fixed lint errors * addressed comments about code-rag datasets: fixed grammar and remove unnessary code and loop * roll back files which is not supposed to change * fixed the comments in split_by_first_newline() and make the methods private by adding a underscore prefix * refactor to use common args * update task descriptions * add entry in benchmarks * correct the alphanumeric order for the dataset * add in tasks.md * add in tasks.md * update task metadata * update importing path * fix lint errors * correct CodeRAG task metadata description field and id for stackoverflow-posts * fix error in test --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * 1.31.8 Automatically generated by python-semantic-release * Leaderboard: Acks (#1930) Add acs * omit instructions.py --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Márton Kardos <power.up1163@gmail.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Mehran Sarmadi <128898167+mehran-sarmadi@users.noreply.github.com> Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> Co-authored-by: Wissam Siblini <36303760+wissam-sib@users.noreply.github.com> Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> Co-authored-by: Pengfei He <hepengfe@gmail.com> Co-authored-by: Niklas Muennighoff <n.muennighoff@gmail.com>
Samoed
added a commit
that referenced
this pull request
Feb 4, 2025
* Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * update stella/jasper metainfo (#1896) update stella meta * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.5 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Feat: Add FaMTEB (Farsi/Persian Text Embedding Benchmark) (#1843) * Add Summary Retrieval Task * Add FaMTEBClassification * Add FaMTEBClustering * Add FaMTEBPairClassification * Add FaMTEBRetrieval and BEIRFA and FaMTEBSTS * Add FaMTEBSummaryRetrieval * Add FaMTEB to benchmarks * fix benchmark names * temporary fix metadata * Fix dataset revisions * Update SummaryRetrievalEvaluator.py * Update task files * Update task files * add data domain and subtask description * Update AbsTaskSummaryRetrieval and FaMTEBSummaryRetrieval * Update AbsTaskSummaryRetrieval * Add mock task * Update AbsTaskSummaryRetrieval * Update AbsTaskSummaryRetrieval * make lint * Refactor SummaryRetrieval to subclass BitextMining * Add aggregated datasets --------- Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> * Update tasks table * Docs: update docs according to current state (#1870) * update docs * Apply suggestions from code review Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * update readme * Update README.md Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Adding a banner to the new MMTEB leaderboard (#1908) * Adding a banner to the new MMTEB leaderboard * linting * Update mteb/leaderboard/app.py Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * adding reference to mteb arena --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: Filling missing metadata for leaderboard release (#1895) * Update ArxivClusteringS2S.py * fill some metadat for retrieval * fill in the reste of missing metadata * fix metadata * fix climatefever metadata * fix: Added CQADupstack annotations * removed annotation for non-exisitant task * format * Added financial to other financial dataset * Moved ArguAna annotation to derivate datasets --------- Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * 1.31.6 Automatically generated by python-semantic-release * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: remove SummaryRetrieval as a type (#1915) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * fix: revert rename and add to description (#1918) * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * Update tasks table * docs: Add sort to domains for task metadata (#1922) Tests currently go into an infinite loop. This should prevent that. * Update tasks table * 1.31.7 Automatically generated by python-semantic-release * docs: Updated citation for mteb(scandinavian) (#1914) fix: Updated citation for mteb(scandinavian) * fix: Add datasets in CodeRAG-Bench (#1595) * add three out of four datasets in CodeRAG-Bench * add verified CodeRAGStackoverflowPostsRetrieval dataset * clean up code and make some comments * fixed lint errors * addressed comments about code-rag datasets: fixed grammar and remove unnessary code and loop * roll back files which is not supposed to change * fixed the comments in split_by_first_newline() and make the methods private by adding a underscore prefix * refactor to use common args * update task descriptions * add entry in benchmarks * correct the alphanumeric order for the dataset * add in tasks.md * add in tasks.md * update task metadata * update importing path * fix lint errors * correct CodeRAG task metadata description field and id for stackoverflow-posts * fix error in test --------- Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> * Update tasks table * 1.31.8 Automatically generated by python-semantic-release * update __init__ * update generate_imports script for aggregational tasks * add descriptive stats * remove print from script generate_imports * add rest of metadata * fix tests * add todo for test * Revert "fix tests" This reverts commit 7e8be03. * add back check for multilingual * fix imports --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Mehran Sarmadi <128898167+mehran-sarmadi@users.noreply.github.com> Co-authored-by: mehran <mehan.sarmadi16@gmail.com> Co-authored-by: e.zeinivand <zeinivand@ymail.com> Co-authored-by: Erfun76 <59398902+Erfun76@users.noreply.github.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> Co-authored-by: Wissam Siblini <36303760+wissam-sib@users.noreply.github.com> Co-authored-by: Imene Kerboua <33312980+imenelydiaker@users.noreply.github.com> Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> Co-authored-by: Pengfei He <hepengfe@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to issue #1886.