Skip to content

Conversation

Samoed
Copy link
Member

@Samoed Samoed commented Mar 1, 2025

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

KennethEnevoldsen and others added 30 commits February 24, 2025 15:23
This simplified the test and also make it a lot simpler. It also removed about 100 test cases which where all to the same API call.
Added a few missing annotations for nvidia-embed
Automatically generated by python-semantic-release
* fix: Update NVIDIA-Embed training data

Added a few missing annotations for nvidia-embed

* fix update annotationf for voyage exp
Automatically generated by python-semantic-release
…2146)

* feat: Add Qodo-Embed-1-7B model metadata and rename existing model

* lint

* fix revision

* update license name

---------

Co-authored-by: Tal Sheffer <tal.s@codium.ai>
Automatically generated by python-semantic-release
add Any2AnyRetrievalDescriptiveStatistics
* Added zero-shot percentages and different filtering scheme

* Update mteb/model_meta.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

---------

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Automatically generated by python-semantic-release
The URL pointed to the settings page instead of the main repo URL. Now it is fixed.
* redo to voyage to only training data

* Add training data annotation for Kalm embeddings #2168

* Add correct training data annotations to Stella #2164

* removed fiqa PL as it does not exist

* remove ArxivClusteringS2S.v2 as it does not exist

* Add training data annotation for GIST embedding #2166

* fix max tokens for kalm models #2162

* remove eli 5
Automatically generated by python-semantic-release
* add mieb and mieb-lite to benchmarks

* add CompositionalityEvaluation and DocumentUnderstanding types

* add VisionCentric type

* add missing comma

* split STS17MultilingualVisualSTS and STSBenchmarkMultilingualSTS to eng and non-eng

* use aggregate task instead so we can name the subsets

* shorten names

* fix import

* alternative strategy to avoid using get_task

* follow other aggregate tasks and skip metadata test

* run LB without errors when selecting MIEB(-lite)

* add back the capability as taks type

* typo

* extend description

* split into mieb(eng) and mieb(multilingual)

* remove unneeded files

* remove aggtask additions for test

* edit descriptions based on screenshots

* shorten

* rename to Compositionality and include ImageCoDeT2IMultiChoice

* re-tag missing VisionCentric tasks

* re-tag rparis and roxford as retrieval and include fixes

* re-tag voc2007 as image cls

* make lint

* correct num task types in descriptions

* add one model to models_to_annotate

* add mieb reference models

* update task types

* relabel to multilingual retrieval task type to align with paper

* fix reference and bibtex

* edit task list to match with final list

* add back agg task to reproduce table column in paper

* fix filtering and import

* update tests

* mieb lite add back missing tasks

* fix metadata test

* multi should have all 4 variants

* fix task counts

* lite has 10 task types

* fix visualSTS-17 lang splits

* Aggregate task can now use subsets & eval langs to filter TaskResults

* fix test and mark VisualSTS17 as multilingual

* fix tests

* add agg task running script

* add voyage meta

* fix citations

* capitalize

* add coarse/fine labels

---------

Co-authored-by: gowitheflow-1998 <jsbs54@durham.ac.uk>
Automatically generated by python-semantic-release
* feat: update training datasets and revision for jina models

* feat: update training datasets and revision for jina models
* redo to voyage to only training data

* Add training data annotation for Kalm embeddings #2168

* Add correct training data annotations to Stella #2164

* removed fiqa PL as it does not exist

* remove ArxivClusteringS2S.v2 as it does not exist

* Add training data annotation for GIST embedding #2166

* fix max tokens for kalm models #2162

* remove eli 5

* fix: add training data for Bilingual Embeddings

fixes #2167
Automatically generated by python-semantic-release
This also resolves the missing data in the leaderboard.

Fixes #2172
* Added training data annotation for MMLW models

* Added GIST annotations Kenneth missed

* Added Stella en 400m training data'
Automatically generated by python-semantic-release
Automatically generated by python-semantic-release
Samoed and others added 16 commits March 1, 2025 16:20
* add similar datasets

* add nano

* update is filled

* Update mteb/abstasks/TaskMetadata.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
* add labse annotation

* Update mteb/models/sentence_transformers_models.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

---------

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
* Fixed leaderboard crash

* Fixed language selection error

* Ran linting
Automatically generated by python-semantic-release
* Added training  data annotation for bge-gemma

* Added missing annotations for Voyage models

* Added training data for sts-multilingual-mpnet

* Added all mteb datasets to STS-multilingual training data
Automatically generated by python-semantic-release
* model loading and get_text_embeddings

* add image_emb, fused_emb, and calc probs methods

* add b16 model

* add llm2clip_openai_l_14_224 (not working yet)

* got llm2clip_openai_l_14_224 working

* make lint

* add training sets and allow py files
* refactor dataset checking

* increase timeout

* increase timeout

* remove timeout
* Add Any2AnyMC descriptive stats

* Add descriptive stats function for ImageTextPC

* add descriptive stats examples

* linter

* update multi choice descriptive stats
* fix: Add training data annotations to uderver-bloom models

fixes #2193

* fix: add mixedbread

---------

Co-authored-by: Márton Kardos <power.up1163@gmail.com>
Automatically generated by python-semantic-release
* remove model size from voyage-3-m-exp model

* Update mteb/models/voyage_models.py

* Update mteb/models/voyage_models.py
# Conflicts:
#	mteb/abstasks/Image/AbsTaskAny2AnyMultiChoice.py
#	mteb/models/bge_models.py
#	mteb/models/e5_instruct.py
#	mteb/models/e5_models.py
Samoed and others added 3 commits March 4, 2025 09:15
* refactor dataset checking

* increase timeout

* increase timeout

* remove timeout

* start

* automatically find datasets

* update comment

* fix aggregate task metadata

* fixes

* lint

* rename

* update fetch check
# Conflicts:
#	mteb/models/ru_sentence_models.py
@Samoed
Copy link
Member Author

Samoed commented Mar 4, 2025

@isaac-chung can you review this PR? There is mostly changes to MIEB tasks

Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MIEB parts look good, thanks!

@Samoed Samoed merged commit d491800 into v2.0.0 Mar 4, 2025
9 checks passed
@Samoed Samoed deleted the merge_main branch March 4, 2025 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v2 Issues and PRs related to `v2` branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants