[v2] Merge main #2485
Merged
[v2] Merge main #2485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mplement CV-Bench (#2414) * refactor CV-Bench * reimplement CV Bench * remove abstask/evaluator/tests for Any2TextMultipleChoice * rerun descriptive stats
fix: Add option to remove leaderboard from leaderboard fixes #2413 This only removed the benchmark from the leaderboard but keep it in MTEB.
* Added VDR Multilingual Dataset * address comments * make lint * Formated Dataset for retrieval * Update mteb/tasks/Retrieval/multilingual/VdrMultilingualRetrieval.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * Update mteb/tasks/Retrieval/multilingual/VdrMultilingualRetrieval.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * make lint * corrected date * fix dataset building * move to image folder --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
* pin setuptools * pin setuptools * pin setuptools in makefile * try ci * fix ci * remove speed from installs
…tering folder (#2422) * add PatentFnBClustering.py * do make lint and revise * rollback Makefile * Update mteb/tasks/Clustering/kor/PatentFnBClustering.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * klue_mrc_domain * make lint * klue_modified_clustering_dataset * clustering & kor folder add __init.py * clustering & kor folder add __init__.py * task.py roll-back * correct text_creation to sample_creation & delete form in MetaData * correct task_subtype in TaskMetaData * delete space * edit metadata * edit task_subtypes --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
* add richinfoai models add richinfoai models * format codes by linter format codes by linter
* Fix typos; add chrono order * Fix spacing
* Add model specific dependencies in pyproject.toml * Update documentation
…mplement r-Oxford and r-Paris (#2442) * MutipleChoiceEvaluationMixin; reimplement r-Oxford and r-Paris; rerun stats * modify benchmark list * fix citation
* Added meta information about SearchMap_Preview model to the model_dir * Added meta information about SearchMap_Preview model to the model_dir * updated revision name * Device loading and cuda cache cleaning step left out * removed task instructions since it's not necessary * changed sentence transformer loader to mteb default loader and passed instructions s model prompts * Included searchmap to the models overview page * Included searchmap to the models overview page * added meta data information about where model was adpated from * Update mteb/models/searchmap_models.py * fix lint * lint --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com>
* Add Background Gradients in Summary and Task Table * Remove warnings and add light green cmap * Address comments * Separate styling function * address comments * added comments
* add ops_moa_models * add custom implementations * Simplify custom implementation and format the code * support SentenceTransformers * add training datasets * Update mteb/models/ops_moa_models.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * update training_datasets --------- Co-authored-by: kunka.xgw <kunka.xgw@taobao.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
…mplement ImageCoDe (#2468) * reimplement ImageCoDe with ImageTextPairClassification * add missing stats file
* feat: added pubmedbert model2vec models * fix: attribute model_name * fix: fixed commit hash for pubmed_bert model2vec models * fix: changes requested in PR 2443
* add_nb_sbert_model * Update nb_sbert.py added n_parameters and release_date * Update mteb/models/nb_sbert.py Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> * Update nb_sbert.py fix: make lint * added nb_sbert to overview.py + ran make lint * Update nb_sbert.py Fix error: Input should be a valid date or datetime, month value is outside expected range of 1-12 --------- Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
* supress logging warnings * remove loggers * return blocks * rename function * fix gme models * add server name * update after merge * fix ruff
rename VisionCentric to VisionCentricQA
Update dataset_loading.yml
isaac-chung
approved these changes
Apr 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Code Quality
make lintto maintain consistent style.Documentation
Testing
make test-with-coverage.make testormake test-with-coverageto ensure no existing functionality is broken.Adding datasets checklist
Reason for dataset addition: ...
mteb -m {model_name} -t {task_name}command.sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2intfloat/multilingual-e5-smallself.stratified_subsampling() under dataset_transform()make test.make lint.Adding a model checklist
mteb.get_model(model_name, revision)andmteb.get_model_meta(model_name, revision)