import mteb
from mteb import MTEB

model = mteb.get_model("intfloat/multilingual-e5-small")
tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])

eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)

evaluation.run(
    model,
    eval_splits=eval_splits,
    corpus_chunk_size=500,
)

INFO:sentence_transformers.SentenceTransformer:Use pytorch device_name: mps
INFO:sentence_transformers.SentenceTransformer:Load pretrained SentenceTransformer: intfloat/multilingual-e5-small
INFO:mteb.models.sentence_transformer_wrapper:Model prompts will be overwritten with {'query': 'query: ', 'passage': 'passage: '}
INFO:mteb.evaluation.MTEB:

## Evaluating 1 tasks:
────────────────────────────────────────────────────────────────────── Selected tasks  ───────────────────────────────────────────────────────────────────────
Retrieval
    - NFCorpus, s2p


INFO:mteb.evaluation.MTEB:

********************** Evaluating NFCorpus **********************
INFO:mteb.abstasks.dataloaders:Loading Corpus...
INFO:mteb.abstasks.dataloaders:Loaded 3633 TEST Documents.
INFO:mteb.abstasks.dataloaders:Doc Example: {'id': 'MED-10', 'title': 'Statin Use and Breast Cancer Survival: A Nationwide Cohort Study from Finland', 'text': 'Recent studies have suggested that statins, an established drug group in the prevention of cardiovascular mortality, could delay or prevent breast cancer recurrence but the effect on disease-specific mortality remains unclear. We evaluated risk of breast cancer death among statin users in a population-based cohort of breast cancer patients. The study cohort included all newly diagnosed breast cancer patients in Finland during 1995–2003 (31,236 cases), identified from the Finnish Cancer Registry. Information on statin use before and after the diagnosis was obtained from a national prescription database. We used the Cox proportional hazards regression method to estimate mortality among statin users with statin use as time-dependent variable. A total of 4,151 participants had used statins. During the median follow-up of 3.25 years after the diagnosis (range 0.08–9.0 years) 6,011 participants died, of which 3,619 (60.2%) was due to breast cancer. After adjustment for age, tumor characteristics, and treatment selection, both post-diagnostic and pre-diagnostic statin use were associated with lowered risk of breast cancer death (HR 0.46, 95% CI 0.38–0.55 and HR 0.54, 95% CI 0.44–0.67, respectively). The risk decrease by post-diagnostic statin use was likely affected by healthy adherer bias; that is, the greater likelihood of dying cancer patients to discontinue statin use as the association was not clearly dose-dependent and observed already at low-dose/short-term use. The dose- and time-dependence of the survival benefit among pre-diagnostic statin users suggests a possible causal effect that should be evaluated further in a clinical trial testing statins’ effect on survival in breast cancer patients.'}
INFO:mteb.abstasks.dataloaders:Loading Queries...
INFO:mteb.abstasks.dataloaders:Loaded 323 TEST Queries.
INFO:mteb.abstasks.dataloaders:Query Example: {'id': 'PLAIN-2', 'text': 'Do Cholesterol Statin Drugs Cause Breast Cancer?'}
WARNING:mteb.evaluation.MTEB:Evaluating multiple MTEB runs simultaniously will produce incorrect CO₂ results
Password:

make test 
--- 🧪 Running tests ---
pytest -n auto --durations=5
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.12.7, pytest-8.3.4, pluggy-1.5.0
rootdir: /Users/A200009373/Documents/Coding/mteb
configfile: pyproject.toml
plugins: cov-6.0.0, typeguard-4.4.1, anyio-4.7.0, xdist-3.6.1
10 workers [2476 items]   
...................................................................................................................................................... [  6%]
...................................................................................................................................................... [ 12%]
...................................................................................................................................................... [ 18%]
...................................................................................................................................................... [ 24%]
...................................................................................................................................................... [ 30%]
...................................................................................................................................................... [ 36%]
...................................................................................................................................................... [ 42%]
...................................................................................................................................................... [ 48%]
............................Password:..............................................