Benchmark tasks are incorrectly selected #1757
Description
For now, when loading MTEB_ENG_CLASSIC, it provides:
from mteb import MTEB_ENG_CLASSIC
MTEB_ENG_CLASSIC.tasks
# MTEBTasks(..., STS17Crosslingual(name='STS17', languages=['ara', 'deu', 'eng', '...']),
MTEB_ENG_CLASSIC.tasks[-17]
# STS22CrosslingualSTS(name='STS22', languages=['cmn', 'deu', 'eng', '...'])
MTEB_ENG_CLASSIC.tasks[-17].hf_subsets
# ['en', 'de-en', 'es-en', 'pl-en', 'zh-en']However, in the old leaderboard, only the en split is selected https://github.com/embeddings-benchmark/leaderboard/blob/80dd9a2b2d4a0abc368a6cea5f79a517b753951b/config.yaml#L124-L125.
This could be an issue because main score for tasks is averaged
mteb/mteb/load_results/task_results.py
Line 500 in 75d78c1
Or old leaderboard version is not correct?
CC @Muennighoff