So for one model that supports MRL the current interface is:

import mteb

tasks = mteb.get_tasks(tasks=["LccSentimentClassification"])
eval = mteb.MTEB(tasks=tasks)


meta = mteb.get_model_meta("openai/text-embedding-3-small")


model = meta.load_model()  # Default embed_dim is 1536
results = eval.run(model, overwrite_results=True)
print(f"Score: {results[0].get_score()} with embedding dimension {meta.embed_dim}")
# Score: 0.5946666666666667 with embedding dimension 1536

# changing the embedding dimension to 256
model = meta.load_model(embed_dim=256)
results = eval.run(model, overwrite_results=True)
print(f"Score: {results[0].get_score()} with embedding dimension 256")
# Score: 0.5706666666666667 with embedding dimension 256

In v1 or v2, nothing is saved to distinguish these two runs, though.

I am not sure what the best solution is, but I am probably leaning toward:

# Proposed interface:
results = eval.run(model, overwrite_results=True, encode_kwargs={"embed_dim": 256})

and then save the encode_kwargs in the model_meta.json? (Though it would be overwritten if you run it with a different set of encode_kwargs, it would also hit the cache across embed_dim settings.)

We could also have it save it as a separate models as @Samoed suggests, but that seems a bit frustrating to me.

Essentially:
openai/text-embedding-3-small, default embedding size
openai/text-embedding-3-small (embed_dim=256)

That is probably the easiest short-term fix, but it will be frustrating once we get more models.

Handle MRL #2832

Description

Enable MRL at various levels for evaluation

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions