We currently do not have a good approach to adding experiment runs to the MTEB leaderboard e.g. experiment with the influence of hyperparameters. Such as using no-prompt for prompt-based models or changing the embedding size.

A solution to implement is to run it with a unique model_name, but there is currently no documentation on how one would need to do that.

edit: a potentially better solution is to add experiments. Where we add a layer to the visualization:

model_name
| - revision1
|    | - task1.json
     | - ...
     | - experiments
     |    | - exp_name1
     |    |    | -  task1.json
     |    |    | - ...
     |    | - exp_name1
     |    |    | -  task1.json
     |    |    | - ...

or potentially a slightly more consistent structure:

model_name
| - revision1
     | - default
     |    | -  task1.json
     |    | - ...
     | - exp_name1
     |    | -  task1.json
     |    | - ...

Experiment names could be e.g. no-instruct/instruct (whether you use instructions) or emb_size=256

To run a model as an experiment we could do:

model = MyModel(**custom_kwargs)
results = mteb.MTEB(model, experiment_name = "no-instruct")