So I have at least a few times gone to the leaderboard and seen that it was restarted.

This takes quite a while, but once it is done the leaderboard is reasonably fast.

There are two solutions to this problem:

• limit restarts (23 people can now rebuild and someone might rebuild the space if they see some oddities not knowing how long it takes)
• reduce restart time, there are multiple ways to do this
  • speed up result loading (mteb.load_results())
    • 1. reduce the results that are loaded by deleting unused files. E.g. if model 1 is run with ArxivClustering both with and without a revision, we might choose to keep only the one with a revision
    • 2. another solution is the speed up the loading of the results.
  • Avoid recomputing the cache:
    • Once the cache is created loading the data is quite quick. However, the cache is recomputed on rebuilt. We could have the cache updated daily instead.

Edit: As mentioned in #1983 the leaderboard also takes up a decent amount of memory on at least Linux machines

@x-tabdeveloping has been the main person working on this. @x-tabdeveloping feel free to edit this issue if there are solutions you would like to add or remove.