Skip to content

Leaderboard 2.0: should we fetch data from model card? #1373

@KennethEnevoldsen

Description

@KennethEnevoldsen

Currently, model metadata files become quite extensive if they cover all tasks in mteb (#1368). This makes it frustrating for users to see what scores a model has (having scores for >30 datasets doesn't really give a great overview). The ideal solution would probably be to either:

  1. Allow users to upload an mteb_results/* folder
  2. Only accept results pushed to embedding-benchmark/results

Let me know what you guys think (@Muennighoff, @imenelydiaker, @isaac-chung, @orionw, @x-tabdeveloping)

Metadata

Metadata

Assignees

No one assigned

    Labels

    leaderboardissues related to the leaderboard

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions