-
Notifications
You must be signed in to change notification settings - Fork 463
Open
Labels
enhancementNew feature or requestNew feature or requestwontfixThis will not be worked onThis will not be worked on
Description
Currently MTEB does not consistently use a ground truth for vector manipulation. A solution would be to adopt the Array API spec. A potentially simpler solution is to use either torch or numpy.
- The benefit of using numpy is that it is very stable and rarely leads to OOM and related GPU issues.
- The benefit of using PyTorch is that it potentially allows evaluation using GPUs (currently, sklearn does not support this, but might in the future, when they adopt they API spec)
partly related to the discussion in: #883
This is not a pressing concern atm.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestwontfixThis will not be worked onThis will not be worked on