Skip to content

[mieb] Winoground not working #1383

@Muennighoff

Description

@Muennighoff
ERROR:mteb.evaluation.MTEB:Error while evaluating Winoground: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
Traceback (most recent call last):
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 81, in <module>
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch_size=1)
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run
    raise e
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 394, in run
    task.load_data(eval_splits=task_eval_splits, **kwargs)
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 192, in load_data
    self.dataset = datasets.load_dataset(**self.metadata_dict["dataset"])  # type: ignore
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/datasets/load.py", line 2132, in load_dataset
    builder_instance = load_dataset_builder(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/datasets/load.py", line 1890, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/datasets/builder.py", line 353, in __init__
    info.update(self._info())
  File "/data/huggingface/modules/datasets_modules/datasets/facebook--winoground/72585f4d9cd5a28790bb9bc2adbdd45633f36dfbf85df529e0756e114e134285/winoground.py", line 61, in _info
    return datasets.DatasetInfo(
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'

Metadata

Metadata

Assignees

No one assigned

    Labels

    miebThe image extension of MTEB

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions