Skip to content

IndicQARetrieval task is broken #2726

@AlexeyVatolin

Description

@AlexeyVatolin

Here is the traceback

ERROR:mteb.evaluation.MTEB:Error while evaluating IndicQARetrieval: BuilderConfig 'corpus' not found. Available: ['as-corpus', 'as-qrels', 'as-queries', 'bn-corpus', 'bn-qrels', 'bn-queries', 'gu-corpus', 'gu-qrels', 'gu-queries', 'hi-corpus', 'hi-qrels', 'hi-queries', 'kn-corpus', 'kn-qrels', 'kn-queries', 'ml-corpus', 'ml-qrels', 'ml-queries', 'mr-corpus', 'mr-qrels', 'mr-queries', 'or-corpus', 'or-qrels', 'or-queries', 'pa-corpus', 'pa-qrels', 'pa-queries', 'ta-corpus', 'ta-qrels', 'ta-queries', 'te-corpus', 'te-qrels', 'te-queries']
Traceback (most recent call last):
  File "/home/vatolin/miniconda3/envs/mteb/bin/mteb", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/vatolin/experiments/mteb/mteb/cli.py", line 388, in main
    args.func(args)
  File "/home/vatolin/experiments/mteb/mteb/cli.py", line 146, in run
    eval.run(
  File "/home/vatolin/experiments/mteb/mteb/evaluation/MTEB.py", line 672, in run
    raise e
  File "/home/vatolin/experiments/mteb/mteb/evaluation/MTEB.py", line 576, in run
    task.load_data(**kwargs)
  File "/home/vatolin/experiments/mteb/mteb/abstasks/AbsTaskRetrieval.py", line 294, in load_data
    ).load(split=split)
      ^^^^^^^^^^^^^^^^^
  File "/home/vatolin/experiments/mteb/mteb/abstasks/AbsTaskRetrieval.py", line 93, in load
    self._load_corpus()
  File "/home/vatolin/experiments/mteb/mteb/abstasks/AbsTaskRetrieval.py", line 130, in _load_corpus
    corpus_ds = load_dataset(
                ^^^^^^^^^^^^^
  File "/home/vatolin/miniconda3/envs/mteb/lib/python3.12/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/vatolin/miniconda3/envs/mteb/lib/python3.12/site-packages/datasets/load.py", line 2314, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
                                       ^^^^^^^^^^^^
  File "/home/vatolin/miniconda3/envs/mteb/lib/python3.12/site-packages/datasets/builder.py", line 374, in __init__
    self.config, self.config_id = self._create_builder_config(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vatolin/miniconda3/envs/mteb/lib/python3.12/site-packages/datasets/builder.py", line 601, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'corpus' not found. Available: ['as-corpus', 'as-qrels', 'as-queries', 'bn-corpus', 'bn-qrels', 'bn-queries', 'gu-corpus', 'gu-qrels', 'gu-queries', 'hi-corpus', 'hi-qrels', 'hi-queries', 'kn-corpus', 'kn-qrels', 'kn-queries', 'ml-corpus', 'ml-qrels', 'ml-queries', 'mr-corpus', 'mr-qrels', 'mr-queries', 'or-corpus', 'or-qrels', 'or-queries', 'pa-corpus', 'pa-qrels', 'pa-queries', 'ta-corpus', 'ta-qrels', 'ta-queries', 'te-corpus', 'te-qrels', 'te-queries']

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions