[v2] Can't compute stats for reranking datasets #2972
Description
Describe the bug
@Samoed seems like I can't compute descriptive stats for reranking datasets:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
File /Users/au56[1](https://file+.vscode-resource.vscode-cdn.net/Users/au561649/Github/mteb/tmp.py:1)649/Github/mteb/tmp.py:1
----> 1 mteb.get_task("JQaRAReranking").calculate_metadata_metrics()
File ~/Github/mteb/mteb/abstasks/AbsTask.py:313, in AbsTask.calculate_metadata_metrics(self, overwrite_results)
311 descriptive_stats[split][hf_subset_stat][hf_subset] = split_details
312 else:
--> [313](https://file+.vscode-resource.vscode-cdn.net/Users/au561649/Github/mteb/~/Github/mteb/mteb/abstasks/AbsTask.py:313) split_details = self._calculate_metrics_from_split(split)
314 descriptive_stats[split] = split_details
316 with self.metadata.descriptive_stat_path.open("w") as f:
File ~/Github/mteb/mteb/abstasks/AbsTaskRetrieval.py:430, in AbsTaskRetrieval._calculate_metrics_from_split(self, split, hf_subset, compute_overall)
426 if "default" in self.dataset and split != "default":
427 return self._calculate_metrics_from_split(
428 split=split, hf_subset="default"
429 )
--> [430](https://file+.vscode-resource.vscode-cdn.net/Users/au561649/Github/mteb/~/Github/mteb/mteb/abstasks/AbsTaskRetrieval.py:430) split_data = self.dataset["default"][split]
431 queries = split_data["queries"]
432 corpus = split_data["corpus"]
KeyError: 'default'
The task in questions are:
XGlueWPRReranking # has a unique error
JQaRAReranking
JaCWIRRerankingTo reproduce
mteb.get_task("JaCWIRReranking").calculate_metadata_metrics()
mteb.get_task("JQaRAReranking").calculate_metadata_metrics()
mteb.get_task("XGlueWPRReranking").calculate_metadata_metrics()
Additional information
No response
Are you interested to contribute a fix for this bug?
Yes