Hi,
I’m submitting this pull request to push the Californian and US splits of the BillSum dataset to MTEB.

BillSum is a dataset created by FiscalNote for the purposes of training and evaluating models capable of summarizing federal and state legislation.

We have reframed the problem in terms of the retrieval of bills based on their summaries, making our reformatted datasets suitable for the evaluation of legal information retrieval models.

We want to improve the coverage of legal domain tasks on MTEB and we believe this dataset will contribute to increasing the diversity and difficulty of MTEB.

This pull request is being submitted courtesy of Isaacus, a legal AI research company.

You may find the original dataset here:
https://huggingface.co/datasets/FiscalNote/billsum

Note that the original dataset contained a large number of examples in both the federal US and Californian test splits and so, we have reduced both splits to 500 randomly selected examples.

Checklist

• I have outlined why this dataset is filling an existing gap in mteb
• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)