Loading evaluation datasets from the glue benchmark require passing a name to the name field (e.g. load_dataset('glue', 'sst2')). Right now the base evaluator data loading code optionally takes split, but it does not take a name field.

The current workaround is to preload your data as a Dataset object so it never hits the data loading function baked into the evaluator, but we'd like to standardize this so harnesses can be defined and run just by writing some JSON.