TaskMetadata eval_langs validation robustness #2500
Description
- Currently, a TaskMetadata with
eval_langs=["eng_Latn"]will passtests/test_TaskMetadata.py::test_all_metadata_is_filled_and_valid(when it should fail). The test should be refactored such that this case can be caught - Since we currently check lang and script with this line:
lang, script = code.split("-"), we should likely check if"-"is in the code to avoid the uncaught error. - Consider using a field_validator instead of running this in pytest? (Cannot when we still have HISTORIC_DATASETS)