From #1329:
Figure out what to do with folder structure. Should we create a folder like evaluation/evaluators/{modality}, or do we want to do it in another way? I also have some concerns re. evaluators (as I would like them to be the same across text and images, but I see that it might not be possible)