With @yulinggu-cs we have added a new ce_loss metric which just scores the cross-entropy loss of the gold answer to tasks, which should have positive correlation with actual task performance.

As examples of using this metric, we added these new tasks:

• trivia_qa_wiki_ppl: TrivaQA 8k validation set (Wiki subset, used in Llama-2 paper)
• natural_qs_open_ppl: NaturalQuestions (nq_open validation subset, 3.6k questions, used in Llama papers)
• arc_easy_ppl: Sample example of adding a CE-loss version of existing task (this is somewhat inefficient, redoing part of the earlier evaluation, so should be optimized if we end up doing this for existing tasks)

We log tasks with this metric to a different tab in wandb.