If anyone is curious here is my run on the Alpaca dataset using another decoder model (codegen-16B-nl). Appears the dataset isn't diverse, multiple closely related answers. I believe this dataset is not capable of generalizing well to new data.

The loss from the original Alpaca training script follows a similar pattern used in OPT-IML to compute loss based on the label.

My run on codegen-16B-nl

Another user's run on LLaMA 7B

Some more discussion: https://twitter.com/abacaj/status/1637310768780648448