🚀 Feature

Improve validation output during training.

Motivation

IMHO it could be very helpful to improve the visualization output during training already. Either by distributing into Wandb/Tensorboard or by output image results of validation since these are calculated anyway. Of course this would increase storage consumption during training but the advantages are very big.

Beside of simple monitor the training process this would also help to isolate hardest training examples at earliest possible.

Is there such a mechanism switch already available which would enhance the training output already?
Do you have any hints what might be further beneficial to achieve the same goals other than post evaluating the model?