loss is nan #6
Description
I train the net from scratch as the redeme said. However, after 5 epoch, the loss is always nan, acc is 0, iou is 0.
Lr: 9.611e-03 | Update: nan mean,nan std | Epoch: [4][990/1047] | Time 1.768 (1.854) | Data 0.618 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:24:23]
Lr: 9.610e-03 | Update: nan mean,nan std | Epoch: [4][1000/1047] | Time 1.738 (1.854) | Data 0.638 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:23:44]
Lr: 9.609e-03 | Update: nan mean,nan std | Epoch: [4][1010/1047] | Time 1.907 (1.854) | Data 0.668 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:23:12]
Lr: 9.608e-03 | Update: nan mean,nan std | Epoch: [4][1020/1047] | Time 1.808 (1.854) | Data 0.648 (0.706) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:22:35]
Lr: 9.608e-03 | Update: nan mean,nan std | Epoch: [4][1030/1047] | Time 1.855 (1.853) | Data 0.633 (0.705) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:21:58]
Lr: 9.607e-03 | Update: nan mean,nan std | Epoch: [4][1040/1047] | Time 1.763 (1.853) | Data 0.627 (0.705) | Loss nan (nan) | acc 0.000 (0.000) | IoU 0.000 (0.000) | [1 day, 18:21:18]