rho for Adaptive Sharpness Aware Minimization (ASAM) #37
Description
Hi.
This is Jungmin Kwon, one of the authors of Adaptive Sharpness Aware Minimization (ASAM).
We really appreciate your great implementation!
I have performed cifar10 tests with your code and we found that ASAM with rho=2.0 shows the best accuracy among [0.5, 1.0, 2.0, 5.0].
The test error rates obtained from the grid search are as follows:
| rho | Test error rate |
|---|---|
| 0.5 | 2.75 % |
| 1.0 | 2.69 % |
| 2.0 | 2.55 % |
| 5.0 | 2.90 % |
In our implementation without bias (or beta for BatchNorm) normalization (https://github.com/SamsungLabs/ASAM), ASAM with rho=0.5 shows the best accuracy (2.37 % for WRN16-8), so we performed all the cifar10 tests with rho=0.5.
If you don't mind, could you update the table of test error rate with the result of rho=2.0 (2.55 %)?
Thank you.