early_stopping_rounds does not work in R but does work in Python with custom objective function #5878
Description
early_stopping_rounds with a custom objective function and eval metric appears to hang on the first round and then stop in R. With Python, this behavior does not happen with the same data and functions. This appears to be a bug specific to R.
## Custom objective and error, ten rounds
[1] train-error:0.046522 eval-error:0.042831
[2] train-error:0.038692 eval-error:0.037244
[3] train-error:0.049900 eval-error:0.049038
[4] train-error:0.025641 eval-error:0.027933
[5] train-error:0.012590 eval-error:0.013656
[6] train-error:0.013358 eval-error:0.015518
[7] train-error:0.007830 eval-error:0.008070
[8] train-error:0.001996 eval-error:0.001862
[9] train-error:0.001996 eval-error:0.001862
[10] train-error:0.001996 eval-error:0.001862
## With early stopping rounds = 3 on the above data
Multiple eval metrics are present. Will use eval_error for early stopping.
Will train until eval_error hasn't improved in 3 rounds.
[1] train-error:0.046522 eval-error:0.042831
[2] train-error:0.046522 eval-error:0.042831
[3] train-error:0.046522 eval-error:0.042831
[4] train-error:0.046522 eval-error:0.042831
Stopping. Best iteration:
[1] train-error:0.046522 eval-error:0.042831
In contrast, this is the output from Python:
[14:35:46] 6513x127 matrix with 143286 entries loaded from /data/pmahler/agaricus.txt.train
[14:35:46] 1611x127 matrix with 35442 entries loaded from /data/pmahler/agaricus.txt.test
[0] eval-rmse:0.61194 train-rmse:0.61767 eval-my-error:0.04283 train-my-error:0.04652
[1] eval-rmse:1.14566 train-rmse:1.14783 eval-my-error:0.03724 train-my-error:0.03869
[2] eval-rmse:1.63907 train-rmse:1.64891 eval-my-error:0.04904 train-my-error:0.04990
[3] eval-rmse:2.08198 train-rmse:2.09920 eval-my-error:0.02793 train-my-error:0.02564
[4] eval-rmse:2.30665 train-rmse:2.32896 eval-my-error:0.01366 train-my-error:0.01259
[5] eval-rmse:2.69563 train-rmse:2.72393 eval-my-error:0.01552 train-my-error:0.01336
[6] eval-rmse:3.00988 train-rmse:3.03360 eval-my-error:0.00807 train-my-error:0.00783
[7] eval-rmse:3.03567 train-rmse:3.06537 eval-my-error:0.00186 train-my-error:0.00200
[8] eval-rmse:3.36498 train-rmse:3.39902 eval-my-error:0.00186 train-my-error:0.00200
[9] eval-rmse:3.73496 train-rmse:3.77317 eval-my-error:0.00186 train-my-error:0.00200
Multiple eval metrics have been passed: 'train-my-error' will be used for early stopping.
Will train until train-my-error hasn't improved in 2 rounds.
[0] eval-rmse:0.61194 train-rmse:0.61767 eval-my-error:0.04283 train-my-error:0.04652
[1] eval-rmse:1.14566 train-rmse:1.14783 eval-my-error:0.03724 train-my-error:0.03869
[2] eval-rmse:1.63907 train-rmse:1.64891 eval-my-error:0.04904 train-my-error:0.04990
[3] eval-rmse:2.08198 train-rmse:2.09920 eval-my-error:0.02793 train-my-error:0.02564
[4] eval-rmse:2.30665 train-rmse:2.32896 eval-my-error:0.01366 train-my-error:0.01259
[5] eval-rmse:2.69563 train-rmse:2.72393 eval-my-error:0.01552 train-my-error:0.01336
[6] eval-rmse:3.00988 train-rmse:3.03360 eval-my-error:0.00807 train-my-error:0.00783
[7] eval-rmse:3.03567 train-rmse:3.06537 eval-my-error:0.00186 train-my-error:0.00200
[8] eval-rmse:3.36498 train-rmse:3.39902 eval-my-error:0.00186 train-my-error:0.00200
[9] eval-rmse:3.73496 train-rmse:3.77317 eval-my-error:0.00186 train-my-error:0.00200
Stopping. Best iteration:
[7] eval-rmse:3.03567 train-rmse:3.06537 eval-my-error:0.00186 train-my-error:0.00200
R Details: R version 4.0.2 on Mac OS 10.15.5; XGBoost 1.1.1.1
Python: 3.6 on Ubuntu 18.04; XGB 1.1.1.1
I have attached two scripts in the above zip file, one in Python and one in R, that reproduce the issue with data shipped with XGB and used in examples.
Metadata
Metadata
Assignees
Labels
No labels