Proof of concept implementation for dask training callback. As it is not consistent with single node implementation, this is an early PR for review and discussion.

The rational behind a new style of callback function is that, the original one is modelled after R implementation. The R impl uses language environment for storing states, while Python doesn't have such facility so the original callback creates an artificial one to be consistent with R. I believe the new style is cleaner, native to Python, and easier customize. With the new interface, #5238 and #5495 can be easily implemented.

To provide a short introduction to new callback function, here is its interface:

class Callback:
    def __init__(self):
        self.history = {}
        pass

    def before_training(self, model):
        pass

    def after_training(self, model):
        pass

    def before_iteration(self, model, epoch):
        return False

    def after_iteration(self, model, epoch):
        return False

All the other callbacks is sub-class of this Callback. This interface is suitable for both single node and distributed training. Function return value is whether training should be stopped, False if not. By using a return value, we can deprecate EarlyStopException, which is created for one particular callback.

To-dos:

• Make sure no data copy is performed.
• Get correct starting rounds: Obtain num boosted rounds directly from booster. #5511 is closed so I need to figure out a way to get correct starting round.
• Be compatible with original callback function. The new Callback can be a wrapper of original callback, I need to port the implementation to non-dask training.
• Support user defined metric.
• Tests.

Future plan

If the new design is approved, I would like to deprecate the following parameters:

• In scikit-learn interface fit function eval_set, eval_metric, early_stopping_rounds, verbose, sample_weight_eval_set,
• From train function evals, feval, maximize, early_stopping_rounds, evals_result, verbose_eval=True
• lastly corresponding parameters in dask.

Related:

#4955
#3822