For users whose whole pipeline is in GPU memory, having predict return results on GPU device memory would be optimal. This particularly comes up in the case of hyperparameter sweeps, where a GPU-based scoring function is available (and fast), but we have to bring predictions to host memory.

Perhaps something like booster.predict(test_dmatrix, device="gpu0") would work as an API, with a default to device="cpu"?