Hi there. Not sure why it's not pinned in the setup so opening an issue instead of a PR directly fixing. Evaluate uses the import

from datasets import DownloadMode

here.

However, this is only possible in datasets>=2.0.0 (just tried with 1.18.4) so it looks like this is the minimal version of Datasets required for evaluate. If that's the case, the setup should pin it otherwise downstream libraries might get weird failures (we just had one reported from the DeepSpeed CI for Accelerate, so it goes a long way down ;-) ). If there is a way to use older versions of Datasets, then a fix should be made for that import.

cc @lvwerra