Waiting for the CI to pass

Pleasure working with you, @qgallouedec . Expect more PRs from me in the future!

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

For historical reasons, I set "k1" to be the default, so everyone's scripts will run the same way as before after the update. We should consider setting "k3" to be the default because it is lower variance, unbiased, and trivial to compute (it just takes an exp and two subtractions).

If you can provide results showing that in the general case, k3 is a better choice, we can change the default value (in a follow up PR).

How would you go about showing that for the general case?
For one, what experiments would I do? I could find three projects which use PPOTrainer, and rerun the code with both estimators, but that is hardly showing that kx is better in the general case.
And how would I evaluate results? KL is the measure of coherence, and I wouldn't just be able to use one of the KL estimators themselves because that would be unfair to the other (unless both estimators agreed that the kx estimator results in lower KL divergence)?

What's important here is to know whether using k3 makes it possible to converge faster and/or towards better models.
My advice would be to take 2 or 3 different configurations (datasets/models) and run the training, then evaluate the checkpoints (the type of evaluation depends on the dataset).

@qgallouedec
I notice that other scripts, such as DPOTrainer use cap_exp instead of exp as a protection against exponentiating extreme values. Should we use that here, too?

I would think it should be unnecessary, because PPOTrainer is on-policy, so logprobs cannot be huge (since that would imply sampling a really unlikely generation/token), but I thought I'd check in.

I'd say, unless we encounter this issue, we can keep the code as is