The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Here is a rough colab notebook that I have created.

In the notebook:

1. Installed trl from the branch
2. Used the openbmb/RLAIF-V-Dataset dataset
3. Used the HuggingFaceTB/SmolVLM-Instruct model

Here are my queries:

1. Reading the code, we expect the loss_type to be a str with , seperated losses.

   training_args = DPOConfig(
       ...
       loss_type="sigmoid, bco_pair", # <-- a collection of losses
   )

   As far as I understand the MPO paper, it comprises of three losses, the DPO loss, the BCO loss, and the SFT loss. Here I have added the BCO and used sigmoid for the DPO loss, do we also have a way to add the SFT loss here somehow (which I think would be cross entropy loss?)
2. The weighting parameter in the config (as in this comment)

@qgallouedec Here is a rough colab notebook with the current MPO training.

Let me know what you think.

Really nice!!!

But don't do this (important):

    # Apply the chat template
    prompt = processor.apply_chat_template(prompt, tokenize=False)
    chosen = processor.apply_chat_template(chosen, tokenize=False)
    rejected = processor.apply_chat_template(rejected, tokenize=False)

The DPO Trainer handle applying the chat template. See #1930 for more info.

This code snippet is present in so many examples online, it's a scourge. 😩