Direct Preference Optimization #530
Conversation
There was a problem hiding this comment.
Thanks a LOT @maxjeblick
Very high quality PR, and it works flawless on many different setups and datasets that I tested.
One thing, that we should change in the future is the dataset import. Currently, only the default settings for Causal modeling can be set during import, so one always needs to change that when starting an experiment in e.g. DPO training.
Also, rewards are logged, but never displayed (only when using neptune) -> potential good new feature as you mentioned.
While still not easy to get better results over standard fine-tuning, as DPO is way more user friendly, I am rooting for fully replacing RLHF (by PPO) with it in a subsequent PR.
Minor change needed:
- Rejected Answer Column is missing a tooltip
| experiment_name: str = field(default_factory=generate_experiment_name) | ||
| _parent_experiment: str = "" | ||
| # 7b model may be unstable (NaN loss) | ||
| llm_backbone: str = "h2oai/h2ogpt-4096-llama2-13b-chat" |
There was a problem hiding this comment.
Nitpick:
Maybe, we should replace by a "h2ogpt-gm" fine-tune that already has the same prompting style.
There was a problem hiding this comment.
Do you have a specific model in mind? I could also change the default prompt style values.
There was a problem hiding this comment.
Yes, probably just change default values to something that works. Such as using mistralai/Mistral-7B-Instruct-v0.1 and its prompting style.
This PR adds DPO (https://github.com/eric-mitchell/direct-preference-optimization) as a new problem type.
Also, IPO can be selected via the associated loss function.
Apart from adding the new problem type, the following changes have been made:
Possible follow-up work: