Skip to content

[KTO] learning rate recomentations for kto #2070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Sep 16, 2024
Merged

[KTO] learning rate recomentations for kto #2070

merged 5 commits into from
Sep 16, 2024

Conversation

kashif
Copy link
Collaborator

@kashif kashif commented Sep 15, 2024

What does this PR do?

recommendation for the KTO learning rate to be small. See: https://x.com/ethayarajh/status/1834780782126153763

@kashif kashif requested a review from lewtun September 15, 2024 10:31
@qgallouedec qgallouedec added the 🏋 KTO Related to KTO label Sep 15, 2024
@qgallouedec
Copy link
Member

What do you think about overriding the default lr of TrainingArgument in this case? It seems to be more convenient for the user.

cc @kawine @corbt

@dataclass
class KTOConfig(TrainingArguments):
    ...
    learning_rate: float = 5e-7

kawine
kawine reviewed Sep 16, 2024
View reviewed changes
@kawine
Copy link
Contributor

kawine commented Sep 16, 2024 •
edited
Loading

thanks for taking the initiative @qgallouedec ! I just had one suggestion re: the docs.

@kashif
Copy link
Collaborator Author

kashif commented Sep 16, 2024

@qgallouedec done

@qgallouedec qgallouedec changed the title [KT0] learning rate recomentations for kto [KTO] learning rate recomentations for kto Sep 16, 2024
qgallouedec
qgallouedec reviewed Sep 16, 2024
View reviewed changes
@qgallouedec
Copy link
Member

Thanks, for the record, we should extend this to all our configs, when necessary

@qgallouedec qgallouedec merged commit cdafc93 into main Sep 16, 2024
10 checks passed
@qgallouedec qgallouedec deleted the kto-learning-rate branch September 16, 2024 10:24
@qgallouedec qgallouedec mentioned this pull request Sep 18, 2024
Merged
5 tasks
@qgallouedec qgallouedec mentioned this pull request Sep 27, 2024
Merged
5 tasks
yxliu-TAMU pushed a commit to mincheolseong/ECEN743-GRPO-Project-Proposal that referenced this pull request Apr 20, 2025
@kashif @qgallouedec
[KTO] Overrides default learning_rate in KTOConfig (huggingface#2070
fcb6074 
)

* learning rate recomentations for kto

* update from suggestion

* override default lr

* add tip tag

* Update trl/trainer/kto_config.py

---------

Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qgallouedec qgallouedec qgallouedec approved these changes

@lewtun lewtun Awaiting requested review from lewtun

+1 more reviewer

@kawine kawine kawine left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
🏋 KTO Related to KTO
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kashif @qgallouedec @kawine