The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @qgallouedec, thanks for setting up the boilerplate.

Are you intending the rewards to be importing as paths to the module or strings in a reward function registry? (that is the impression I got from skimming your changes). As a user, I would think it would be most intuitive to import reward callables from the reward module instead and include them as a List[Callable] type like the current API. Please correct me if I'm misunderstanding. That would also give very simple extensibility for custom reward functions.

If you do intend the registry option, maybe something similar to the gym register environment API would be interesting? For example, you could use a string argument for names of functions in the registry and have a static method to query the existing functions and their descriptions. And for custom reward functions, there could be a register_reward_fn which updates the registry.

Are you intending the rewards to be importing as paths to the module or strings in a reward function registry?

No no, this is just for the script (ie when you run trl grpo --reward_funcs my_lib.my_reward), not for the trainer per se. The idea is not support this in the trainer

If you do intend the registry option, maybe something similar to the gym register environment API would be interesting?

I'm strongly against this idea tbh. Gym registry is a nightmare and not pythonic at all

If you do intend the registry option, maybe something similar to the gym register environment API would be interesting?

I'm strongly against this idea tbh. Gym registry is a nightmare and not pythonic at all

Ah ok 😅, fair enough.

So, to clarify, will the structure remain the same outside of the cli, where the user includes their rewards functions as a list of callables? But, now they can import from this module as well for common reward functions? (Just want to scope out how implementing rewards functions for the module would work).

My idea is that everything remain the same for the user. The only difference would be that they would be able to do

from trl.rewards import think_format_reward

Instead of having to implement it themselves, or copy from another library

Awesome, that makes sense and aligns with what I was thinking as well. Once this change is out, happy to work on adding to the submodule!

Perfect! Thanks!

If you want to start now, you can you create a new branch from this one