What does this PR do?

As part of the refactoring efforts, this PR consolidate the actor config in ppo-trainer and ppo-megatron-trainer. The shared configs are in actor/actor.yaml, and both megatron_actor.yaml and dp_actor.yaml inherits from it. If the change is reasonable we can expand it to critic, ref, rollout, data, etc.

Backward compatibility:

• making ppo_trainer hierarchical means that we rely on hydra to perform config resolution.
• using hydra with ppo_trainer.yaml or any config inheriting from ppo_trainer.yaml works fine as usually, however
• directly loading ppo_trainer.yaml with OmegaConf now need to change to the following:

-        cfg = OmegaConf.load("verl/trainer/config/ppo_trainer.yaml")
+        import os
+
+        from hydra import compose, initialize_config_dir
+
+        with initialize_config_dir(config_dir=os.path.abspath("verl/trainer/config")):
+            cfg = compose(config_name="ppo_trainer")

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For backward compatibility, a copy of the original legacy configs are stored in the test folder. A test is added to make sure the loaded new config is equivalent to legacy config. This test, however, should be disabled in the future since we will only maintain and update the new modular config going forward.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace.