Summary

known issues in v0.4.0 release

PPO

• There's a bug when calculating ppo vf loss, fixed in [algo] fix: vf_loss factor #2016

SFT

• a bug in max_position_embedding [trainer] fix: fix sft max_position_embeddings #2019

Breaking changes after v0.4 (main branch)

Megatron

• Megatron lr schedule config change: [feat][BREAKING] Megatron: Support learning rate scheduler #1701

# BEFORE (deprecated)
actor_rollout_ref:
  optim:
    lr: 1.0e-5
    warmup_ratio: 0.1      # ❌ now removed

# AFTER
actor_rollout_ref:
  optim:
    lr: 1.0e-5
    min_lr: 5.0e-6
    lr_warmup_init: 0.0
    lr_warmup_steps: 1000
    lr_decay_steps: 50000
    lr_decay_style: linear     # {linear|cosine|constant}
    use_checkpoint_opt_param_scheduler: true
# Apply the same block under critic.optim if you train an explicit value model.

SFT

• verl/trainer/config/sft_trainer.yaml: https://github.com/volcengine/verl/pull/1908/files
  • prompt_dict_keys: default val changed from ['question'] to null
  • response_dict_keys: default val changed from ['answer'] to null

PPO ray trainer

• verl.trainer.ppo.ray_trainer._timer is replaced by simple_timer in [perf] feat: Add verl profiling support from Nvidia Nsight System #1820. We noticed many ppl import it although its a private function.

Checkpoint manager

• the Megatron checkpoint directory layout is changed in [megatron] feat: Support of dist checkpoint #2125
• Checkpoint manager constructor now takes checkpoint_config as the keyword to replace checkpoint_contents

Deprecations after v0.4 (main branch)

• FSDP1 is being deprecated. Please use fsdp2