volcengine#274)

Related issue: volcengine#273

- Add `remove_previous_ckpt_in_save` and `del_local_ckpt_after_load`
configuration option in `ppo_trainer.yaml`
- Update `RayPPOTrainer` to support optional checkpoint deletion during
loading
- Modify `ActorRolloutRefWorker` and `CriticWorker` to pass checkpoint
removal flag

…gine#282)

Co-authored-by: zhangshulai <zhangshulai@bytedance.com>

The split placement example is outdated, I tried it and encountered some
errors. To address this, the following changes were made in this PR
1. Copied the content from `verl/trainer/config/ppo_trainer.yaml` to
`examples/split_placement/config/ppo_trainer_split.yaml`
2. Copied `RayPPOTrainer.fit` method into the `fit` func in
`examples/split_placement/split_monkey_patch.py` and modified it to get
the futures of `critic_output` and `actor_output`

…ring setup (volcengine#286)

…ne#266)

### **Enhancement: Support for `extra_info` in Reward Calculation**  

#### **Summary**  
This update enhances the reward computation process by introducing an
additional `extra_info` parameter. This allows users to pass in more
contextual information when calculating rewards, improving flexibility
for different datasets.

#### **Changes Made**  
- **Updated `_default_compute_score`** to accept an `extra_info`
argument:
  ```python
def _default_compute_score(data_source, solution_str, ground_truth,
extra_info):
  ```
- **Modified the reward manager (`naive.py`)** to pass `extra_info` from
`data_item.non_tensor_batch` to `compute_score`:
  ```python
  extra_info = data_item.non_tensor_batch['extra_info']
  score = self.compute_score(
      data_source=data_source,
      solution_str=sequences_str,
      ground_truth=ground_truth,
      extra_info=extra_info,
  )
  ```
  
#### **Why This Change?**  
- Some datasets require additional context beyond `data_source`,
`solution_str`, and `ground_truth` for accurate reward computation.
- The new `extra_info` field allows users to pass custom metadata,
ideally in dictionary form, as specified in the [official
documentation](https://verl.readthedocs.io/en/latest/preparation/prepare_data.html).
- This change maintains compatibility with existing dataset processing
scripts, as they already include the `extra_info` field.

#### **Impact**  
- **Improved flexibility**: Users can now pass additional contextual
information, making reward computation more adaptable to different
datasets.
- **Backward compatibility**: Since all example datasets already include
`extra_info`, this update should integrate seamlessly.

Let me know if any modifications are needed!

…olcengine#284)

- Fixed FSDP1 model offload
- With `actor_rollout_ref.actor.fsdp_config.param_offload=True \` and
`actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \ `. The GPU
memory utilization can increase to 0.9
- With actor, critic and reference offload all enabled, there will only
be one model copy at a time in the GPU memory. Therefore, we can further
increase the `micro_batch_size_per_gpu` or `max_token_per_gpu`

**Specifically:**
- During rollout, only rollout model and KVCache are in the GPU memory.
- During critic compute values, only the critic model will stay in the
GPU memory while its optimizer and other model states are in CPU main
memory
- During actor update, the actor model, optimizer are stored on GPU
while the reference model and critic model, critic optimizer are
offloaded to CPU.

1 fix wrong notes description.
2 fix wrong code path.

Signed-off-by: zhanluxianshen <zhanluxianshen@163.com>

Avoid CPU-to-device loading or offloading when the optimizer is not
initialized to prevent the incorrect creation of the optimizer.state

We need to specify the minimum  permission in the workflow.

A working Slurm example adapted from
https://docs.ray.io/en/latest/ray-core/starting-ray.html

)

Support Qwen2 Megatron backend

The code is primarily adapted from the llama folder, with modifications
to use QKV bias and remove the rope_scaling of RoPE in
`verl/models/qwen2/megatron/layers/parallel_attention.py`.

- Train using Qwen2-7B-Instruct with PPO, GSM8k score can reach 0.87 at
step 75.
- not support saver now

…#327)

…olcengine#318)

This PR adds Ray Serve to the requirements to enable support for
multi-node training. It addresses the issue described here:
volcengine#87 (comment)

Co-authored-by: Yu Feng <fengyufengyu@didiglobal.com>

Implement RLOO algorithm according to https://arxiv.org/abs/2402.14740

…cengine#343)

Tracking backend support vemlp wandb

---------

Co-authored-by: liudayuan.carrot <liudayuan.carrot@bytedance.com>

Validation datasets are sent to inference engines as a whole batch,
which will schedule the memory themselves.

- [x] Remove `val_batch_size` from examples
- [x] Set default values of `val_batch_size` in configs as `null` and
add DEPRECATED comments
- [x] Add deprecation warnings about `val_batch_size` in
`_validate_config`

fix the issue[volcengine#331](volcengine#331)

….48 (volcengine#357)

close volcengine#312 

Add support for ulysses sp for transformers >= 0.48

I've tested transformers 0.45.0, 0.46.0, 0.47.0, 0.48.0 and 0.49.0,
using sp=2 with the following script in my local env
```bash
#!/bin/bash

set -ex
VERSIONS=("4.45.0" "4.46.0" "4.47.0" "4.48.0" "4.49.0")

for version in "${VERSIONS[@]}"; do
    echo "Testing with Transformers version ${version}"
    echo "----------------------------------------"
    
    pip install "transformers==${version}"
    
    PYTHONPATH=./ torchrun --nproc_per_node=2 tests/model/test_transformers_ulysses.py
    
    echo "----------------------------------------"
    echo "Completed testing for version ${version}"
    echo ""
done
```

…engine#944)

### Changes 

Add gradient checkpointing (aka `activation recomputation`) config and
support from Megatron core
(https://github.com/NVIDIA/Megatron-LM/blob/b7ec711cf66cf500b98d8783f2c7f3c3a7d5ba31/megatron/core/transformer/transformer_config.py#L208-L233)
to make activation checkpointing more efficient for LLMs with 20B+
parameters.

```
 gradient_checkpointing_kwargs:
     activations_checkpoint_method: null
     activations_checkpoint_granularity: null
     activations_checkpoint_num_layers: null
```

### Test 
Tested on loading Qwen7b/32b of 16k input prompts and bypass the OOM
issues after adding gradient checkpointing.

### Next Step

Add one `ppo_trainer for megatron` doc to explain the config details in
https://verl.readthedocs.io/en/latest/examples/config.html

…volcengine#925)

As mentioned in volcengine#903, the
model_merger script has some problem when dealing with saved fsdp
checkpoint trained with `trainer.n_gpus_per_node=1`. The loaded `weight`
is of type `Tensor` instead of `DTensor`. This PR supported this
situation.

… time configurable & providing better error info (volcengine#947)

## Summary

As mentioned in volcengine#491, the `register_center` named actor could be `None`
after 2mins waiting time and crash the job for some verl users.

This might be due to (1) uncleaned ray resources from previous runs; or
(2) too short waiting time of 120s if the `named_actor` launching task
is delayed in the cluster.

This PR makes the `register_cetner` named actor waiting time
configurable and longer by default . This PR also provides better error
info to help users to self debug the issue.

## Related issues

volcengine#491

---------

Signed-off-by: Hongpeng Guo <hg5@illinois.edu>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…engine#958)

users using fsdp backend may no have megatron installed, directly
running this script will lead to an import error.

…ne#959)

Support to set warmup_style=='cosine'.

Fixing volcengine#950

…gine#976)

Co-authored-by: HL <linhaibin.eric@gmail.com>

Currently, `pg_clipfrac_lower` is always 0 by mistake.

Currently the model merger does not support HSDP (the `ddp` mesh dim is
not considered). This PR fixes this.

See
vllm-project/vllm@8b66470
In summary, now when using external launcher in vLLM, a Seed must be
set.

---------

Co-authored-by: hoshi-hiyouga <hiyouga@buaa.edu.cn>

This PR suggest a fix on a bug that when `_switch_chat_template()`
method is called.

According to

https://github.com/volcengine/verl/blob/main/verl/utils/dataset/rl_dataset.py#L222
`data.non_tensor_batch['raw_prompt'][i]` is already a list if
`data.return_raw_chat=True`.

Calling `.tolist()` again will result an error. Now we check if it is a
list before run this method.