Current Status Tracker

training setting support:

rollout feature support:

tool support:

algorithm support

Road Map

• Update 1 Multi-turn rollout Update #1 #113

  • Add request-level async rollout with tool interaction
  • Support FSDP multi-turn and train correctly, wandb log: https://wandb.ai/swordfaith/gsm8k_async_rl/runs/ta7jhvgq?nw=nwuserswordfaith

• Update 2 (ETA June early) Multi-turn rollout Update #2 #132

  • Add real world tools support (at least Search & Code Interpreter)
    • search & search-r1 reproduce [sglang] Feat: Search Tool Invocation in Multi-Turn RL Training volcengine/verl#1682
    • code sandbox, sandbox fusion supported feat: sandbox fusion for multi-turn volcengine/verl#1525, working on retool reproduction (sft: [training_utils] Add qwen3 multi-turn sft support volcengine/verl#1889 )
  • Add Qwen3 training example
  • Add Megatron support
  • Add multi-node support trouble shooting doc
  • Add FSDP2 support
  • Add VLM support [sglang] feat: add multimodal input to multiturn async rollout volcengine/verl#2014

• Update 3 (In discussion)

  • Add Ray Agentic Trainer, The current async request-level approach avoids batch tool calls and generation barriers. A better approach is to use an agentic trainer: fetch → rollout → reward calculation → filter micro-batch in a seamless async loop.
  • Add partial rollout and replay buffer in agentic trainer with partial rollout setting
  • Add more tools and examples
  • Add GenRM support

• Update 4

  • Introduce user interaction simulation [sglang] feat: Support async multi-turn rollout with simulation feedback in sglang volcengine/verl#1630
  • Introduce co-training with judge agent
  • Introduce MCP tools or other wrap solution for current implemented tools

Refactor To-dos

• Combine the current sglang and sglang_async rollouts seamlessly so that users remain unaware of the changes. (ETA: 5.19，dev phase done, waiting to merge prs) [sglang] Fix megatron support in sglang and add sglang_async support & CI tasks volcengine/verl#1602 [sglang] refactor: Unify async rollout under SGLangRollout, and support sglang==0.4.6.post5 volcengine/verl#1717
• An asynchronous tool from the SPMD resource pool designed to isolate the global Ray actor, enhancing the management of the global rate limit.
• Keep using chat template in async rollout req, to avoid mismatch risk. Qwen chat template community fix before refactor Support multi-turn rollout with Qwen chat template volcengine/verl#1593 [sglang] feat: Efficient and model-agnostic multi-turn messages tokenization and masking volcengine/verl#1668
• Decoupling max_response_length, max_model_len, max_new_tokens
• Support parse_args api in tool, to custom tool parse parameters logic.
• Decoupling loss_mask and response_mask, and redesign this part of logic.

Trouble Shooting Tacker