• w/o tool + only first user round have image list (Nan Jiang @ Amazon is working on it ETA 5.20)
  • request add image data support, data structure(message schema + async rollout requests)
  • support rollout in _async_rollout_a_request & _preprocess_prompt_to_async_rollout_requests https://github.com/volcengine/verl/blob/main/verl/workers/rollout/sglang_rollout/async_sglang_rollout.py#L430
• w tool + only first user round have image list (Nan Jiang @ Amazon & Congkai Xie @ RealLM Lab) [sglang] feat: add multimodal input to multiturn async rollout volcengine/verl#2014
  • calc_geo3k_reward, ground truth is also text.
  • RL example script & wandb log
• any round have image list supported
  • request add image data support,
  • support rollout in _async_rollout_a_request
• Advance multi-modal tool (not part of update 2 Multi-turn rollout Update #2 #132 )
  • tool_call arg/resp need image ? what's the scenerio

Xinyuan Tong & Ji Li is also helping on this.

• Notion Knowledge Base
• Base branch: https://github.com/SwordFaith/verl/tree/refactor/merge_sgl_rollouts_and_bump_to_0.4.6.post4