ROLL: Reinforcement Learning Optimization for Large-Scale Learning

🚀 An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models 🚀

ROLL is an efficient and user-friendly RL library designed for Large Language Models (LLMs) utilizing Large Scale GPU resources. It significantly enhances LLM performance in key areas such as human preference alignment, complex reasoning, and multi-turn agentic interaction scenarios.

Leveraging a multi-role distributed architecture with Ray for flexible resource allocation and heterogeneous task scheduling, ROLL integrates cutting-edge technologies like Megatron-Core, SGLang and vLLM to accelerate model training and inference.

Documents

Quick Start based on alicloud
Installation
Config guide

RLVR Pipeline
Agentic RL Pipeline

• Multi-task RL Training (RLVR): Covers mathematics, coding, general reasoning, open-ended Q&A, instruction following, etc.
  • Flexible domain_batch_size distribution control.
  • Sample-level asynchronous parallel Rollout, asynchronous reward calculation, and dynamic sampling.
  • Asynchronous training under implementation.
• Agentic RL: Multi-turn interaction capabilities for games, multi-turn dialogues, tool use, etc.
  • Environment-level asynchronous parallel rollout.
  • Supports asynchronous training.
  • Multi-turn interaction rollout supports local debugging, improving multi-turn interaction business development efficiency.
  • Supports TrajectoryWise (StartPO) and StepWise (GiGPO) training paradigms.
• Algorithm-Friendly: Provides flexible and rich RL strategy configurations by default.
  • Over 20 rich reinforcement learning strategy options, such as reward normalization, reward clipping, various advantage estimation methods, etc.
  • Out-of-the-box support for reinforcement learning algorithms, such as PPO, GRPO, Reinforce++, TOPR, RAFT++, GSPO, etc.
• Rich Training and Inference Engine: Ray-based multi-role distributed architecture; Strategy abstraction unifies various backends, enabling easy operation from single machines to thousands-of-GPU clusters.
  • Inference/Generation supports vLLM, SGLang.
  • Training supports DeepSpeed (ZeRO), Megatron-LM 5D parallelism (mcore-adapter, dp/tp/pp/cp/ep), FSDP under implementation.
  • Extreme offload/reload capabilities.
  • Supports LoRA training.
  • Supports FP8 rollout (FP8 inference for LLM as judge, FP8 rollout with BF16 training under development).
• AutoDeviceMapping: Supports custom device mapping for different roles, flexibly managing colocated and disaggregated deployments.
• Observability: Integrated with SwanLab / WandB / TensorBoard, tracking of performance for each domain and reward type.
• Rich Post-training Technical Support:
  • Agentic RL LLM & VLM
  • RLVR LLM & VLM
  • Distill Pipeline LLM & VLM
  • DPO Pipeline
  • SFT Pipeline under development

We are continuously working to expand ROLL's capabilities:

• ⏱️ Async RLVR pipeline: For even more efficient and streamlined asynchronous operations.
• ⚙️ FSDP2: Integrating the latest Fully Sharded Data Parallel techniques.
• 🔍 Support DeepseekV3: Adding compatibility for the newest Deepseek models.

• RecGPT: a next-generation, LLM-driven framework that places user intent at the core of recommender systems, fostering a more sustainable and mutually beneficial ecosystem.

ROLL is inspired by the design of OpenRLHF, VeRL, Nemo-Aligner, and RAGEN. The project is developed by Alibaba TAOBAO & TMALL Group and Alibaba Group. The code is distributed under the Apache License (Version 2.0). This product contains various third-party components under other open-source licenses. See the NOTICE file for more information.

The following repositories have been used in ROLL, either in their close-to-original form or as an inspiration:

• NVIDIA/Megatron-LM
• microsoft/DeepSpeed
• sgl-project/sglang
• vllm-project/vllm

If you use ROLL in your research or project, please consider citing us:

@article{wang2025reinforcement,
  title={Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library},
  author={Wang, Weixun and Xiong, Shaopan and Chen, Gengru and Gao, Wei and Guo, Sheng and He, Yancheng and Huang, Ju and Liu, Jiaheng and Li, Zhendong and Li, Xiaoyang and others},
  journal={arXiv preprint arXiv:2506.06122},
  year={2025}
}

ROLL is a project jointly developed by Taotian Future Life Lab and Aicheng Technology, with a strong emphasis on pioneering the future of Reinforcement Learning (RL). Our mission is to explore and shape innovative forms of future living powered by advanced RL technologies. If you are passionate about the future of RL and want to be part of its evolution, we warmly welcome you to join us! Learn more about the ROLL Team through our official channels below👇

We are HIRING!

• Post Training Infra 研发工程师 JD link
• 大模型训练专家：
  • （社招）JD link
  • （校招）JD link
• Infra 研究型实习生 JD link