Welcome to the Era of Experience. --David Silver, Richard S. Sutton
-
[2025-08-17] We bump into verl v0.4.1, and now you can enable TTRL by simply setting
+ttrl.enable=True! -
[2025-05-23] We update both the paper and the code, with the implementation based on the verl.
-
[2025-04-24] We release the code and experimental logs. Check it out: Getting Started.
-
[2025-04-23] We present TTRL (Test-Time Reinforcement Learning), an open-source solution for online RL on data without ground-truth labels, especially test data.
We investigate Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training.
Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 211% on AIME 2024 with only unlabeled test data.
Furthermore, although TTRL is only supervised by the maj@n metric, TTRL has demonstrated performance to consistently surpass this upper limit of the initial model, and approach the performance of models trained directly on test data with ground-truth labels.
git clone https://github.com/PRIME-RL/TTRL.git
cd TTRL/verl
conda create -n ttrl python==3.10
conda activate ttrl
bash scripts/install_ttrl_deps.sh
pip install -e .You can reproduce the results on AIME 2024 with the following commands:
bash examples/ttrl/Qwen2.5/aime.shNote
- You can use the script verl/data/preprocess.py to convert data from the
JSONformat to theParquetformat for training with verl. - We provide scripts in the verl/examples/ttrl directory for running TTRL on multiple models across various benchmarks.
- For further details regarding the code, please refer to the verl documentation.
We additionally conducted three independent runs using the preview version of our code. Two of the runs achieved a pass@1 (greedy) of 43.3, while one run reached 46.7. Please refer to the Weights & Biases logs.
All experiments were conducted on 8 x NVIDIA A100 80GB GPUs.
Pseudo-Code
The implementation of TTRL can be achieved rapidly by simply modifying the reward function. Please refer to the following code snippet for details:
-
Kaiyan Zhang: zhang-ky22@mails.tsinghua.edu.cn
-
Ning Ding: dingning@mail.tsinghua.edu.cn
If you find TTRL helpful, please cite us.
@article{zuo2025ttrl,
title={Ttrl: Test-time reinforcement learning},
author={Zuo, Yuxin and Zhang, Kaiyan and Sheng, Li and Qu, Shang and Cui, Ganqu and Zhu, Xuekai and Li, Haozhan and Zhang, Yuchen and Long, Xinwei and Hua, Ermo and others},
journal={arXiv preprint arXiv:2504.16084},
year={2025}
}