If our project helps you, please give us a star ⭐ and cite our paper!
- [2026.05] 🎉 Our paper is accepted to ACL 2026 Main Conference!
- 08.26.2025, Code is released.
- Release the model checkpoints
- Release the inference and evaluation code
- Release the training data
- Release the training code
In this project, we
- Investigate Guided GRPO, and provide comprehensive study of various guidance configurations.
- Introduce G2RPO-A, an adaptive algorithm that automatically adjusts guidance length in response to the evolving training state.
Guided GRPO can improve reasoning and code generation by injecting external guidance into policy optimization. However, the effectiveness of guidance strongly depends on how much guidance is provided at different stages of training.
G2RPO-A addresses this challenge with adaptive guidance. It automatically adjusts the guidance length according to the evolving training state, reducing manual tuning while preserving the benefits of guided optimization.
G²RPO-A builds on Open-R1 with a customized TRL GRPO trainer. Our recorded environment uses TRL 0.15.2 and Transformers 4.52.3. These versions are particularly relevant when reproducing the Qwen3 experiments.
The original environment setup uses Python 3.11 and CUDA 12.4. The paper reports training on 8 NVIDIA A100 GPUs, and the provided recipes use BF16, DeepSpeed, vLLM, and FlashAttention 2.
The following installed versions are recorded in the package inventory archived with the supplementary materials (opern1.txt):
| Component | Version |
|---|---|
| TRL | 0.15.2 |
| Transformers | 4.52.3 |
| PyTorch | 2.5.1 |
| vLLM | 0.7.2 |
| Accelerate | 1.4.0 |
| DeepSpeed | 0.15.4 |
| FlashAttention | 2.7.4.post1 |
| Datasets | 3.6.0 |
| Math-Verify | 0.5.2 |
| Liger Kernel | 0.6.0 |
| SwanLab | 0.6.3 |
The recorded training environment uses trl==0.15.2. The older Open-R1 dependency declaration contains a reference to Git commit 69ad852e5654a77f1695eb4c608906fe0c7e8624 (0.16.0.dev0); use the installed-package record above when reconstructing our environment.
G²RPO-A also requires our custom GRPO trainer, which implements adaptive guidance. This file defines GRPOTrainer and uses TRL-internal relative imports; it is intended to replace trl/trainer/grpo_trainer.py in the TRL installation. Installing the standard TRL package alone does not enable G²RPO-A. The rule-based trainer provides the alternative guidance-decay policy.
After preparing the dependencies, install the recorded TRL version explicitly:
python -m pip install --no-deps "trl==0.15.2"--no-deps preserves the separately configured dependencies; this command only installs TRL.
The original Open-R1 configuration pins Transformers 4.49.0, which does not support Qwen3 out of the box. Qwen3 support was introduced in Transformers 4.51.0, and the official Qwen3 model card states that older versions raise KeyError: 'qwen3'.
For the recorded project environment, use transformers==4.52.3. Here, 4.51.0 is the minimum version for recognizing Qwen3, while 4.52.3 is the version recorded in our environment. Support for the model architecture alone does not establish compatibility with the entire training stack.
The current setup.py still contains older Transformers and Liger Kernel pins, and its TRL dependency is commented out. Those declarations need to be reconciled with the versions above when recreating the environment; installing the repository alone does not reproduce the recorded environment. A fresh installation and GPU training run have not yet been validated for the current repository layout.
The following model checkpoint is publicly available on Hugging Face.
| Model | Backbone | Task | Download |
|---|---|---|---|
| Qwen3-1.7B-Math | Qwen3-1.7B | Mathematical reasoning | Hugging Face |
- Math-Curriculum-1K: 1,000 mathematical reasoning training examples with verified reasoning trajectories, organized into five source-based curriculum tiers from easy to hard. See the dataset card for construction details, field descriptions, and loading instructions.
- Code-CoT: 1,086 Python programming problems with reasoning trajectories generated by QwQ-32B-Preview and executable test cases, curated from Open-R1 Verifiable Coding Problems Python for guided reinforcement learning. See the dataset card for construction details, source composition, and loading instructions.
G2RPO-A training
ACCELERATE_LOG_LEVEL=info \
accelerate launch \
--config_file recipes/accelerate_configs/zero2.yaml \
--num_processes=7 \
src/open_r1/grpo_code_adagui.py \
--config recipes/Qwen3-1.7B/grpo/qwen38code.yaml
Use the following EvalScope command as an evaluation example. Replace /path/to/checkpoint with your model checkpoint directory.
evalscope eval \
--model /path/to/checkpoint \
--datasets math_500 \
--dataset-args '{"math_500": {"few_shot_num": 0}}' \
--eval-batch-size 128 \
--generation-config '{"max_tokens":25000,"temperature":0.6,"top_p":0.95,"top_k":20,"n":1}' \
--timeout 60000 \
--stream \
--limit 102This example uses zero-shot evaluation and limits the run to 102 examples. Remove --limit 102 to evaluate the full MATH-500 dataset. The generation settings use a maximum of 25,000 tokens, temperature 0.6, top-p 0.95, and top-k 20.
We thank the developers and contributors of the following open-source projects and datasets:
- Training framework: Open-R1, which provides the foundation for our implementation.
- Training datasets: OpenR1-Math-220k, Verifiable Coding Problems Python, and s1K-1.1, which support our mathematical reasoning, code generation, and guidance experiments.
- Evaluation benchmarks: MATH-500, Minerva-Math, GPQA, AIME24, AIME25, HumanEval, and LiveCodeBench.
- Evaluation tools: EvalScope from the ModelScope community, LightEval, and LM Evaluation Harness.
- Answer verification and code execution: Math-Verify and E2B.
If you find this project helpful, please consider citing our work:
@inproceedings{guo2026g2rpoa,
title={G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance},
author={Guo, Yongxin and Deng, Wenbo and Cheng, Zhenglin and Tang, Xiaoying},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)},
year={2026}
}
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/zero2.yaml --num_processes=7 src/open_r1/grpo_code_adagui.py --config recipes/Qwen3-1.7B/grpo/qwen38code.yaml
