SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for
  Open Base Models in the Wild
DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can
naturally emerge through a simple reinforcement learning (RL) framework with
rule-based rewards, where the training may directly start from the base
models-a paradigm referred to as zero RL training. Most recent efforts to
reproduce zero RL training have primarily focused on the Qwen2.5 model series,
which may not be representative as we find the base models already exhibit
strong instruction-following and self-reflection abilities. In this work, we
investigate zero RL training across 10 diverse base models, spanning different
families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B,
Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several
key design strategies-such as adjusting format reward and controlling query
difficulty-we achieve substantial improvements in both reasoning accuracy and
response length across most settings. However, by carefully monitoring the
training dynamics, we observe that different base models exhibit distinct
patterns during training. For instance, the increased response length does not
always correlate with the emergence of certain cognitive behaviors such as
verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for
the first time in small models not from the Qwen family. We share the key
designs that enable successful zero RL training, along with our findings and
practices. To facilitate further research, we open-source the code, models, and
analysis tools.
67e22ce1155ea10f2fdbe5d2
Weihao Zeng
claimed_verified
2025-03-25T08:20:18.367Z
62751082b43ccfeef483424f
/avatars/fec83e4478e7d1731ba6033328131852.svg
WeihaoZeng
user
AndrewZeng
67e22ce1155ea10f2fdbe5d3
Yuzhen Huang
claimed_verified
2025-03-25T08:20:13.781Z
6462def82a83863b97c0611e
/avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
Yuzhen Huang
user
yuzhen17
67e22ce1155ea10f2fdbe5d4
Qian Liu
claimed_verified
2025-03-25T08:20:15.927Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
67e22ce1155ea10f2fdbe5d5
Wei Liu
claimed_verified
2025-05-22T07:18:37.813Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
user
PeterV09
67e22ce1155ea10f2fdbe5d6
Keqing He
admin_assigned
2025-03-25T09:17:09.770Z
64bf71792915a87970c07446
/avatars/b24403f9fa699e0143e441b56528e6af.svg
Keqing He
user
HelicHe
67e22ce1155ea10f2fdbe5d7
Zejun Ma
67e22ce1155ea10f2fdbe5d8
Junxian He
admin_assigned
2025-03-25T09:16:49.208Z
615f34ec3f6d24d67c1b5c78
/avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
Junxian He
user
jxhe
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for
  Open Base Models in the Wild
DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can
naturally emerge through a simple reinforcement learning (RL) framework with
rule-based rewards, where the training may directly start from the base
models-a paradigm referred to as zero RL training. Most recent efforts to
reproduce zero RL training have primarily focused on the Qwen2.5 model series,
which may not be representative as we find the base models already exhibit
strong instruction-following and self-reflection abilities. In this work, we
investigate zero RL training across 10 diverse base models, spanning different
families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B,
Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several
key design strategies-such as adjusting format reward and controlling query
difficulty-we achieve substantial improvements in both reasoning accuracy and
response length across most settings. However, by carefully monitoring the
training dynamics, we observe that different base models exhibit distinct
patterns during training. For instance, the increased response length does not
always correlate with the emergence of certain cognitive behaviors such as
verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for
the first time in small models not from the Qwen family. We share the key
designs that enable successful zero RL training, along with our findings and
practices. To facilitate further research, we open-source the code, models, and
analysis tools.
Zero reinforcement learning training across diverse base models improves reasoning accuracy and response length through strategic design adjustments, revealing novel cognitive behaviors in smaller models.
reinforcement learning zero RL training long chain-of-thought reasoning rule-based rewards base models Qwen2.5 LLama3-8B Mistral-7B/24B DeepSeek-Math-7B Qwen2.5-math-7B format reward query difficulty reasoning accuracy response length cognitive behaviors "aha moment" open-source code analysis tools