AceReason-Nemotron: Advancing Math and Code Reasoning through
  Reinforcement Learning
Despite recent progress in large-scale reinforcement learning (RL) for
reasoning, the training recipe for building high-performing reasoning models
remains elusive. Key implementation details of frontier models, such as
DeepSeek-R1, including data curation strategies and RL training recipe, are
often omitted. Moreover, recent research indicates distillation remains more
effective than RL for smaller models. In this work, we demonstrate that
large-scale RL can significantly enhance the reasoning capabilities of strong,
small- and mid-sized models, achieving results that surpass those of
state-of-the-art distillation-based models. We systematically study the RL
training process through extensive ablations and propose a simple yet effective
approach: first training on math-only prompts, then on code-only prompts.
Notably, we find that math-only RL not only significantly enhances the
performance of strong distilled models on math benchmarks (e.g., +14.6% /
+17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks
(e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition,
extended code-only RL iterations further improve performance on code benchmarks
with minimal or no degradation in math results. We develop a robust data
curation pipeline to collect challenging prompts with high-quality, verifiable
answers and test cases to enable verification-based RL across both domains.
Finally, we identify key experimental insights, including curriculum learning
with progressively increasing response lengths and the stabilizing effect of
on-policy parameter updates. We find that RL not only elicits the foundational
reasoning capabilities acquired during pretraining and supervised fine-tuning
(e.g., distillation), but also pushes the limits of the model's reasoning
ability, enabling it to solve problems that were previously unsolvable.
682fe6644640a9db4d1f31d9
Yang Chen
claimed_verified
2025-05-26T08:16:02.553Z
62bc9d90e81dfd65cced9316
/avatars/05df14cd1fdbc7d6a80d2960a05a94f0.svg
Yang Chen
user
ychenNLP
682fe6644640a9db4d1f31da
Zhuolin Yang
extracted_confirmed
2025-06-17T17:45:42.828Z
67d75b0117c2acac528f47b6
/avatars/619aacd1a619aab64de3499ac3ee2229.svg
Zhuolin Yang
user
zhuoliny
682fe6644640a9db4d1f31db
Zihan Liu
claimed_verified
2025-06-17T07:26:23.907Z
65f33b1c9f7970ccc0234cbf
/avatars/99fbab303912e3674663251c04279907.svg
Zihan Liu
user
zihanliu
682fe6644640a9db4d1f31dc
Chankyu Lee
682fe6644640a9db4d1f31dd
Peng Xu
682fe6644640a9db4d1f31de
Mohammad Shoeybi
682fe6644640a9db4d1f31df
Bryan Catanzaro
682fe6644640a9db4d1f31e0
Wei Ping
extracted_pending
2025-05-23T03:07:17.257Z
663ee43bfeeb49803537da98
/avatars/17c3e9c435cc36fb04b4589e6176a243.svg
Wei Ping
user
wping
AceReason-Nemotron: Advancing Math and Code Reasoning through
  Reinforcement Learning
Despite recent progress in large-scale reinforcement learning (RL) for
reasoning, the training recipe for building high-performing reasoning models
remains elusive. Key implementation details of frontier models, such as
DeepSeek-R1, including data curation strategies and RL training recipe, are
often omitted. Moreover, recent research indicates distillation remains more
effective than RL for smaller models. In this work, we demonstrate that
large-scale RL can significantly enhance the reasoning capabilities of strong,
small- and mid-sized models, achieving results that surpass those of
state-of-the-art distillation-based models. We systematically study the RL
training process through extensive ablations and propose a simple yet effective
approach: first training on math-only prompts, then on code-only prompts.
Notably, we find that math-only RL not only significantly enhances the
performance of strong distilled models on math benchmarks (e.g., +14.6% /
+17.2% on AIME 2025 for the 7B / 14B models), but also code reasoning tasks
(e.g., +6.8% / +5.8% on LiveCodeBench for the 7B / 14B models). In addition,
extended code-only RL iterations further improve performance on code benchmarks
with minimal or no degradation in math results. We develop a robust data
curation pipeline to collect challenging prompts with high-quality, verifiable
answers and test cases to enable verification-based RL across both domains.
Finally, we identify key experimental insights, including curriculum learning
with progressively increasing response lengths and the stabilizing effect of
on-policy parameter updates. We find that RL not only elicits the foundational
reasoning capabilities acquired during pretraining and supervised fine-tuning
(e.g., distillation), but also pushes the limits of the model's reasoning
ability, enabling it to solve problems that were previously unsolvable.
Large-scale reinforcement learning enhances reasoning capabilities in small and mid-sized models more effectively than distillation, achieving superior results in both math and code benchmarks.
reinforcement learning RL DeepSeek-R1 data curation distillation math-only prompts code-only prompts AIME 2025 LiveCodeBench curriculum learning on-policy parameter updates