Reasoning or Memorization? Unreliable Results of Reinforcement Learning
  Due to Data Contamination
The reasoning capabilities of large language models (LLMs) have been a
longstanding focus of research. Recent works have further enhanced these
capabilities using reinforcement learning (RL), with many new methods claiming
significant improvements with minimal or no external supervision. Surprisingly,
some studies even suggest that random or incorrect reward signals can enhance
reasoning performance. However, these breakthroughs are mostly reported on the
Qwen2.5 model family and evaluated on well-known benchmarks such as MATH-500,
AMC, and AIME, while failing to achieve similar gains on other models like
Llama, which warrants further investigation. Our analysis shows that although
Qwen2.5 achieves strong mathematical reasoning performance, its pretraining on
large-scale web corpora makes it vulnerable to data contamination in popular
benchmarks. As a result, results derived from these benchmarks may be
unreliable. To address this, we introduce a generator that produces fully
synthetic arithmetic problems of arbitrary length and difficulty, yielding a
clean dataset we call RandomCalculation. Using these leakage-free datasets, we
show that only accurate reward signals consistently improve performance, while
noisy or incorrect signals do not. We advocate for evaluating RL methods on
uncontaminated benchmarks and across diverse model families to ensure
trustworthy conclusions.
6875f107257d4f0435370613
Mingqi Wu
6875f107257d4f0435370614
Zhihao Zhang
6875f107257d4f0435370615
Qiaole Dong
6875f107257d4f0435370616
Zhiheng Xi
6875f107257d4f0435370617
Jun Zhao
6875f107257d4f0435370618
Senjie Jin
6875f107257d4f0435370619
Xiaoran Fan
6875f107257d4f043537061a
Yuhao Zhou
6875f107257d4f043537061b
Yanwei Fu
6875f107257d4f043537061c
Qin Liu
6875f107257d4f043537061d
Songyang Zhang
claimed_verified
2025-07-16T08:15:52.205Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
6875f107257d4f043537061e
Qi Zhang
Reasoning or Memorization? Unreliable Results of Reinforcement Learning
  Due to Data Contamination
The reasoning capabilities of large language models (LLMs) have been a
longstanding focus of research. Recent works have further enhanced these
capabilities using reinforcement learning (RL), with many new methods claiming
significant improvements with minimal or no external supervision. Surprisingly,
some studies even suggest that random or incorrect reward signals can enhance
reasoning performance. However, these breakthroughs are mostly reported on the
Qwen2.5 model family and evaluated on well-known benchmarks such as MATH-500,
AMC, and AIME, while failing to achieve similar gains on other models like
Llama, which warrants further investigation. Our analysis shows that although
Qwen2.5 achieves strong mathematical reasoning performance, its pretraining on
large-scale web corpora makes it vulnerable to data contamination in popular
benchmarks. As a result, results derived from these benchmarks may be
unreliable. To address this, we introduce a generator that produces fully
synthetic arithmetic problems of arbitrary length and difficulty, yielding a
clean dataset we call RandomCalculation. Using these leakage-free datasets, we
show that only accurate reward signals consistently improve performance, while
noisy or incorrect signals do not. We advocate for evaluating RL methods on
uncontaminated benchmarks and across diverse model families to ensure
trustworthy conclusions.
Research on enhancing reasoning capabilities in large language models using reinforcement learning reveals that accurate reward signals are crucial for performance improvement, and current benchmarks may be unreliable due to data contamination.
large language models reinforcement learning Qwen2.5 MATH-500 AMC AIME Llama pretraining web corpora data contamination synthetic arithmetic problems RandomCalculation leakage-free datasets