SIRI: Scaling Iterative Reinforcement Learning with Interleaved
  Compression
We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved
Compression, a simple yet effective RL approach for Large Reasoning Models
(LRMs) that enables more efficient and accurate reasoning. Existing studies
have observed repetitive thinking patterns in LRMs, and attempts to reduce them
often come at the cost of performance. In this paper, we show that this
trade-off can be overcome through a training regime that iteratively alternates
between compressing and expanding the reasoning budget, by dynamically
adjusting the maximum rollout length during training. The compression phase
cuts the rollout length, forcing the model to make precise and valuable
decisions within a limited context, which effectively reduces redundant tokens
and increases reasoning density. The expansion phase then relaxes the length
limit, providing space for the model to explore and plan in long-horizon
settings. Remarkably, we find that after each compression-expansion cycle, the
model's performance improves even as its output length decreases, steadily
pushing it closer to the Pareto frontier in the performance-efficiency
trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves
performance on AIME24 by 43.2% while reducing token usage by 46.9% after three
iterations, and SIRI-high achieves the highest accuracy compared to all other
methods (Figure 1). Our findings shed light on the potential of periodically
oscillating the LRM's output truncation length during training to dynamically
balance exploration and efficiency in reasoning, converging towards an optimal
"sweet spot" between the two. Our models are publicly available.
68db421ad2bf1f4b15ec72fd
Haoming Wen
68db421ad2bf1f4b15ec72fe
Yushi Bai
68db421ad2bf1f4b15ec72ff
Juanzi Li
68db421ad2bf1f4b15ec7300
Jie Tang
SIRI: Scaling Iterative Reinforcement Learning with Interleaved
  Compression
We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved
Compression, a simple yet effective RL approach for Large Reasoning Models
(LRMs) that enables more efficient and accurate reasoning. Existing studies
have observed repetitive thinking patterns in LRMs, and attempts to reduce them
often come at the cost of performance. In this paper, we show that this
trade-off can be overcome through a training regime that iteratively alternates
between compressing and expanding the reasoning budget, by dynamically
adjusting the maximum rollout length during training. The compression phase
cuts the rollout length, forcing the model to make precise and valuable
decisions within a limited context, which effectively reduces redundant tokens
and increases reasoning density. The expansion phase then relaxes the length
limit, providing space for the model to explore and plan in long-horizon
settings. Remarkably, we find that after each compression-expansion cycle, the
model's performance improves even as its output length decreases, steadily
pushing it closer to the Pareto frontier in the performance-efficiency
trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves
performance on AIME24 by 43.2% while reducing token usage by 46.9% after three
iterations, and SIRI-high achieves the highest accuracy compared to all other
methods (Figure 1). Our findings shed light on the potential of periodically
oscillating the LRM's output truncation length during training to dynamically
balance exploration and efficiency in reasoning, converging towards an optimal
"sweet spot" between the two. Our models are publicly available.
SIRI, a reinforcement learning approach with interleaved compression and expansion, enhances the efficiency and accuracy of large reasoning models by dynamically adjusting the reasoning budget.
Scaling Iterative Reinforcement Learning Interleaved Compression Large Reasoning Models RL rollout length reasoning density DeepSeek-R1-Distill-Qwen-1.5B AIME24 Pareto frontier performance-efficiency trade-off