Reinforcement Mid-Training
The development of state-of-the-art large language models is commonly
understood as a two-stage process involving pre-training and post-training. We
point out the need for an additional intermediate stage called reinforcement
mid-training with potential for strong performance gains. In this paper, we
formally define the problem and identify three key challenges: (1) inefficient
training due to excessive reasoning steps, (2) disregard of the imbalanced
token entropy distribution, and (3) underutilization of token information. To
address these challenges, we propose RMT, a framework for efficient, adaptive,
and unified reinforcement mid-training with various innovative components. In
particular, we first introduce a dynamic token budget mechanism that constrains
unnecessary reasoning steps and mitigates model overthinking. Next, we design a
curriculum-based adaptive sampling method that fosters a progressive learning
trajectory from easy to hard tokens. Finally, we present a dual training
strategy that combines reinforcement learning with next-token prediction,
ensuring targeted learning on key tokens and full exploitation of all token
information. Extensive experiments demonstrate the superiority of RMT over
state-of-the-art methods, achieving up to +64.91% performance improvement with
only 21% of the reasoning length in language modeling. We also show that
checkpoints obtained after reinforcement mid-training can benefit the
subsequent post-training, yielding up to +18.76% improvement in the
mathematical domain.
68e722347ae125f9582e69bd
Yijun Tian
68e722347ae125f9582e69be
Shaoyu Chen
claimed_verified
2025-10-10T09:32:15.984Z
65e57e10d26b426e3ead1de4
/avatars/0a9dc65b92761b918756d14f1ece0e06.svg
Shawn_csy
user
Shawn21csy
68e722347ae125f9582e69bf
Zhichao Xu
68e722347ae125f9582e69c0
Yawei Wang
68e722347ae125f9582e69c1
Jinhe Bi
68e722347ae125f9582e69c2
Peng Han
68e722347ae125f9582e69c3
Wei Wang
Reinforcement Mid-Training
The development of state-of-the-art large language models is commonly
understood as a two-stage process involving pre-training and post-training. We
point out the need for an additional intermediate stage called reinforcement
mid-training with potential for strong performance gains. In this paper, we
formally define the problem and identify three key challenges: (1) inefficient
training due to excessive reasoning steps, (2) disregard of the imbalanced
token entropy distribution, and (3) underutilization of token information. To
address these challenges, we propose RMT, a framework for efficient, adaptive,
and unified reinforcement mid-training with various innovative components. In
particular, we first introduce a dynamic token budget mechanism that constrains
unnecessary reasoning steps and mitigates model overthinking. Next, we design a
curriculum-based adaptive sampling method that fosters a progressive learning
trajectory from easy to hard tokens. Finally, we present a dual training
strategy that combines reinforcement learning with next-token prediction,
ensuring targeted learning on key tokens and full exploitation of all token
information. Extensive experiments demonstrate the superiority of RMT over
state-of-the-art methods, achieving up to +64.91% performance improvement with
only 21% of the reasoning length in language modeling. We also show that
checkpoints obtained after reinforcement mid-training can benefit the
subsequent post-training, yielding up to +18.76% improvement in the
mathematical domain.
A reinforcement mid-training framework (RMT) improves large language models by addressing inefficiencies and underutilization of token information, leading to significant performance gains.
reinforcement mid-training dynamic token budget mechanism curriculum-based adaptive sampling dual training strategy reinforcement learning next-token prediction language modeling mathematical domain