On Predictability of Reinforcement Learning Dynamics for Large Language
  Models
Recent advances in reasoning capabilities of large language models (LLMs) are
largely driven by reinforcement learning (RL), yet the underlying parameter
dynamics during RL training remain poorly understood. This work identifies two
fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1
Dominance, where the top singular subspace of the parameter update matrix
nearly fully determines reasoning improvements, recovering over 99\% of
performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace
evolves linearly throughout training, enabling accurate prediction from early
checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the
generalizability of these properties. More importantly, based on these
findings, we propose AlphaRL, a plug-in acceleration framework that
extrapolates the final parameter update using a short early training window,
achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning
performance without extra modules or hyperparameter tuning. This positions our
finding as a versatile and practical tool for large-scale RL, opening a path
toward principled, interpretable, and efficient training paradigm for LLMs.
68dddc7b6024653e8a3ed08a
Yuchen Cai
claimed_verified
2025-10-03T12:54:25.052Z
651e2a19242e10766e61a669
/avatars/e5351fb14997269d8d3b9539f6f27d9e.svg
caiyuchen
user
caiyuchen
68dddc7b6024653e8a3ed08b
Ding Cao
68dddc7b6024653e8a3ed08c
Xin Xu
claimed_verified
2025-10-02T03:07:53.608Z
64e2d169d2af12910d682130
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg
xuxin
user
xx18
68dddc7b6024653e8a3ed08d
Zijun Yao
68dddc7b6024653e8a3ed08e
Yuqing Huang
68dddc7b6024653e8a3ed08f
Zhenyu Tan
68dddc7b6024653e8a3ed090
Benyi Zhang
68dddc7b6024653e8a3ed091
Guiquan Liu
68dddc7b6024653e8a3ed092
Junfeng Fang
On Predictability of Reinforcement Learning Dynamics for Large Language
  Models
Recent advances in reasoning capabilities of large language models (LLMs) are
largely driven by reinforcement learning (RL), yet the underlying parameter
dynamics during RL training remain poorly understood. This work identifies two
fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1
Dominance, where the top singular subspace of the parameter update matrix
nearly fully determines reasoning improvements, recovering over 99\% of
performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace
evolves linearly throughout training, enabling accurate prediction from early
checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the
generalizability of these properties. More importantly, based on these
findings, we propose AlphaRL, a plug-in acceleration framework that
extrapolates the final parameter update using a short early training window,
achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning
performance without extra modules or hyperparameter tuning. This positions our
finding as a versatile and practical tool for large-scale RL, opening a path
toward principled, interpretable, and efficient training paradigm for LLMs.
Two fundamental properties of reinforcement learning-induced parameter updates in large language models are identified, leading to a plug-in acceleration framework that significantly speeds up training without sacrificing performance.
reinforcement learning parameter updates Rank-1 Dominance singular subspace Rank-1 Linear Dynamics AlphaRL large language models