Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's
  Own Signals
Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models
(LLMs) suffer from sample inefficiency due to the redundant exposure of
identical queries under uniform data sampling. While previous work has explored
curriculum learning via heuristic difficulty metrics, these strategies exhibit
limitations by neglecting the intrinsic learning signals generated by the model
itself, thus leading to suboptimal training regimes. In this paper, we identify
a model-inherent signal termed angle concentration that effectively reflects an
LLM's capacity to learn from specific data. We theoretically and empirically
demonstrate a correlation between the angular distribution of token hidden
state vectors and the resulting gradient, revealing a learning preference for
data exhibiting higher angle concentration. Inspired by this finding, we
propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By
leveraging the model's intrinsic angle concentration signal, GAIN-RL
dynamically selects training data in each epoch, ensuring consistently
impactful gradient updates and thus significantly enhancing overall training
efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x
acceleration in training efficiency across diverse mathematical and coding
tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient
sampling yields data-efficient training, achieving better performance with half
the original data compared to vanilla GRPO with full training data. Code is
realsed at https://github.com/wangqinsi1/GAINRL/tree/main.
6840c71b2361fd58c84588b1
Qinsi Wang
claimed_verified
2025-06-23T08:31:43.602Z
66128e8b7e0e7a64652dbbdf
/avatars/b72ea5b14b55ff3af920c06b69a60b3f.svg
Wang
user
Qinsi1
6840c71b2361fd58c84588b2
Jinghan Ke
6840c71b2361fd58c84588b3
Hancheng Ye
admin_assigned
2025-06-05T07:33:36.043Z
6597f70d260bcd5fb5529613
/avatars/8fd7a210f3cb0de1229b49a3cd1ad8a0.svg
Hancheng Ye
user
HankYe
6840c71b2361fd58c84588b4
Yueqian Lin
admin_assigned
2025-06-05T07:33:42.287Z
64b5198c25882acb62fb77ef
https://cdn-avatars.huggingface.co/v1/production/uploads/64b5198c25882acb62fb77ef/HX9pfMEPQlfjvSAgSLplY.png
Yueqian Lin
user
linyueqian
6840c71b2361fd58c84588b5
Yuzhe Fu
admin_assigned
2025-06-05T07:33:49.125Z
66d9052a60730595a867e58f
/avatars/3a84156253153150a573d374b731a286.svg
Yuzhe Fu
user
YuzheFu
6840c71b2361fd58c84588b6
Jianyi Zhang
admin_assigned
2025-06-05T07:33:56.480Z
676d402ad886f8125af92360
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Pby6l-_ymnvPZSAAWWeS_.png
jianyi.zhang
user
jianyi0115
6840c71b2361fd58c84588b7
Kurt Keutzer
admin_assigned
2025-06-05T07:34:01.864Z
6251bf4b183aa4266924ad91
https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
Kurt Keutzer
user
kurtkeutzer
6840c71b2361fd58c84588b8
Chenfeng Xu
admin_assigned
2025-06-05T07:34:10.768Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
6840c71b2361fd58c84588b9
Yiran Chen
Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's
  Own Signals
Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models
(LLMs) suffer from sample inefficiency due to the redundant exposure of
identical queries under uniform data sampling. While previous work has explored
curriculum learning via heuristic difficulty metrics, these strategies exhibit
limitations by neglecting the intrinsic learning signals generated by the model
itself, thus leading to suboptimal training regimes. In this paper, we identify
a model-inherent signal termed angle concentration that effectively reflects an
LLM's capacity to learn from specific data. We theoretically and empirically
demonstrate a correlation between the angular distribution of token hidden
state vectors and the resulting gradient, revealing a learning preference for
data exhibiting higher angle concentration. Inspired by this finding, we
propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By
leveraging the model's intrinsic angle concentration signal, GAIN-RL
dynamically selects training data in each epoch, ensuring consistently
impactful gradient updates and thus significantly enhancing overall training
efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x
acceleration in training efficiency across diverse mathematical and coding
tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient
sampling yields data-efficient training, achieving better performance with half
the original data compared to vanilla GRPO with full training data. Code is
realsed at https://github.com/wangqinsi1/GAINRL/tree/main.
GAIN-RL leverages angle concentration signals to improve training efficiency and data efficiency in Reinforcement Fine-tuning of Large Language Models.
Reinforcement Fine-tuning Large Language Models sample inefficiency curriculum learning angle concentration gradient updates token hidden state vectors GAIN-RL Gradient-driven Angle-Informed Navigated RL