Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement
  Learning
Recent studies on reasoning models explore the meta-awareness of language
models, the ability to know how to think by itself. We argue that large
reasoning models lack this meta-awareness property by proving severe
misalignment between true rollouts and predicted meta information. We posit
that aligning meta-prediction with true rollouts will lead to significant
performance gains. To verify this hypothesis, we design a training pipeline
that boosts Meta-Awareness via Self-Alignment (MASA), and prove that enhanced
meta-awareness directly translates to improved accuracy. Unlike existing
meta-cognitive reasoning models, our method does not require external training
sources but leverages self-generated signals to train meta-awareness. Moreover,
our method enables efficient training by i) filtering out zero-variance prompts
that are either trivial or unsolvable and ii) cutting off lengthy rollouts when
they are unlikely to lead to correct answers. The results are inspiring: our
strategy yields significant improvements in both accuracy and training
efficiency on in-domain tasks and shows strong generalization to out-of-domain
benchmarks. More specifically, our method can speed up GRPO training by over
1.28x to reach the same performance, and achieve a 19.3% gain in accuracy on
AIME25, and a 6.2 % average gain over six mathematics benchmarks. Training with
meta-cognitive guidance enhances out-of-domain generalization, giving a 3.87 %
boost on GPQA-Diamond and a 2.08 % overall accuracy gain across 13 benchmarks
spanning logical, scientific, and coding domains.
68e88d9c95e8e6771df38b0b
Yoonjeon Kim
68e88d9c95e8e6771df38b0c
Doohyuk Jang
claimed_verified
2025-10-10T08:54:37.386Z
62845957b410bd779033759c
/avatars/4feef73c06f2f7de6abf7a4789ac13f9.svg
Doohyuk Jang
user
jadohu
68e88d9c95e8e6771df38b0d
Eunho Yang
Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement
  Learning
Recent studies on reasoning models explore the meta-awareness of language
models, the ability to know how to think by itself. We argue that large
reasoning models lack this meta-awareness property by proving severe
misalignment between true rollouts and predicted meta information. We posit
that aligning meta-prediction with true rollouts will lead to significant
performance gains. To verify this hypothesis, we design a training pipeline
that boosts Meta-Awareness via Self-Alignment (MASA), and prove that enhanced
meta-awareness directly translates to improved accuracy. Unlike existing
meta-cognitive reasoning models, our method does not require external training
sources but leverages self-generated signals to train meta-awareness. Moreover,
our method enables efficient training by i) filtering out zero-variance prompts
that are either trivial or unsolvable and ii) cutting off lengthy rollouts when
they are unlikely to lead to correct answers. The results are inspiring: our
strategy yields significant improvements in both accuracy and training
efficiency on in-domain tasks and shows strong generalization to out-of-domain
benchmarks. More specifically, our method can speed up GRPO training by over
1.28x to reach the same performance, and achieve a 19.3% gain in accuracy on
AIME25, and a 6.2 % average gain over six mathematics benchmarks. Training with
meta-cognitive guidance enhances out-of-domain generalization, giving a 3.87 %
boost on GPQA-Diamond and a 2.08 % overall accuracy gain across 13 benchmarks
spanning logical, scientific, and coding domains.
A training pipeline called MASA enhances meta-awareness in reasoning models, leading to improved accuracy and efficiency across various benchmarks.
meta-awareness reasoning models meta-prediction rollouts self-alignment zero-variance prompts GRPO AIME25 GPQA-Diamond