LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a core paradigm for enhancing the reasoning capabilities of Large Language
Models (LLMs). To address the lack of verification signals at test time, prior
studies incorporate the training of model's self-verification capability into
the standard RLVR process, thereby unifying reasoning and verification
capabilities within a single LLM. However, previous practice requires the LLM
to sequentially generate solutions and self-verifications using two separate
prompt templates, which significantly reduces efficiency. In this work, we
theoretically reveal that the closed-form solution to the RL objective of
self-verification can be reduced to a remarkably simple form: the true
reasoning reward of a solution is equal to its last-token self-rewarding score,
which is computed as the difference between the policy model's next-token
log-probability assigned to any pre-specified token at the solution's last
token and a pre-calculated constant, scaled by the KL coefficient. Based on
this insight, we propose LaSeR (Reinforcement Learning with Last-Token
Self-Rewarding), an algorithm that simply augments the original RLVR loss with
a MSE loss that aligns the last-token self-rewarding scores with verifier-based
reasoning rewards, jointly optimizing the reasoning and self-rewarding
capabilities of LLMs. The optimized self-rewarding scores can be utilized in
both training and testing to enhance model performance. Notably, our algorithm
derives these scores from the predicted next-token probability distribution of
the last token immediately after generation, incurring only the minimal extra
cost of one additional token inference. Experiments show that our method not
only improves the model's reasoning performance but also equips it with
remarkable self-rewarding capability, thereby boosting its inference-time
scaling performance.
68f19d716e0bef323a68fc51
Wenkai Yang
claimed_verified
2025-10-17T04:11:28.203Z
64b7df742f5a966b973e25f7
/avatars/e24e7769188d441317b3b7d10ef8fd60.svg
Wenkai Yang
user
Keven16
68f19d716e0bef323a68fc52
Weijie Liu
68f19d716e0bef323a68fc53
Ruobing Xie
68f19d716e0bef323a68fc54
Yiju Guo
68f19d716e0bef323a68fc55
Lulu Wu
68f19d716e0bef323a68fc56
Saiyong Yang
68f19d716e0bef323a68fc57
Yankai Lin
LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a core paradigm for enhancing the reasoning capabilities of Large Language
Models (LLMs). To address the lack of verification signals at test time, prior
studies incorporate the training of model's self-verification capability into
the standard RLVR process, thereby unifying reasoning and verification
capabilities within a single LLM. However, previous practice requires the LLM
to sequentially generate solutions and self-verifications using two separate
prompt templates, which significantly reduces efficiency. In this work, we
theoretically reveal that the closed-form solution to the RL objective of
self-verification can be reduced to a remarkably simple form: the true
reasoning reward of a solution is equal to its last-token self-rewarding score,
which is computed as the difference between the policy model's next-token
log-probability assigned to any pre-specified token at the solution's last
token and a pre-calculated constant, scaled by the KL coefficient. Based on
this insight, we propose LaSeR (Reinforcement Learning with Last-Token
Self-Rewarding), an algorithm that simply augments the original RLVR loss with
a MSE loss that aligns the last-token self-rewarding scores with verifier-based
reasoning rewards, jointly optimizing the reasoning and self-rewarding
capabilities of LLMs. The optimized self-rewarding scores can be utilized in
both training and testing to enhance model performance. Notably, our algorithm
derives these scores from the predicted next-token probability distribution of
the last token immediately after generation, incurring only the minimal extra
cost of one additional token inference. Experiments show that our method not
only improves the model's reasoning performance but also equips it with
remarkable self-rewarding capability, thereby boosting its inference-time
scaling performance.
LaSeR, a reinforcement learning algorithm, enhances Large Language Models by aligning last-token self-rewarding scores with verifier-based reasoning rewards, improving reasoning performance and inference-time scaling.
Reinforcement Learning with Verifiable Rewards Large Language Models self-verification RLVR last-token self-rewarding policy model next-token log-probability KL coefficient MSE loss reasoning performance inference-time scaling