Implicit Actor Critic Coupling via a Supervised Learning Framework for
  RLVR
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have
empowered large language models (LLMs) to tackle challenging reasoning tasks
such as mathematics and programming. RLVR leverages verifiable outcome rewards
to guide policy optimization, enabling LLMs to progressively improve output
quality in a grounded and reliable manner. Despite its promise, the RLVR
paradigm poses significant challenges, as existing methods often suffer from
sparse reward signals and unstable policy gradient updates, particularly in
RL-based approaches. To address the challenges, we propose PACS, a
novel RLVR framework that achieves imPlicit Actor
Critic coupling via a Supervised learning framework. By
treating the outcome reward as a predictable label, we reformulate the RLVR
problem into a supervised learning task over a score function parameterized by
the policy model and optimized using cross-entropy loss. A detailed gradient
analysis shows that this supervised formulation inherently recovers the
classical policy gradient update while implicitly coupling actor and critic
roles, yielding more stable and efficient training. Benchmarking on challenging
mathematical reasoning tasks, PACS outperforms strong RLVR baselines, such as
PPO and GRPO, achieving superior reasoning performance. For instance, PACS
achieves 59.78\% at pass@256 on AIME 2025, representing improvements of 13.32
and 14.36 points over PPO and GRPO. This simple yet powerful framework offers a
promising avenue for LLMs post-training with verifiable rewards. Our code and
data are available as open source at https://github.com/ritzz-ai/PACS.
68b7ad8a295f15ff609112bb
Jiaming Li
claimed_verified
2025-09-03T08:27:32.504Z
6355eaf660c1b72f6269bc64
/avatars/dd176b9d6db2ac63c19c3170566a3f35.svg
Jiaming Li
user
Geaming
68b7ad8a295f15ff609112bc
Longze Chen
claimed_verified
2025-09-04T08:44:21.407Z
64c7b4d1c547ed5243c07b6c
https://cdn-avatars.huggingface.co/v1/production/uploads/64c7b4d1c547ed5243c07b6c/h96CLBj6dcm01soK2UJzr.jpeg
Longze Chen
user
lzchen2001
68b7ad8a295f15ff609112bd
Ze Gong
68b7ad8a295f15ff609112be
Yukun Chen
68b7ad8a295f15ff609112bf
Lu Wang
68b7ad8a295f15ff609112c0
Wanwei He
68b7ad8a295f15ff609112c1
Run Luo
68b7ad8a295f15ff609112c2
Min Yang
Implicit Actor Critic Coupling via a Supervised Learning Framework for
  RLVR
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have
empowered large language models (LLMs) to tackle challenging reasoning tasks
such as mathematics and programming. RLVR leverages verifiable outcome rewards
to guide policy optimization, enabling LLMs to progressively improve output
quality in a grounded and reliable manner. Despite its promise, the RLVR
paradigm poses significant challenges, as existing methods often suffer from
sparse reward signals and unstable policy gradient updates, particularly in
RL-based approaches. To address the challenges, we propose PACS, a
novel RLVR framework that achieves imPlicit Actor
Critic coupling via a Supervised learning framework. By
treating the outcome reward as a predictable label, we reformulate the RLVR
problem into a supervised learning task over a score function parameterized by
the policy model and optimized using cross-entropy loss. A detailed gradient
analysis shows that this supervised formulation inherently recovers the
classical policy gradient update while implicitly coupling actor and critic
roles, yielding more stable and efficient training. Benchmarking on challenging
mathematical reasoning tasks, PACS outperforms strong RLVR baselines, such as
PPO and GRPO, achieving superior reasoning performance. For instance, PACS
achieves 59.78\% at pass@256 on AIME 2025, representing improvements of 13.32
and 14.36 points over PPO and GRPO. This simple yet powerful framework offers a
promising avenue for LLMs post-training with verifiable rewards. Our code and
data are available as open source at https://github.com/ritzz-ai/PACS.
PACS, a novel RLVR framework, reformulates RLVR as a supervised learning task, improving stability and efficiency in training large language models for reasoning tasks.
Reinforcement Learning with Verifiable Rewards (RLVR) large language models (LLMs) verifiable outcome rewards policy optimization sparse reward signals unstable policy gradient updates PACS Implicit Actor Critic coupling supervised learning score function cross-entropy loss policy gradient update AIME 2025 pass@256 PPO GRPO