SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Evaluating the step-by-step reliability of large language model (LLM)
reasoning, such as Chain-of-Thought, remains challenging due to the difficulty
and cost of obtaining high-quality step-level supervision. In this paper, we
introduce Self-Play Critic (SPC), a novel approach where a critic model evolves
its ability to assess reasoning steps through adversarial self-play games,
eliminating the need for manual step-level annotation. SPC involves fine-tuning
two copies of a base model to play two roles, namely a "sneaky generator" that
deliberately produces erroneous steps designed to be difficult to detect, and a
"critic" that analyzes the correctness of reasoning steps. These two models
engage in an adversarial game in which the generator aims to fool the critic,
while the critic model seeks to identify the generator's errors. Using
reinforcement learning based on the game outcomes, the models iteratively
improve; the winner of each confrontation receives a positive reward and the
loser receives a negative reward, driving continuous self-evolution.
Experiments on three reasoning process benchmarks (ProcessBench, PRM800K,
DeltaBench) demonstrate that our SPC progressively enhances its error detection
capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and
surpasses strong baselines, including distilled R1 model. Furthermore, applying
SPC to guide the test-time search of diverse LLMs significantly improves their
mathematical reasoning performance on MATH500 and AIME2024, outperforming
state-of-the-art process reward models.
6810799e10b86ba322c27fb8
Jiaqi Chen
claimed_verified
2025-04-29T07:58:59.273Z
61b859ddbdf1fac5ed499992
/avatars/2387fb9b8a46840bfc75248462f0a410.svg
Jiaqi Chen
user
judge
6810799e10b86ba322c27fb9
Bang Zhang
6810799e10b86ba322c27fba
Ruotian Ma
admin_assigned
2025-04-29T12:24:07.283Z
648294b2eb4befee378951c1
/avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
Ruotian Ma
user
vvibt
6810799e10b86ba322c27fbb
Peisong Wang
admin_assigned
2025-04-29T12:24:12.917Z
626f98528a894872cfbf620c
/avatars/fe31d20313e6ca85e96bc249424c5383.svg
Peisong Wang
user
duke1852022
6810799e10b86ba322c27fbc
Xiaodan Liang
6810799e10b86ba322c27fbd
Zhaopeng Tu
admin_assigned
2025-04-29T12:24:26.048Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
6810799e10b86ba322c27fbe
Xiaolong Li
6810799e10b86ba322c27fbf
Kwan-Yee K. Wong
SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
Evaluating the step-by-step reliability of large language model (LLM)
reasoning, such as Chain-of-Thought, remains challenging due to the difficulty
and cost of obtaining high-quality step-level supervision. In this paper, we
introduce Self-Play Critic (SPC), a novel approach where a critic model evolves
its ability to assess reasoning steps through adversarial self-play games,
eliminating the need for manual step-level annotation. SPC involves fine-tuning
two copies of a base model to play two roles, namely a "sneaky generator" that
deliberately produces erroneous steps designed to be difficult to detect, and a
"critic" that analyzes the correctness of reasoning steps. These two models
engage in an adversarial game in which the generator aims to fool the critic,
while the critic model seeks to identify the generator's errors. Using
reinforcement learning based on the game outcomes, the models iteratively
improve; the winner of each confrontation receives a positive reward and the
loser receives a negative reward, driving continuous self-evolution.
Experiments on three reasoning process benchmarks (ProcessBench, PRM800K,
DeltaBench) demonstrate that our SPC progressively enhances its error detection
capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and
surpasses strong baselines, including distilled R1 model. Furthermore, applying
SPC to guide the test-time search of diverse LLMs significantly improves their
mathematical reasoning performance on MATH500 and AIME2024, outperforming
state-of-the-art process reward models.
Self-Play Critic (SPC) enhances LLM reasoning reliability by using adversarial self-play to improve error detection and reasoning performance without requiring manual step-level annotation.
Self-Play Critic Chain-of-Thought adversarial self-play reinforcement learning reasoning steps error detection ProcessBench PRM800K DeltaBench mathematical reasoning MATH500 AIME2024