SPARK: Synergistic Policy And Reward Co-Evolving Framework
Recent Large Language Models (LLMs) and Large Vision-Language Models (LVLMs)
increasingly use Reinforcement Learning (RL) for post-pretraining, such as RL
with Verifiable Rewards (RLVR) for objective tasks and RL from Human Feedback
(RLHF) for subjective tasks. However, RLHF incurs high costs and potential
reward-policy mismatch due to reliance on human preferences, while RLVR still
wastes supervision by discarding rollouts and correctness signals after each
update. To address these challenges, we introduce the Synergistic Policy And
Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable
method that builds on RLVR. Instead of discarding rollouts and correctness
data, SPARK recycles this valuable information to simultaneously train the
model itself as a generative reward model. This auxiliary training uses a mix
of objectives, such as pointwise reward score, pairwise comparison, and
evaluation conditioned on further-reflection responses, to teach the model to
evaluate and improve its own responses. Our process eliminates the need for a
separate reward model and costly human preference data. SPARK creates a
positive co-evolving feedback loop: improved reward accuracy yields better
policy gradients, which in turn produce higher-quality rollouts that further
refine the reward model. Our unified framework supports test-time scaling via
self-reflection without external reward models and their associated costs. We
show that SPARK achieves significant performance gains on multiple LLM and LVLM
models and multiple reasoning, reward models, and general benchmarks. For
example, SPARK-VL-7B achieves an average 9.7% gain on 7 reasoning benchmarks,
12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over the
baselines, demonstrating robustness and broad generalization.
68d9ed460177a6054b013aab
Ziyu Liu
68d9ed460177a6054b013aac
Yuhang Zang
claimed_verified
2025-09-29T05:58:32.038Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
user
yuhangzang
68d9ed460177a6054b013aad
Shengyuan Ding
68d9ed460177a6054b013aae
Yuhang Cao
68d9ed460177a6054b013aaf
Xiaoyi Dong
68d9ed460177a6054b013ab0
Haodong Duan
claimed_verified
2025-10-22T15:47:15.875Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
68d9ed460177a6054b013ab1
Dahua Lin
68d9ed460177a6054b013ab2
Jiaqi Wang
claimed_verified
2025-09-29T05:58:29.391Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
SPARK: Synergistic Policy And Reward Co-Evolving Framework
Recent Large Language Models (LLMs) and Large Vision-Language Models (LVLMs)
increasingly use Reinforcement Learning (RL) for post-pretraining, such as RL
with Verifiable Rewards (RLVR) for objective tasks and RL from Human Feedback
(RLHF) for subjective tasks. However, RLHF incurs high costs and potential
reward-policy mismatch due to reliance on human preferences, while RLVR still
wastes supervision by discarding rollouts and correctness signals after each
update. To address these challenges, we introduce the Synergistic Policy And
Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable
method that builds on RLVR. Instead of discarding rollouts and correctness
data, SPARK recycles this valuable information to simultaneously train the
model itself as a generative reward model. This auxiliary training uses a mix
of objectives, such as pointwise reward score, pairwise comparison, and
evaluation conditioned on further-reflection responses, to teach the model to
evaluate and improve its own responses. Our process eliminates the need for a
separate reward model and costly human preference data. SPARK creates a
positive co-evolving feedback loop: improved reward accuracy yields better
policy gradients, which in turn produce higher-quality rollouts that further
refine the reward model. Our unified framework supports test-time scaling via
self-reflection without external reward models and their associated costs. We
show that SPARK achieves significant performance gains on multiple LLM and LVLM
models and multiple reasoning, reward models, and general benchmarks. For
example, SPARK-VL-7B achieves an average 9.7% gain on 7 reasoning benchmarks,
12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over the
baselines, demonstrating robustness and broad generalization.
SPARK, a synergistic policy and reward co-evolving framework, enhances LLMs and LVLMs by recycling rollouts and correctness data to train a generative reward model, reducing reliance on human preferences and external reward models.
Reinforcement Learning RL with Verifiable Rewards RL from Human Feedback Synergistic Policy And Reward Co-Evolving Framework SPARK on-policy generative reward model pointwise reward score pairwise comparison self-reflection policy gradients rollouts reward accuracy performance gains reasoning benchmarks reward benchmarks general benchmarks