Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
Despite the critical role of reward models (RMs) in reinforcement learning
from human feedback (RLHF), current state-of-the-art open RMs perform poorly on
most existing evaluation benchmarks, failing to capture the spectrum of nuanced
and sophisticated human preferences. Even approaches that incorporate advanced
training techniques have not yielded meaningful performance improvements. We
hypothesize that this brittleness stems primarily from limitations in
preference datasets, which are often narrowly scoped, synthetically labeled, or
lack rigorous quality control. To address these challenges, we present a
large-scale preference dataset comprising 40 million preference pairs, named
SynPref-40M. To enable data curation at scale, we design a human-AI synergistic
two-stage pipeline that leverages the complementary strengths of human
annotation quality and AI scalability. In this pipeline, humans provide
verified annotations, while large language models perform automatic curation
based on human guidance. Training on this preference mixture, we introduce
Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B
parameters, trained on a carefully curated subset of 26 million preference
pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile
across a wide range of capabilities, including alignment with human
preferences, objective correctness, safety, resistance to stylistic biases, and
best-of-N scaling, achieving state-of-the-art performance across seven major
reward model benchmarks. Ablation studies confirm that the effectiveness of our
approach stems not only from data scale but also from high-quality curation.
The Skywork-Reward-V2 series represents substantial progress in open reward
models, highlighting the untapped potential of existing preference datasets and
demonstrating how human-AI curation synergy can unlock significantly higher
data quality.
6865cdc28c83dab5f72d1e18
Chris Yuhao Liu
claimed_verified
2025-07-04T07:50:33.676Z
658229ef5f6d83438257fce5
/avatars/b4417de9a338e95dc69cc547a46348e8.svg
Chris (Yuhao) Liu
user
chrisliu298
6865cdc28c83dab5f72d1e19
Liang Zeng
claimed_verified
2025-07-03T16:19:08.221Z
6621efe1a6eec3ad03e38759
/avatars/c35acce69f244ec0833dffd53eedf6a3.svg
Liang Zeng
user
zengliangcs
6865cdc28c83dab5f72d1e1a
Yuzhen Xiao
claimed_verified
2025-07-04T07:50:29.586Z
658ceb595a8f8a309ea417a1
/avatars/43bfa8c919aa802f2611439ebb7430b8.svg
Ricky Shaw
user
RickyShaw999
6865cdc28c83dab5f72d1e1b
Jujie He
6865cdc28c83dab5f72d1e1c
Jiacai Liu
6865cdc28c83dab5f72d1e1d
Chaojie Wang
6865cdc28c83dab5f72d1e1e
Rui Yan
6865cdc28c83dab5f72d1e1f
Wei Shen
6865cdc28c83dab5f72d1e20
Fuxiang Zhang
6865cdc28c83dab5f72d1e21
Jiacheng Xu
6865cdc28c83dab5f72d1e22
Yang Liu
6865cdc28c83dab5f72d1e23
Yahui Zhou
Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
Despite the critical role of reward models (RMs) in reinforcement learning
from human feedback (RLHF), current state-of-the-art open RMs perform poorly on
most existing evaluation benchmarks, failing to capture the spectrum of nuanced
and sophisticated human preferences. Even approaches that incorporate advanced
training techniques have not yielded meaningful performance improvements. We
hypothesize that this brittleness stems primarily from limitations in
preference datasets, which are often narrowly scoped, synthetically labeled, or
lack rigorous quality control. To address these challenges, we present a
large-scale preference dataset comprising 40 million preference pairs, named
SynPref-40M. To enable data curation at scale, we design a human-AI synergistic
two-stage pipeline that leverages the complementary strengths of human
annotation quality and AI scalability. In this pipeline, humans provide
verified annotations, while large language models perform automatic curation
based on human guidance. Training on this preference mixture, we introduce
Skywork-Reward-V2, a suite of eight reward models ranging from 0.6B to 8B
parameters, trained on a carefully curated subset of 26 million preference
pairs from SynPref-40M. We demonstrate that Skywork-Reward-V2 is versatile
across a wide range of capabilities, including alignment with human
preferences, objective correctness, safety, resistance to stylistic biases, and
best-of-N scaling, achieving state-of-the-art performance across seven major
reward model benchmarks. Ablation studies confirm that the effectiveness of our
approach stems not only from data scale but also from high-quality curation.
The Skywork-Reward-V2 series represents substantial progress in open reward
models, highlighting the untapped potential of existing preference datasets and
demonstrating how human-AI curation synergy can unlock significantly higher
data quality.
Skywork-Reward-V2, a suite of reward models trained on a large-scale, high-quality preference dataset, achieves state-of-the-art performance across various benchmarks by leveraging human-AI synergistic curation.
reward models reinforcement learning from human feedback preference datasets human-AI synergistic pipeline large language models preference pairs Skywork-Reward-V2 alignment with human preferences objective correctness safety stylistic biases best-of-N scaling