DuPO: Enabling Reliable LLM Self-Verification via Dual Preference
  Optimization
We present DuPO, a dual learning-based preference optimization framework that
generates annotation-free feedback via a generalized duality. DuPO addresses
two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)'s
reliance on costly labels and applicability restricted to verifiable tasks, and
traditional dual learning's restriction to strictly dual task pairs (e.g.,
translation and back-translation). Specifically, DuPO decomposes a primal
task's input into known and unknown components, then constructs its dual task
to reconstruct the unknown part using the primal output and known information
(e.g., reversing math solutions to recover hidden variables), broadening
applicability to non-invertible tasks. The quality of this reconstruction
serves as a self-supervised reward to optimize the primal task, synergizing
with LLMs' ability to instantiate both tasks via a single model. Empirically,
DuPO achieves substantial gains across diverse tasks: it enhances the average
translation quality by 2.13 COMET over 756 directions, boosts the mathematical
reasoning accuracy by an average of 6.4 points on three challenge benchmarks,
and enhances performance by 9.3 points as an inference-time reranker (trading
computation for accuracy). These results position DuPO as a scalable, general,
and annotation-free paradigm for LLM optimization.
68a69c4a9e4b49496aac699d
Shuaijie She
admin_assigned
2025-08-21T10:54:56.836Z
61a9ccca3e8d72e791476614
https://cdn-avatars.huggingface.co/v1/production/uploads/61a9ccca3e8d72e791476614/icULFDp3dPKgwUePllNly.png
Shuaijie She
user
kevinpro
68a69c4a9e4b49496aac699e
Yu Bao
claimed_verified
2025-08-21T13:32:14.025Z
6142f4041fbcc4d3c42020fa
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/51c_M6rDZoJvnuC3eauqZ.png
Yu Bao
user
baoy
68a69c4a9e4b49496aac699f
Yu Lu
claimed_verified
2025-08-21T13:32:09.682Z
65e56e44e8b017ee13408589
/avatars/f9a8a7f7d2ff4474cad752a66452ce1e.svg
Yu Lu
user
YuLu0713
68a69c4a9e4b49496aac69a0
Lu Xu
68a69c4a9e4b49496aac69a1
Tao Li
68a69c4a9e4b49496aac69a2
Wenhao Zhu
claimed_verified
2025-08-21T13:32:11.842Z
649d7d8968586ca9bf7f5fe6
/avatars/b444240770d4025dea41871cf38126dc.svg
Wenhao Zhu
user
Wenhao97
68a69c4a9e4b49496aac69a3
Shujian Huang
admin_assigned
2025-08-21T10:55:13.771Z
687ba59e595ea10fe729316e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mHa4lsl7glwop3S9lq4BP.png
Shujian Huang
user
ShujianHuang
68a69c4a9e4b49496aac69a4
Shanbo Cheng
admin_assigned
2025-08-21T10:55:07.743Z
632ab0407fb39c2b6350c10a
/avatars/fb02cad2a017654965486418bf370157.svg
Cheng
user
Shanbo
68a69c4a9e4b49496aac69a5
Lu Lu
68a69c4a9e4b49496aac69a6
Yuxuan Wang
DuPO: Enabling Reliable LLM Self-Verification via Dual Preference
  Optimization
We present DuPO, a dual learning-based preference optimization framework that
generates annotation-free feedback via a generalized duality. DuPO addresses
two key limitations: Reinforcement Learning with Verifiable Rewards (RLVR)'s
reliance on costly labels and applicability restricted to verifiable tasks, and
traditional dual learning's restriction to strictly dual task pairs (e.g.,
translation and back-translation). Specifically, DuPO decomposes a primal
task's input into known and unknown components, then constructs its dual task
to reconstruct the unknown part using the primal output and known information
(e.g., reversing math solutions to recover hidden variables), broadening
applicability to non-invertible tasks. The quality of this reconstruction
serves as a self-supervised reward to optimize the primal task, synergizing
with LLMs' ability to instantiate both tasks via a single model. Empirically,
DuPO achieves substantial gains across diverse tasks: it enhances the average
translation quality by 2.13 COMET over 756 directions, boosts the mathematical
reasoning accuracy by an average of 6.4 points on three challenge benchmarks,
and enhances performance by 9.3 points as an inference-time reranker (trading
computation for accuracy). These results position DuPO as a scalable, general,
and annotation-free paradigm for LLM optimization.
DuPO is a dual learning framework that generates annotation-free feedback using a generalized duality, enhancing performance across various tasks without relying on costly labels.
Reinforcement Learning with Verifiable Rewards dual learning primal task dual task self-supervised reward LLMs translation quality mathematical reasoning accuracy inference-time reranker