Jointly Reinforcing Diversity and Quality in Language Model Generations
Post-training of Large Language Models (LMs) often prioritizes accuracy and
helpfulness at the expense of diversity. This creates a tension: while
post-training improves response quality, it also sharpens output distributions
and reduces the range of ideas, limiting the usefulness of LMs in creative and
exploratory tasks such as brainstorming, storytelling, or problem solving. We
address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a
framework that jointly optimizes for response quality and semantic diversity.
At its core, DARLING introduces a learned partition function to measure
diversity beyond surface-level lexical variations. This diversity signal is
then combined with a quality reward during online reinforcement learning,
encouraging models to generate outputs that are both high-quality and distinct.
Experiments across multiple model families and sizes show that DARLING
generalizes to two regimes: non-verifiable tasks (instruction following and
creative writing) and verifiable tasks (competition math). On five benchmarks
in the first setting, DARLING consistently outperforms quality-only RL
baselines, producing outputs that are simultaneously of higher quality and
novelty. In the second setting, DARLING achieves higher pass@1 (solution
quality) and pass@k (solution variety). Most strikingly, explicitly optimizing
for diversity catalyzes exploration in online RL, which manifests itself as
higher-quality responses.
68b7b787295f15ff609112f8
Tianjian Li
claimed_verified
2025-09-03T08:27:19.866Z
62bc72d464b7a623f3a242e9
/avatars/db79848dc78a5c4927e27de4426c3b47.svg
Tianjian Li
user
dogtooth
68b7b787295f15ff609112f9
Yiming Zhang
68b7b787295f15ff609112fa
Ping Yu
68b7b787295f15ff609112fb
Swarnadeep Saha
68b7b787295f15ff609112fc
Daniel Khashabi
admin_assigned
2025-09-03T08:58:17.626Z
5f6540c65e78cc6b0ed3199d
/avatars/0280d4df417855965a0964d22766c012.svg
Daniel Khashabi
user
danyaljj
68b7b787295f15ff609112fd
Jason Weston
admin_assigned
2025-09-03T08:58:38.308Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
68b7b787295f15ff609112fe
Jack Lanchantin
admin_assigned
2025-09-03T08:58:44.081Z
65dd30a6ea1a202d3cb469f7
/avatars/14029def026380e6e20822d916b25a72.svg
Jack Lanchantin
user
jcklcn
68b7b787295f15ff609112ff
Tianlu Wang
admin_assigned
2025-09-03T08:58:53.722Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
Jointly Reinforcing Diversity and Quality in Language Model Generations
Post-training of Large Language Models (LMs) often prioritizes accuracy and
helpfulness at the expense of diversity. This creates a tension: while
post-training improves response quality, it also sharpens output distributions
and reduces the range of ideas, limiting the usefulness of LMs in creative and
exploratory tasks such as brainstorming, storytelling, or problem solving. We
address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a
framework that jointly optimizes for response quality and semantic diversity.
At its core, DARLING introduces a learned partition function to measure
diversity beyond surface-level lexical variations. This diversity signal is
then combined with a quality reward during online reinforcement learning,
encouraging models to generate outputs that are both high-quality and distinct.
Experiments across multiple model families and sizes show that DARLING
generalizes to two regimes: non-verifiable tasks (instruction following and
creative writing) and verifiable tasks (competition math). On five benchmarks
in the first setting, DARLING consistently outperforms quality-only RL
baselines, producing outputs that are simultaneously of higher quality and
novelty. In the second setting, DARLING achieves higher pass@1 (solution
quality) and pass@k (solution variety). Most strikingly, explicitly optimizing
for diversity catalyzes exploration in online RL, which manifests itself as
higher-quality responses.
DARLING, a diversity-aware reinforcement learning framework, enhances both the quality and diversity of large language model outputs across various tasks.
Diversity-Aware Reinforcement Learning DARLING learned partition function semantic diversity quality reward online reinforcement learning non-verifiable tasks verifiable tasks instruction following creative writing competition math pass@1 pass@k exploration