Diversity-Enhanced Reasoning for Subjective Questions
Large reasoning models (LRM) with long chain-of-thought (CoT) capabilities
have shown strong performance on objective tasks, such as math reasoning and
coding. However, their effectiveness on subjective questions that may have
different responses from different perspectives is still limited by a tendency
towards homogeneous reasoning, introduced by the reliance on a single ground
truth in supervised fine-tuning and verifiable reward in reinforcement
learning. Motivated by the finding that increasing role perspectives
consistently improves performance, we propose MultiRole-R1, a
diversity-enhanced framework with multiple role perspectives, to improve the
accuracy and diversity in subjective reasoning tasks. MultiRole-R1 features an
unsupervised data construction pipeline that generates reasoning chains that
incorporate diverse role perspectives. We further employ reinforcement learning
via Group Relative Policy Optimization (GRPO) with reward shaping, by taking
diversity as a reward signal in addition to the verifiable reward. With
specially designed reward functions, we successfully promote perspective
diversity and lexical diversity, uncovering a positive relation between
reasoning diversity and accuracy. Our experiment on six benchmarks demonstrates
MultiRole-R1's effectiveness and generalizability in enhancing both subjective
and objective reasoning, showcasing the potential of diversity-enhanced
training in LRMs.
688889bbcb0983f285a65202
Yumeng Wang
688889bbcb0983f285a65203
Zhiyuan Fan
claimed_verified
2025-07-29T12:50:48.491Z
65867b038dd42194878e08dc
/avatars/8965ac2f8e3ccb88759f6e4d84a30c2f.svg
Zhiyuan Fan
user
Zhiyuan-Fan
688889bbcb0983f285a65204
Jiayu Liu
claimed_verified
2025-07-30T09:05:07.555Z
66783baec3f824dde8f783ac
/avatars/eb9a0441986be50274c5e661f7039e2c.svg
Jeff
user
JiayuJeff
688889bbcb0983f285a65205
Yi R. Fung
Diversity-Enhanced Reasoning for Subjective Questions
Large reasoning models (LRM) with long chain-of-thought (CoT) capabilities
have shown strong performance on objective tasks, such as math reasoning and
coding. However, their effectiveness on subjective questions that may have
different responses from different perspectives is still limited by a tendency
towards homogeneous reasoning, introduced by the reliance on a single ground
truth in supervised fine-tuning and verifiable reward in reinforcement
learning. Motivated by the finding that increasing role perspectives
consistently improves performance, we propose MultiRole-R1, a
diversity-enhanced framework with multiple role perspectives, to improve the
accuracy and diversity in subjective reasoning tasks. MultiRole-R1 features an
unsupervised data construction pipeline that generates reasoning chains that
incorporate diverse role perspectives. We further employ reinforcement learning
via Group Relative Policy Optimization (GRPO) with reward shaping, by taking
diversity as a reward signal in addition to the verifiable reward. With
specially designed reward functions, we successfully promote perspective
diversity and lexical diversity, uncovering a positive relation between
reasoning diversity and accuracy. Our experiment on six benchmarks demonstrates
MultiRole-R1's effectiveness and generalizability in enhancing both subjective
and objective reasoning, showcasing the potential of diversity-enhanced
training in LRMs.
A diversity-enhanced framework with multiple role perspectives improves accuracy and diversity in subjective reasoning tasks through unsupervised data construction and reinforcement learning with reward shaping.
Large reasoning models chain-of-thought homogeneous reasoning supervised fine-tuning reinforcement learning Group Relative Policy Optimization reward shaping perspective diversity lexical diversity subjective reasoning objective reasoning