Rethinking Diverse Human Preference Learning through Principal Component
  Analysis
Understanding human preferences is crucial for improving foundation models
and building personalized AI systems. However, preferences are inherently
diverse and complex, making it difficult for traditional reward models to
capture their full range. While fine-grained preference data can help,
collecting it is expensive and hard to scale. In this paper, we introduce
Decomposed Reward Models (DRMs), a novel approach that extracts diverse human
preferences from binary comparisons without requiring fine-grained annotations.
Our key insight is to represent human preferences as vectors and analyze them
using Principal Component Analysis (PCA). By constructing a dataset of
embedding differences between preferred and rejected responses, DRMs identify
orthogonal basis vectors that capture distinct aspects of preference. These
decomposed rewards can be flexibly combined to align with different user needs,
offering an interpretable and scalable alternative to traditional reward
models. We demonstrate that DRMs effectively extract meaningful preference
dimensions (e.g., helpfulness, safety, humor) and adapt to new users without
additional training. Our results highlight DRMs as a powerful framework for
personalized and interpretable LLM alignment.
67b5461d29cc269e5a4eb823
Feng Luo
67b5461d29cc269e5a4eb824
Rui Yang
claimed_verified
2025-02-19T09:01:23.095Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
user
Ray2333
67b5461d29cc269e5a4eb825
Hao Sun
67b5461d29cc269e5a4eb826
Chunyuan Deng
admin_assigned
2025-02-19T09:56:33.053Z
634b9914dcf125e4da02498b
https://cdn-avatars.huggingface.co/v1/production/uploads/634b9914dcf125e4da02498b/crRgFroWq5U6XWtvlTXSZ.jpeg
Chunyuan Deng
user
CharlesDDDD
67b5461d29cc269e5a4eb827
Jiarui Yao
claimed_verified
2025-05-06T08:37:11.978Z
649369b34f0e40ee1a0ed5ba
/avatars/50d0e77883579d5002906c8d29c26ec5.svg
Maxwell Yao
user
MaxwellJryao
67b5461d29cc269e5a4eb828
Jingyan Shen
67b5461d29cc269e5a4eb829
Huan Zhang
admin_assigned
2025-02-19T09:52:47.329Z
6719d581a6cad13741b8bc7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
Huan Zhang
user
huanzhang12
67b5461d29cc269e5a4eb82a
Hanjie Chen
Rethinking Diverse Human Preference Learning through Principal Component
  Analysis
Understanding human preferences is crucial for improving foundation models
and building personalized AI systems. However, preferences are inherently
diverse and complex, making it difficult for traditional reward models to
capture their full range. While fine-grained preference data can help,
collecting it is expensive and hard to scale. In this paper, we introduce
Decomposed Reward Models (DRMs), a novel approach that extracts diverse human
preferences from binary comparisons without requiring fine-grained annotations.
Our key insight is to represent human preferences as vectors and analyze them
using Principal Component Analysis (PCA). By constructing a dataset of
embedding differences between preferred and rejected responses, DRMs identify
orthogonal basis vectors that capture distinct aspects of preference. These
decomposed rewards can be flexibly combined to align with different user needs,
offering an interpretable and scalable alternative to traditional reward
models. We demonstrate that DRMs effectively extract meaningful preference
dimensions (e.g., helpfulness, safety, humor) and adapt to new users without
additional training. Our results highlight DRMs as a powerful framework for
personalized and interpretable LLM alignment.
Decomposed Reward Models (DRMs) use PCA to extract interpretable preference dimensions from binary comparisons, providing a scalable alternative to traditional reward models for personalized AI systems.
Decomposed Reward Models (DRMs) Principal Component Analysis (PCA) preference vectors orthogonal basis vectors preference dimensions personalized AI systems LLM alignment