DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging
Reinforcement learning from human feedback (RLHF) is a popular strategy for
aligning large language models (LLMs) with desired behaviors. Reward modeling
is a crucial step in RLHF. However, collecting paired preference data for
training reward models is often costly and time-consuming, especially for
domain-specific preferences requiring expert annotation. To address this
challenge, we propose the Domain knowledge merged
Reward Model (DogeRM), a novel framework that integrates
domain-specific knowledge into a general reward model by model merging. The
experiments demonstrate that DogeRM enhances performance across different
benchmarks and provide a detailed analysis showcasing the effects of model
merging, showing the great potential of facilitating model alignment.
6683648cacdf070b0e255757
Tzu-Han Lin
claimed_verified
2024-07-02T07:39:02.988Z
6602787b1827b6d37ee527be
/avatars/f69c1779b4347a042dad1a0d962145af.svg
Tzu-Han Lin
user
hank0316
6683648cacdf070b0e255758
Chen-An Li
claimed_verified
2024-07-04T08:05:00.275Z
64206697b286e8c464f50440
/avatars/d628a269fcc4b572ed5868135e10f65a.svg
Chen An, Li
user
lca0503
6683648cacdf070b0e255759
Hung-yi Lee
admin_assigned
2024-07-03T15:08:52.318Z
629b6eb3ad498ab19282ae6f
/avatars/adade8c512122120b06e5c50724a178d.svg
Hung-yi Lee
user
hungyilee
6683648cacdf070b0e25575a
Yun-Nung Chen
DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging
Reinforcement learning from human feedback (RLHF) is a popular strategy for
aligning large language models (LLMs) with desired behaviors. Reward modeling
is a crucial step in RLHF. However, collecting paired preference data for
training reward models is often costly and time-consuming, especially for
domain-specific preferences requiring expert annotation. To address this
challenge, we propose the Domain knowledge merged
Reward Model (DogeRM), a novel framework that integrates
domain-specific knowledge into a general reward model by model merging. The
experiments demonstrate that DogeRM enhances performance across different
benchmarks and provide a detailed analysis showcasing the effects of model
merging, showing the great potential of facilitating model alignment.
reinforcement learning from human feedback (RLHF) large language models (LLMs) reward modeling paired preference data domain-specific preferences expert annotation Domain Knowledge merged Reward Model (DogeRM) model merging