DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging
Reinforcement learning from human feedback (RLHF) is a popular strategy for
aligning large language models (LLMs) with desired behaviors. Reward modeling
is a crucial step in RLHF. However, collecting paired preference data for
training reward models is often costly and time-consuming, especially for
domain-specific preferences requiring expert annotation. To address this
challenge, we propose the Domain knowledge merged
Reward Model (DogeRM), a novel framework that integrates
domain-specific knowledge into a general reward model by model merging. The
experiments demonstrate that DogeRM enhances performance across different
benchmarks and provide a detailed analysis showcasing the effects of model
merging, showing the great potential of facilitating model alignment.
6683648cacdf070b0e255757
Tzu-Han Lin
claimed_verified
2024-07-02T07:39:02.988Z
6602787b1827b6d37ee527be
/avatars/f69c1779b4347a042dad1a0d962145af.svg
Tzu-Han Lin
user
hank0316
6683648cacdf070b0e255758
Chen-An Li
claimed_verified
2024-07-04T08:05:00.275Z
64206697b286e8c464f50440
/avatars/d628a269fcc4b572ed5868135e10f65a.svg
Chen An, Li
user
lca0503
6683648cacdf070b0e255759
Hung-yi Lee
admin_assigned
2024-07-03T15:08:52.318Z
629b6eb3ad498ab19282ae6f
/avatars/adade8c512122120b06e5c50724a178d.svg
Hung-yi Lee
user
hungyilee
6683648cacdf070b0e25575a
Yun-Nung Chen
DogeRM: Equipping Reward Models with Domain Knowledge through Model
  Merging
Reinforcement learning from human feedback (RLHF) is a popular strategy for
aligning large language models (LLMs) with desired behaviors. Reward modeling
is a crucial step in RLHF. However, collecting paired preference data for
training reward models is often costly and time-consuming, especially for
domain-specific preferences requiring expert annotation. To address this
challenge, we propose the Domain knowledge merged
Reward Model (DogeRM), a novel framework that integrates
domain-specific knowledge into a general reward model by model merging. The
experiments demonstrate that DogeRM enhances performance across different
benchmarks and provide a detailed analysis showcasing the effects of model
merging, showing the great potential of facilitating model alignment.
DogeRM, a novel framework, integrates domain-specific knowledge to enhance reward models in reinforcement learning from human feedback, improving model alignment across benchmarks.
reinforcement learning from human feedback large language models reward modeling domain-specific preferences expert annotation model merging domain knowledge merged reward model DogeRM model alignment