RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented
  Generation for Preference Alignment
Despite the significant progress made by existing retrieval augmented
language models (RALMs) in providing trustworthy responses and grounding in
reliable sources, they often overlook effective alignment with human
preferences. In the alignment process, reward models (RMs) act as a crucial
proxy for human values to guide optimization. However, it remains unclear how
to evaluate and select a reliable RM for preference alignment in RALMs. To this
end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG
settings. First, we design four crucial and challenging RAG-specific scenarios
to assess RMs, including multi-hop reasoning, fine-grained citation,
appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG
subsets, six retrievers, and 24 RALMs to increase the diversity of data
sources. Finally, we adopt an LLM-as-a-judge approach to improve preference
annotation efficiency and effectiveness, exhibiting a strong correlation with
human annotations. Based on the RAG-RewardBench, we conduct a comprehensive
evaluation of 45 RMs and uncover their limitations in RAG scenarios.
Additionally, we also reveal that existing trained RALMs show almost no
improvement in preference alignment, highlighting the need for a shift towards
preference-aligned training.We release our benchmark and code publicly at
https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
67638ce2995c144ae40aaf21
Zhuoran Jin
admin_assigned
2024-12-20T10:08:30.400Z
643379416c6ecd58798421b3
/avatars/831db7eab2663abc33b176cf386b02f2.svg
Zhuoran Jin
user
jinzhuoran
67638ce2995c144ae40aaf22
Hongbang Yuan
admin_assigned
2024-12-20T10:08:37.222Z
6538d3089d66a6c304ee2f4b
/avatars/6e93ea6c937128765e5323fb4a5d1cea.svg
HongbangYuan
user
HongbangYuan
67638ce2995c144ae40aaf23
Tianyi Men
67638ce2995c144ae40aaf24
Pengfei Cao
admin_assigned
2024-12-20T10:08:59.825Z
6552d416b2d47c6675c580df
/avatars/b3bb5332e716a968ca8b8d92e5766cf2.svg
caopengfei
user
pengfei0626
67638ce2995c144ae40aaf25
Yubo Chen
admin_assigned
2024-12-20T10:09:19.607Z
64423a910f373bdb0463bf25
/avatars/6b41c4eb817836d50d0072009f124f29.svg
Yubo Chen
user
ybch14
67638ce2995c144ae40aaf26
Kang Liu
admin_assigned
2024-12-20T10:09:45.059Z
62d18d7d5f924243758dd23c
/avatars/d9d575558f10be661fe856236f4b72e3.svg
Kang Liu
user
cruyffio
67638ce2995c144ae40aaf27
Jun Zhao
RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented
  Generation for Preference Alignment
Despite the significant progress made by existing retrieval augmented
language models (RALMs) in providing trustworthy responses and grounding in
reliable sources, they often overlook effective alignment with human
preferences. In the alignment process, reward models (RMs) act as a crucial
proxy for human values to guide optimization. However, it remains unclear how
to evaluate and select a reliable RM for preference alignment in RALMs. To this
end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG
settings. First, we design four crucial and challenging RAG-specific scenarios
to assess RMs, including multi-hop reasoning, fine-grained citation,
appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG
subsets, six retrievers, and 24 RALMs to increase the diversity of data
sources. Finally, we adopt an LLM-as-a-judge approach to improve preference
annotation efficiency and effectiveness, exhibiting a strong correlation with
human annotations. Based on the RAG-RewardBench, we conduct a comprehensive
evaluation of 45 RMs and uncover their limitations in RAG scenarios.
Additionally, we also reveal that existing trained RALMs show almost no
improvement in preference alignment, highlighting the need for a shift towards
preference-aligned training.We release our benchmark and code publicly at
https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.
A benchmark named RAG-RewardBench evaluates and selects reward models for preference alignment in retrieval augmented language models, uncovering their limitations and highlighting the need for preference-aligned training.
retrieval augmented language models RALMs reward models RMs RAG-RewardBench multi-hop reasoning fine-grained citation appropriate abstain conflict robustness LLM-as-a-judge