M-RewardBench: Evaluating Reward Models in Multilingual Settings
Reward models (RMs) have driven the state-of-the-art performance of LLMs
today by enabling the integration of human feedback into the language modeling
process. However, RMs are primarily trained and evaluated in English, and their
capabilities in multilingual settings remain largely understudied. In this
work, we conduct a systematic evaluation of several reward models in
multilingual settings. We first construct the first-of-its-kind multilingual RM
evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
for 23 typologically diverse languages, that tests the chat, safety, reasoning,
and translation capabilities of RMs. We then rigorously evaluate a wide range
of reward models on M-RewardBench, offering fresh insights into their
performance across diverse languages. We identify a significant gap in RMs'
performances between English and non-English languages and show that RM
preferences can change substantially from one language to another. We also
present several findings on how different multilingual aspects impact RM
performance. Specifically, we show that the performance of RMs is improved with
improved translation quality. Similarly, we demonstrate that the models exhibit
better performance for high-resource languages. We release M-RewardBench
dataset and the codebase in this study to facilitate a better understanding of
RM evaluation in multilingual settings.
671917223aa9749caf1004d1
Srishti Gureja
extracted_confirmed
2024-10-23T16:35:20.509Z
625d6ed9c4086370158824eb
https://cdn-avatars.huggingface.co/v1/production/uploads/625d6ed9c4086370158824eb/KG_Plv-2rcydKzVf-9AAj.jpeg
Srishti Gureja
user
srishti-hf1110
671917223aa9749caf1004d2
Lester James V. Miranda
claimed_verified
2024-10-25T09:31:03.286Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
671917223aa9749caf1004d3
Shayekh Bin Islam
claimed_verified
2024-10-24T07:15:41.072Z
5e4b943a37cb5b49818287b5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e4b943a37cb5b49818287b5/Sd8QOGFY-_B1z4UlFTuPf.jpeg
Shayekh Islam
user
shayekh
671917223aa9749caf1004d4
Rishabh Maheshwary
admin_assigned
2024-10-24T14:24:26.868Z
645c26d423ed9b7788d5e24b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/cZMUluWpYUlSLcn6yoC7c.jpeg
Rishabh Maheshwary
user
rmahesh
671917223aa9749caf1004d5
Drishti Sharma
admin_assigned
2024-10-24T14:24:20.823Z
618c1ad1c74578e0a4a4d074
https://cdn-avatars.huggingface.co/v1/production/uploads/618c1ad1c74578e0a4a4d074/8u_AkeHt4d6xtQ8hzaffU.jpeg
Drishti Sharma
user
DrishtiSharma
671917223aa9749caf1004d6
Gusti Winata
claimed_verified
2024-10-28T11:25:14.521Z
61e3d0829d24288616e5205e
https://cdn-avatars.huggingface.co/v1/production/uploads/61e3d0829d24288616e5205e/bwsQBncSA5oGGP8H--Wi-.jpeg
Gusti Triandi Winata
user
sanggusti
671917223aa9749caf1004d7
Nathan Lambert
admin_assigned
2024-10-24T14:24:04.907Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
671917223aa9749caf1004d8
Sebastian Ruder
admin_assigned
2024-10-24T14:23:31.119Z
624575789dc26c59bae6a19e
https://cdn-avatars.huggingface.co/v1/production/uploads/1648719221694-noauth.jpeg
Sebastian Ruder
user
ruder
671917223aa9749caf1004d9
Sara Hooker
admin_assigned
2024-10-24T14:23:23.838Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
671917223aa9749caf1004da
Marzieh Fadaee
admin_assigned
2024-10-24T14:23:17.162Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
M-RewardBench: Evaluating Reward Models in Multilingual Settings
Reward models (RMs) have driven the state-of-the-art performance of LLMs
today by enabling the integration of human feedback into the language modeling
process. However, RMs are primarily trained and evaluated in English, and their
capabilities in multilingual settings remain largely understudied. In this
work, we conduct a systematic evaluation of several reward models in
multilingual settings. We first construct the first-of-its-kind multilingual RM
evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
for 23 typologically diverse languages, that tests the chat, safety, reasoning,
and translation capabilities of RMs. We then rigorously evaluate a wide range
of reward models on M-RewardBench, offering fresh insights into their
performance across diverse languages. We identify a significant gap in RMs'
performances between English and non-English languages and show that RM
preferences can change substantially from one language to another. We also
present several findings on how different multilingual aspects impact RM
performance. Specifically, we show that the performance of RMs is improved with
improved translation quality. Similarly, we demonstrate that the models exhibit
better performance for high-resource languages. We release M-RewardBench
dataset and the codebase in this study to facilitate a better understanding of
RM evaluation in multilingual settings.
A systematic evaluation of reward models in multilingual settings reveals significant performance gaps and dependencies on translation quality and resource availability.
Reward models multilingual settings M-RewardBench preference instances translation quality high-resource languages