The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
As large language models (LLMs) continue to advance in linguistic
capabilities, robust multilingual evaluation has become essential for promoting
equitable technological progress. This position paper examines over 2,000
multilingual (non-English) benchmarks from 148 countries, published between
2021 and 2024, to evaluate past, present, and future practices in multilingual
benchmarking. Our findings reveal that, despite significant investments
amounting to tens of millions of dollars, English remains significantly
overrepresented in these benchmarks. Additionally, most benchmarks rely on
original language content rather than translations, with the majority sourced
from high-resource countries such as China, India, Germany, the UK, and the
USA. Furthermore, a comparison of benchmark performance with human judgments
highlights notable disparities. STEM-related tasks exhibit strong correlations
with human evaluations (0.70 to 0.85), while traditional NLP tasks like
question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
Moreover, translating English benchmarks into other languages proves
insufficient, as localized benchmarks demonstrate significantly higher
alignment with local human judgments (0.68) than their translated counterparts
(0.47). This underscores the importance of creating culturally and
linguistically tailored benchmarks rather than relying solely on translations.
Through this comprehensive analysis, we highlight six key limitations in
current multilingual evaluation practices, propose the guiding principles
accordingly for effective multilingual benchmarking, and outline five critical
research directions to drive progress in the field. Finally, we call for a
global collaborative effort to develop human-aligned benchmarks that prioritize
real-world applications.
6808458f07e80b69b2df2440
Minghao Wu
admin_assigned
2025-04-23T13:03:02.862Z
62d4bf8c97ab9eb08762a975
/avatars/73c6228e317cf37b4e3c3e7a4b3d8ae8.svg
Minghao Wu
user
minghaowu
6808458f07e80b69b2df2441
Weixuan Wang
admin_assigned
2025-04-23T13:03:08.988Z
675c52ecdea6ceb2f6ce0ea3
/avatars/b5ea0e2bc5b001446f62343a907e95f1.svg
weixuan wang
user
yourdadishere
6808458f07e80b69b2df2442
Sinuo Liu
admin_assigned
2025-04-23T13:03:19.254Z
635504620d6e89270d440050
/avatars/3790bf4a68f943a122af59b1362b07f2.svg
LiuSinuo
user
SNF
6808458f07e80b69b2df2443
Huifeng Yin
6808458f07e80b69b2df2444
Xintong Wang
claimed_verified
2025-05-12T06:51:30.134Z
674879885d97f2d66a14006a
/avatars/cdeff44a898560cb460223f30922b081.svg
Xintong Wang
user
shanewang
6808458f07e80b69b2df2445
Yu Zhao
6808458f07e80b69b2df2446
Chenyang Lyu
claimed_verified
2025-04-23T08:28:16.770Z
6527d8b077bceabaab382a75
/avatars/69caacf9153dbf6a3796693a968b363f.svg
Chenyang Lyu
user
ChenyangLyu
6808458f07e80b69b2df2447
Longyue Wang
admin_assigned
2025-04-23T13:04:14.454Z
636b030c328133bdb3a523bc
https://cdn-avatars.huggingface.co/v1/production/uploads/636b030c328133bdb3a523bc/f-OdbqqHiywkxQF1KVCLp.jpeg
Longyue Wang
user
longyuewang
6808458f07e80b69b2df2448
Weihua Luo
admin_assigned
2025-04-23T13:04:20.964Z
66b03cedd59c09785e39711e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N5yfQBSP3oAKPCz4ylR09.png
Weihua Luo
user
acecamel1977
6808458f07e80b69b2df2449
Kaifu Zhang
admin_assigned
2025-04-23T13:04:27.022Z
63f87ebadf053017d1acbfdd
/avatars/e497ba5f41a2587837b4a6118d9367bb.svg
Kaifu Zhang
user
zhangkaifu314
The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
As large language models (LLMs) continue to advance in linguistic
capabilities, robust multilingual evaluation has become essential for promoting
equitable technological progress. This position paper examines over 2,000
multilingual (non-English) benchmarks from 148 countries, published between
2021 and 2024, to evaluate past, present, and future practices in multilingual
benchmarking. Our findings reveal that, despite significant investments
amounting to tens of millions of dollars, English remains significantly
overrepresented in these benchmarks. Additionally, most benchmarks rely on
original language content rather than translations, with the majority sourced
from high-resource countries such as China, India, Germany, the UK, and the
USA. Furthermore, a comparison of benchmark performance with human judgments
highlights notable disparities. STEM-related tasks exhibit strong correlations
with human evaluations (0.70 to 0.85), while traditional NLP tasks like
question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
Moreover, translating English benchmarks into other languages proves
insufficient, as localized benchmarks demonstrate significantly higher
alignment with local human judgments (0.68) than their translated counterparts
(0.47). This underscores the importance of creating culturally and
linguistically tailored benchmarks rather than relying solely on translations.
Through this comprehensive analysis, we highlight six key limitations in
current multilingual evaluation practices, propose the guiding principles
accordingly for effective multilingual benchmarking, and outline five critical
research directions to drive progress in the field. Finally, we call for a
global collaborative effort to develop human-aligned benchmarks that prioritize
real-world applications.
Research reveals significant disparities in multilingual benchmark evaluations, emphasizing the need for culturally and linguistically tailored benchmarks over translations to achieve equitable technological progress.