BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large
  Language Models
Previous multilingual benchmarks focus primarily on simple understanding
tasks, but for large language models(LLMs), we emphasize proficiency in
instruction following, reasoning, long context understanding, code generation,
and so on. However, measuring these advanced capabilities across languages is
underexplored. To address the disparity, we introduce BenchMAX, a multi-way
multilingual evaluation benchmark that allows for fair comparisons of these
important abilities across languages. To maintain high quality, three distinct
native-speaking annotators independently annotate each sample within all tasks
after the data was machine-translated from English into 16 other languages.
Additionally, we present a novel translation challenge stemming from dataset
construction. Extensive experiments on BenchMAX reveal varying effectiveness of
core capabilities across languages, highlighting performance gaps that cannot
be bridged by simply scaling up model size. BenchMAX serves as a comprehensive
multilingual evaluation platform, providing a promising test bed to promote the
development of multilingual language models. The dataset and code are publicly
accessible.
67ac4e046b8c86e0cc7988f0
Xu Huang
claimed_verified
2025-02-13T08:25:17.555Z
649d1d4c379eada9a580cf59
https://cdn-avatars.huggingface.co/v1/production/uploads/649d1d4c379eada9a580cf59/ucXv7KoJDEB3Phgn-Dn5E.png
xuhuang
user
xuhuang87
67ac4e046b8c86e0cc7988f1
Wenhao Zhu
67ac4e046b8c86e0cc7988f2
Hanxu Hu
67ac4e046b8c86e0cc7988f3
Conghui He
67ac4e046b8c86e0cc7988f4
Lei Li
67ac4e046b8c86e0cc7988f5
Shujian Huang
67ac4e046b8c86e0cc7988f6
Fei Yuan
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large
  Language Models
Previous multilingual benchmarks focus primarily on simple understanding
tasks, but for large language models(LLMs), we emphasize proficiency in
instruction following, reasoning, long context understanding, code generation,
and so on. However, measuring these advanced capabilities across languages is
underexplored. To address the disparity, we introduce BenchMAX, a multi-way
multilingual evaluation benchmark that allows for fair comparisons of these
important abilities across languages. To maintain high quality, three distinct
native-speaking annotators independently annotate each sample within all tasks
after the data was machine-translated from English into 16 other languages.
Additionally, we present a novel translation challenge stemming from dataset
construction. Extensive experiments on BenchMAX reveal varying effectiveness of
core capabilities across languages, highlighting performance gaps that cannot
be bridged by simply scaling up model size. BenchMAX serves as a comprehensive
multilingual evaluation platform, providing a promising test bed to promote the
development of multilingual language models. The dataset and code are publicly
accessible.
BenchMAX evaluates multilingual large language models across advanced capabilities like instruction following, reasoning, and code generation, revealing language-specific performance gaps.
multilingual benchmarks large language models (LLMs) instruction following reasoning long context understanding code generation BenchMAX native-speaking annotators machine translation translation challenge multilingual evaluation platform