IberBench: LLM Evaluation on Iberian Languages
Large Language Models (LLMs) remain difficult to evaluate comprehensively,
particularly for languages other than English, where high-quality data is often
limited. Existing benchmarks and leaderboards are predominantly
English-centric, with only a few addressing other languages. These benchmarks
fall short in several key areas: they overlook the diversity of language
varieties, prioritize fundamental Natural Language Processing (NLP)
capabilities over tasks of industrial relevance, and are static. With these
aspects in mind, we present IberBench, a comprehensive and extensible benchmark
designed to assess LLM performance on both fundamental and industry-relevant
NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America.
IberBench integrates 101 datasets from evaluation campaigns and recent
benchmarks, covering 22 task categories such as sentiment and emotion analysis,
toxicity detection, and summarization. The benchmark addresses key limitations
in current evaluation practices, such as the lack of linguistic diversity and
static evaluation setups by enabling continual updates and community-driven
model and dataset submissions moderated by a committee of experts. We evaluate
23 LLMs ranging from 100 million to 14 billion parameters and provide empirical
insights into their strengths and limitations. Our findings indicate that (i)
LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii)
performance is on average lower for Galician and Basque, (iii) some tasks show
results close to random, and (iv) in other tasks LLMs perform above random but
below shared task systems. IberBench offers open-source implementations for the
entire evaluation pipeline, including dataset normalization and hosting,
incremental evaluation of LLMs, and a publicly accessible leaderboard.
680b40774d69b6950c4eabda
José Ángel González
claimed_verified
2025-04-25T12:27:15.281Z
613f7e43e9cc36f11c7715b1
https://cdn-avatars.huggingface.co/v1/production/uploads/1631558612397-613f7e43e9cc36f11c7715b1.png
José Ángel González
user
jogonba2
680b40774d69b6950c4eabdb
Ian Borrego Obrador
680b40774d69b6950c4eabdc
Álvaro Romo Herrero
680b40774d69b6950c4eabdd
Areg Mikael Sarvazyan
680b40774d69b6950c4eabde
Mara Chinea-Ríos
extracted_confirmed
2025-04-25T07:58:32.606Z
60f95c8fda0985b973d59d77
/avatars/5606f0191b9f86e6b55f7e5ab6cc8bb6.svg
Mara Chinea Rios
user
mchinea
680b40774d69b6950c4eabdf
Angelo Basile
680b40774d69b6950c4eabe0
Marc Franco-Salvador
IberBench: LLM Evaluation on Iberian Languages
Large Language Models (LLMs) remain difficult to evaluate comprehensively,
particularly for languages other than English, where high-quality data is often
limited. Existing benchmarks and leaderboards are predominantly
English-centric, with only a few addressing other languages. These benchmarks
fall short in several key areas: they overlook the diversity of language
varieties, prioritize fundamental Natural Language Processing (NLP)
capabilities over tasks of industrial relevance, and are static. With these
aspects in mind, we present IberBench, a comprehensive and extensible benchmark
designed to assess LLM performance on both fundamental and industry-relevant
NLP tasks, in languages spoken across the Iberian Peninsula and Ibero-America.
IberBench integrates 101 datasets from evaluation campaigns and recent
benchmarks, covering 22 task categories such as sentiment and emotion analysis,
toxicity detection, and summarization. The benchmark addresses key limitations
in current evaluation practices, such as the lack of linguistic diversity and
static evaluation setups by enabling continual updates and community-driven
model and dataset submissions moderated by a committee of experts. We evaluate
23 LLMs ranging from 100 million to 14 billion parameters and provide empirical
insights into their strengths and limitations. Our findings indicate that (i)
LLMs perform worse on industry-relevant tasks than in fundamental ones, (ii)
performance is on average lower for Galician and Basque, (iii) some tasks show
results close to random, and (iv) in other tasks LLMs perform above random but
below shared task systems. IberBench offers open-source implementations for the
entire evaluation pipeline, including dataset normalization and hosting,
incremental evaluation of LLMs, and a publicly accessible leaderboard.
IberBench is a comprehensive benchmark for evaluating Large Language Models on diverse tasks across languages spoken in the Iberian Peninsula and Ibero-America, addressing limitations of current evaluation practices.
Large Language Models LLMs IberBench Natural Language Processing NLP sentiment and emotion analysis toxicity detection summarization linguistic diversity dataset normalization