Llama-GENBA-10B: A Trilingual Large Language Model for German, English
  and Bavarian
We present Llama-GENBA-10B, a trilingual foundation model addressing
English-centric bias in large language models. Built on Llama 3.1-8B and scaled
to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens
(82B English, 82B German, and 80M Bavarian), balancing resources while
preventing English dominance. Targeted at the German NLP community, the model
also promotes Bavarian as a low-resource language. Development tackled four
challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2)
creating a unified tokenizer for English, German, and Bavarian, (3) optimizing
architecture and language-ratio hyperparameters for cross-lingual transfer, and
(4) establishing the first standardized trilingual evaluation suite by
translating German benchmarks into Bavarian. Evaluations show that
Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned
variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing
itself as the best model in its class for this language, while also
outperforming EuroLLM in English and matching its results in German. Training
on the Cerebras CS-2 demonstrated efficient large-scale multilingual
pretraining with documented energy use, offering a blueprint for inclusive
foundation models that integrate low-resource languages.
68bf9108207285de11b07b62
Michael Hoffmann
68bf9108207285de11b07b63
Jophin John
68bf9108207285de11b07b64
Stefan Schweter
claimed_verified
2025-09-09T13:48:14.824Z
5e6a3d4ea9afd5125d9ec064
https://cdn-avatars.huggingface.co/v1/production/uploads/1584020801691-noauth.jpeg
Stefan Schweter
user
stefan-it
68bf9108207285de11b07b65
Gokul Ramakrishnan
68bf9108207285de11b07b66
Hoi-Fong Mak
68bf9108207285de11b07b67
Alice Zhang
68bf9108207285de11b07b68
Dmitry Gaynullin
claimed_verified
2025-09-17T13:08:38.581Z
646e940fcb6ea6e6b6e042ff
/avatars/1f0f21908b05e5cd165e44f07fdd8c09.svg
Dmitry Gaynullin
user
gaydmi
68bf9108207285de11b07b69
Nicolay J. Hammer
Llama-GENBA-10B: A Trilingual Large Language Model for German, English
  and Bavarian
We present Llama-GENBA-10B, a trilingual foundation model addressing
English-centric bias in large language models. Built on Llama 3.1-8B and scaled
to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens
(82B English, 82B German, and 80M Bavarian), balancing resources while
preventing English dominance. Targeted at the German NLP community, the model
also promotes Bavarian as a low-resource language. Development tackled four
challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2)
creating a unified tokenizer for English, German, and Bavarian, (3) optimizing
architecture and language-ratio hyperparameters for cross-lingual transfer, and
(4) establishing the first standardized trilingual evaluation suite by
translating German benchmarks into Bavarian. Evaluations show that
Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned
variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing
itself as the best model in its class for this language, while also
outperforming EuroLLM in English and matching its results in German. Training
on the Cerebras CS-2 demonstrated efficient large-scale multilingual
pretraining with documented energy use, offering a blueprint for inclusive
foundation models that integrate low-resource languages.
Llama-GENBA-10B, a trilingual foundation model, addresses English-centric bias by balancing English, German, and Bavarian training, achieving strong cross-lingual performance and setting new benchmarks for Bavarian.
trilingual foundation model Llama 3.1-8B Llama-GENBA-10B multilingual corpus unified tokenizer cross-lingual transfer trilingual evaluation suite fine-tuned variant Apertus-8B-2509 gemma-2-9b EuroLLM Cerebras CS-2 large-scale multilingual pretraining