The German Commons - 154 Billion Tokens of Openly Licensed Text for
  German Language Models
Large language model development relies on large-scale training corpora, yet
most contain data of unclear licensing status, limiting the development of
truly open models. This problem is exacerbated for non-English languages, where
openly licensed text remains critically scarce. We introduce the German
Commons, the largest collection of openly licensed German text to date. It
compiles data from 41 sources across seven domains, encompassing legal,
scientific, cultural, political, news, economic, and web text. Through
systematic sourcing from established data providers with verifiable licensing,
it yields 154.56 billion tokens of high-quality text for language model
training. Our processing pipeline implements comprehensive quality filtering,
deduplication, and text formatting fixes, ensuring consistent quality across
heterogeneous text sources. All domain subsets feature licenses of at least
CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and
redistribution. The German Commons therefore addresses the critical gap in
openly licensed German pretraining data, and enables the development of truly
open German language models. We also release code for corpus construction and
data filtering tailored to German language text, rendering the German Commons
fully reproducible and extensible.
68f1d4a56e0bef323a68ff90
Lukas Gienapp
68f1d4a56e0bef323a68ff91
Christopher Schröder
68f1d4a56e0bef323a68ff92
Stefan Schweter
claimed_verified
2025-10-17T05:56:03.613Z
5e6a3d4ea9afd5125d9ec064
https://cdn-avatars.huggingface.co/v1/production/uploads/1584020801691-noauth.jpeg
Stefan Schweter
user
stefan-it
68f1d4a56e0bef323a68ff93
Christopher Akiki
claimed_verified
2025-10-18T04:49:04.109Z
5e70f6048ce3c604d78fe133
https://cdn-avatars.huggingface.co/v1/production/uploads/1646492542174-5e70f6048ce3c604d78fe133.jpeg
Christopher Akiki
user
christopher
68f1d4a56e0bef323a68ff94
Ferdinand Schlatt
68f1d4a56e0bef323a68ff95
Arden Zimmermann
68f1d4a56e0bef323a68ff96
Phillipe Genêt
68f1d4a56e0bef323a68ff97
Martin Potthast
The German Commons - 154 Billion Tokens of Openly Licensed Text for
  German Language Models
Large language model development relies on large-scale training corpora, yet
most contain data of unclear licensing status, limiting the development of
truly open models. This problem is exacerbated for non-English languages, where
openly licensed text remains critically scarce. We introduce the German
Commons, the largest collection of openly licensed German text to date. It
compiles data from 41 sources across seven domains, encompassing legal,
scientific, cultural, political, news, economic, and web text. Through
systematic sourcing from established data providers with verifiable licensing,
it yields 154.56 billion tokens of high-quality text for language model
training. Our processing pipeline implements comprehensive quality filtering,
deduplication, and text formatting fixes, ensuring consistent quality across
heterogeneous text sources. All domain subsets feature licenses of at least
CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and
redistribution. The German Commons therefore addresses the critical gap in
openly licensed German pretraining data, and enables the development of truly
open German language models. We also release code for corpus construction and
data filtering tailored to German language text, rendering the German Commons
fully reproducible and extensible.
The German Commons provides a large-scale, openly licensed dataset for training German language models, addressing the scarcity of such data.
language model training corpora licensing status openly licensed text German Commons data sources legal scientific cultural political news economic web text high-quality text quality filtering deduplication text formatting CC-BY-SA 4.0 legal compliance model training redistribution corpus construction data filtering