The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly
  Licensed Text
Large language models (LLMs) are typically trained on enormous quantities of
unlicensed text, a practice that has led to scrutiny due to possible
intellectual property infringement and ethical concerns. Training LLMs on
openly licensed text presents a first step towards addressing these issues, but
prior data collection efforts have yielded datasets too small or low-quality to
produce performant LLMs. To address this gap, we collect, curate, and release
the Common Pile v0.1, an eight terabyte collection of openly licensed text
designed for LLM pretraining. The Common Pile comprises content from 30 sources
that span diverse domains including research papers, code, books,
encyclopedias, educational materials, audio transcripts, and more. Crucially,
we validate our efforts by training two 7 billion parameter LLMs on text from
the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion
tokens respectively. Both models attain competitive performance to LLMs trained
on unlicensed text with similar computational budgets, such as Llama 1 and 2
7B. In addition to releasing the Common Pile v0.1 itself, we also release the
code used in its creation as well as the training mixture and checkpoints for
the Comma v0.1 models.
684247f35d537e0e5ecb724b
Nikhil Kandpal
684247f35d537e0e5ecb724c
Brian Lester
684247f35d537e0e5ecb724d
Colin Raffel
claimed_verified
2025-06-07T05:49:07.948Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
684247f35d537e0e5ecb724e
Sebastian Majstorovic
claimed_verified
2025-06-06T07:41:37.270Z
636071759ddc44e710e0f5ce
https://cdn-avatars.huggingface.co/v1/production/uploads/636071759ddc44e710e0f5ce/HMIw2KzYN66vTzz-e3k3V.jpeg
Sebastian Majstorovic
user
storytracer
684247f35d537e0e5ecb724f
Stella Biderman
claimed_verified
2025-06-07T05:48:59.510Z
60347d3660e3dd96631c9093
https://cdn-avatars.huggingface.co/v1/production/uploads/60347d3660e3dd96631c9093/B3fuZer5N04tZIAYrLnz4.jpeg
Stella Biderman
user
stellaathena
684247f35d537e0e5ecb7250
Baber Abbasi
684247f35d537e0e5ecb7251
Luca Soldaini
684247f35d537e0e5ecb7252
Enrico Shippole
claimed_verified
2025-06-23T08:31:38.409Z
6276ba3c2d26ac639e5a2b01
https://cdn-avatars.huggingface.co/v1/production/uploads/6276ba3c2d26ac639e5a2b01/ULDyT9ijOye7rfXqTWFVG.jpeg
Enrico Shippole
user
conceptofmind
684247f35d537e0e5ecb7253
A. Feder Cooper
684247f35d537e0e5ecb7254
Aviya Skowron
684247f35d537e0e5ecb7255
John Kirchenbauer
684247f35d537e0e5ecb7256
Shayne Longpre
684247f35d537e0e5ecb7257
Lintang Sutawika
684247f35d537e0e5ecb7258
Alon Albalak
claimed_verified
2025-06-07T05:49:05.890Z
611a7ec4289467cafea62d13
https://cdn-avatars.huggingface.co/v1/production/uploads/611a7ec4289467cafea62d13/pck-0fmPQkoU7yzh6-WoL.jpeg
Alon Albalak
user
alon-albalak
684247f35d537e0e5ecb7259
Zhenlin Xu
claimed_verified
2025-06-16T09:52:47.853Z
63791b5f438c92a3c129c007
/avatars/289f4b816bf6880d762cdbdc6246eaab.svg
Zhenlin Xu
user
wildphoton
684247f35d537e0e5ecb725a
Guilherme Penedo
claimed_verified
2025-06-07T05:49:10.010Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
684247f35d537e0e5ecb725b
Loubna Ben Allal
claimed_verified
2025-06-07T13:33:58.334Z
61c141342aac764ce1654e43
https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
Loubna Ben Allal
user
loubnabnl
684247f35d537e0e5ecb725c
Elie Bakouch
claimed_verified
2025-06-07T05:49:01.927Z
651e96991b97c9f33d26bde6
https://cdn-avatars.huggingface.co/v1/production/uploads/651e96991b97c9f33d26bde6/-Bqs6qrmz0yCfwtB2e-6q.jpeg
Elie Bakouch
user
eliebak
684247f35d537e0e5ecb725d
John David Pressman
684247f35d537e0e5ecb725e
Honglu Fan
684247f35d537e0e5ecb725f
Dashiell Stander
684247f35d537e0e5ecb7260
Guangyu Song
684247f35d537e0e5ecb7261
Aaron Gokaslan
684247f35d537e0e5ecb7262
Tom Goldstein
684247f35d537e0e5ecb7263
Brian R. Bartoldson
684247f35d537e0e5ecb7264
Bhavya Kailkhura
claimed_verified
2025-06-10T09:22:31.905Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
684247f35d537e0e5ecb7265
Tyler Murray
claimed_verified
2025-06-07T05:49:03.957Z
65e5fefbe8cbae176d9ca005
https://cdn-avatars.huggingface.co/v1/production/uploads/65e5fefbe8cbae176d9ca005/5SLREDsVzycEwVsPNv765.jpeg
Tyler Murray
user
undfined
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly
  Licensed Text
Large language models (LLMs) are typically trained on enormous quantities of
unlicensed text, a practice that has led to scrutiny due to possible
intellectual property infringement and ethical concerns. Training LLMs on
openly licensed text presents a first step towards addressing these issues, but
prior data collection efforts have yielded datasets too small or low-quality to
produce performant LLMs. To address this gap, we collect, curate, and release
the Common Pile v0.1, an eight terabyte collection of openly licensed text
designed for LLM pretraining. The Common Pile comprises content from 30 sources
that span diverse domains including research papers, code, books,
encyclopedias, educational materials, audio transcripts, and more. Crucially,
we validate our efforts by training two 7 billion parameter LLMs on text from
the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion
tokens respectively. Both models attain competitive performance to LLMs trained
on unlicensed text with similar computational budgets, such as Llama 1 and 2
7B. In addition to releasing the Common Pile v0.1 itself, we also release the
code used in its creation as well as the training mixture and checkpoints for
the Comma v0.1 models.
The Common Pile v0.1 dataset, an 8-terabyte collection of openly licensed text, is used to train competitive 7 billion parameter LLMs.
Large language models LLMs openly licensed text Common Pile v0.1 parameter-efficient fine-tuning Llama 1 and 2 7B training mixture checkpoints