RedPajama: an Open Dataset for Training Large Language Models
Large language models are increasingly becoming a cornerstone technology in
artificial intelligence, the sciences, and society as a whole, yet the optimal
strategies for dataset composition and filtering remain largely elusive. Many
of the top-performing models lack transparency in their dataset curation and
model development processes, posing an obstacle to the development of fully
open language models. In this paper, we identify three core data-related
challenges that must be addressed to advance open-source language models. These
include (1) transparency in model development, including the data curation
process, (2) access to large quantities of high-quality data, and (3)
availability of artifacts and metadata for dataset curation and analysis. To
address these challenges, we release RedPajama-V1, an open reproduction of the
LLaMA training dataset. In addition, we release RedPajama-V2, a massive
web-only dataset consisting of raw, unfiltered text data together with quality
signals and metadata. Together, the RedPajama datasets comprise over 100
trillion tokens spanning multiple domains and with their quality signals
facilitate the filtering of data, aiming to inspire the development of numerous
new datasets. To date, these datasets have already been used in the training of
strong language models used in production, such as Snowflake Arctic,
Salesforce's XGen and AI2's OLMo. To provide insight into the quality of
RedPajama, we present a series of analyses and ablation studies with
decoder-only language models with up to 1.6B parameters. Our findings
demonstrate how quality signals for web data can be effectively leveraged to
curate high-quality subsets of the dataset, underscoring the potential of
RedPajama to advance the development of transparent and high-performing
language models at scale.
673db8720916efb6cf79a413
Maurice Weber
admin_assigned
2024-11-20T13:20:10.244Z
6329ee3dab49d487dd1439ec
https://cdn-avatars.huggingface.co/v1/production/uploads/6329ee3dab49d487dd1439ec/vxGvdBK0XMZaCpc5dGOIa.jpeg
Maurice Weber
user
mauriceweber
673db8720916efb6cf79a414
Daniel Fu
673db8720916efb6cf79a415
Quentin Anthony
673db8720916efb6cf79a416
Yonatan Oren
673db8720916efb6cf79a417
Shane Adams
673db8720916efb6cf79a418
Anton Alexandrov
673db8720916efb6cf79a419
Xiaozhong Lyu
admin_assigned
2024-11-20T13:30:20.737Z
6512adc7d4556f9e6508e89f
/avatars/fbae9a9092d6c02ef49e60bc38c6640c.svg
xiaozhong lyu
user
xiaozhong-idata
673db8720916efb6cf79a41a
Huu Nguyen
claimed_verified
2025-03-02T20:19:30.683Z
5fc6879e1c5ee87b1164876d
https://cdn-avatars.huggingface.co/v1/production/uploads/5fc6879e1c5ee87b1164876d/Tjnm_lv0Bq0gPbFOTDH6E.jpeg
Huu Nguyen
user
huu-ontocord
673db8720916efb6cf79a41b
Xiaozhe Yao
admin_assigned
2024-11-20T13:28:26.931Z
61919e59d221f4281b3833d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1669164541734-61919e59d221f4281b3833d5.jpeg
Xiaozhe Yao
user
xzyao
673db8720916efb6cf79a41c
Virginia Adams
admin_assigned
2024-11-20T13:28:43.998Z
664d56756083f276c4197f46
/avatars/64032612a471a0d53f5d04d89617f393.svg
Adams
user
VirginiaAdams
673db8720916efb6cf79a41d
Ben Athiwaratkun
admin_assigned
2024-11-20T13:29:45.831Z
6395124b3d608ea4d9cd2339
/avatars/13f3dc014495dfdef2c460e9ce9c9869.svg
Ben Athiwaratkun
user
benathi
673db8720916efb6cf79a41e
Rahul Chalamala
admin_assigned
2024-11-20T13:30:03.317Z
648abf5f0c2a2d8592ca07e7
/avatars/333a96e0b2e9420b2abacf538c776c18.svg
Rahul Chalamala
user
rchalamala
673db8720916efb6cf79a41f
Kezhen Chen
admin_assigned
2024-11-20T13:30:10.106Z
65caa69b8870f8c093a8abc1
/avatars/1b76dcc6a31b827a5b597024a0f29164.svg
Kezhen Chen
user
kezhentogether
673db8720916efb6cf79a420
Max Ryabinin
admin_assigned
2024-11-20T13:27:41.096Z
607d59fb921db717010c7ccc
https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
Max Ryabinin
user
mryab
673db8720916efb6cf79a421
Tri Dao
admin_assigned
2024-11-20T13:27:47.803Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
673db8720916efb6cf79a422
Percy Liang
admin_assigned
2024-11-20T13:27:53.332Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
673db8720916efb6cf79a423
Christopher Ré
673db8720916efb6cf79a424
Irina Rish
admin_assigned
2024-11-20T13:23:17.710Z
6227f764236b7b2eefa7f173
/avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
Irina Rish
user
irinarish
673db8720916efb6cf79a425
Ce Zhang
RedPajama: an Open Dataset for Training Large Language Models
Large language models are increasingly becoming a cornerstone technology in
artificial intelligence, the sciences, and society as a whole, yet the optimal
strategies for dataset composition and filtering remain largely elusive. Many
of the top-performing models lack transparency in their dataset curation and
model development processes, posing an obstacle to the development of fully
open language models. In this paper, we identify three core data-related
challenges that must be addressed to advance open-source language models. These
include (1) transparency in model development, including the data curation
process, (2) access to large quantities of high-quality data, and (3)
availability of artifacts and metadata for dataset curation and analysis. To
address these challenges, we release RedPajama-V1, an open reproduction of the
LLaMA training dataset. In addition, we release RedPajama-V2, a massive
web-only dataset consisting of raw, unfiltered text data together with quality
signals and metadata. Together, the RedPajama datasets comprise over 100
trillion tokens spanning multiple domains and with their quality signals
facilitate the filtering of data, aiming to inspire the development of numerous
new datasets. To date, these datasets have already been used in the training of
strong language models used in production, such as Snowflake Arctic,
Salesforce's XGen and AI2's OLMo. To provide insight into the quality of
RedPajama, we present a series of analyses and ablation studies with
decoder-only language models with up to 1.6B parameters. Our findings
demonstrate how quality signals for web data can be effectively leveraged to
curate high-quality subsets of the dataset, underscoring the potential of
RedPajama to advance the development of transparent and high-performing
language models at scale.
The RedPajama datasets are introduced to address core challenges for open-source language models by providing transparent data curation, large volumes of high-quality text, and quality signals for web data analysis.
decoder-only language models