Programming Every Example: Lifting Pre-training Data Quality like
  Experts at Scale
Large language model pre-training has traditionally relied on human experts
to craft heuristics for improving the corpora quality, resulting in numerous
rules developed to date. However, these rules lack the flexibility to address
the unique characteristics of individual example effectively. Meanwhile,
applying tailored rules to every example is impractical for human experts. In
this paper, we demonstrate that even small language models, with as few as 0.3B
parameters, can exhibit substantial data refining capabilities comparable to
those of human experts. We introduce Programming Every Example (ProX), a novel
framework that treats data refinement as a programming task, enabling models to
refine corpora by generating and executing fine-grained operations, such as
string normalization, for each individual example at scale. Experimental
results show that models pre-trained on ProX-curated data outperform either
original data or data filtered by other selection methods by more than 2%
across various downstream benchmarks. Its effectiveness spans various model
sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb.
Furthermore, ProX exhibits significant potential in domain-specific continual
pre-training: without domain specific design, models trained on OpenWebMath
refined by ProX outperform human-crafted rule-based methods, improving average
accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for
CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B
trained on 200B tokens. Further analysis highlights that ProX significantly
saves training FLOPs, offering a promising path for efficient LLM
pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing
all training and implementation details for reproducible research and future
innovation. Code: https://github.com/GAIR-NLP/ProX
66f4c2f7529e23b71ff72e1e
Fan Zhou
claimed_verified
2024-09-26T08:23:08.164Z
628f6e5ab90dde28ef57d293
https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
Fan Zhou
user
koalazf99
66f4c2f7529e23b71ff72e1f
Zengzhi Wang
admin_assigned
2024-09-26T08:41:23.506Z
62cbeb2d72dfd24b86bdf977
https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
Zengzhi Wang
user
SinclairWang
66f4c2f7529e23b71ff72e20
Qian Liu
claimed_verified
2024-09-30T08:07:20.519Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
66f4c2f7529e23b71ff72e21
Junlong Li
admin_assigned
2024-09-26T08:42:06.216Z
621e40ac944c7e36aaec2369
https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
Junlong Li
user
lockon
66f4c2f7529e23b71ff72e22
Pengfei Liu
admin_assigned
2024-09-26T08:42:23.621Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
Programming Every Example: Lifting Pre-training Data Quality like
  Experts at Scale
Large language model pre-training has traditionally relied on human experts
to craft heuristics for improving the corpora quality, resulting in numerous
rules developed to date. However, these rules lack the flexibility to address
the unique characteristics of individual example effectively. Meanwhile,
applying tailored rules to every example is impractical for human experts. In
this paper, we demonstrate that even small language models, with as few as 0.3B
parameters, can exhibit substantial data refining capabilities comparable to
those of human experts. We introduce Programming Every Example (ProX), a novel
framework that treats data refinement as a programming task, enabling models to
refine corpora by generating and executing fine-grained operations, such as
string normalization, for each individual example at scale. Experimental
results show that models pre-trained on ProX-curated data outperform either
original data or data filtered by other selection methods by more than 2%
across various downstream benchmarks. Its effectiveness spans various model
sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb.
Furthermore, ProX exhibits significant potential in domain-specific continual
pre-training: without domain specific design, models trained on OpenWebMath
refined by ProX outperform human-crafted rule-based methods, improving average
accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for
CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B
trained on 200B tokens. Further analysis highlights that ProX significantly
saves training FLOPs, offering a promising path for efficient LLM
pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing
all training and implementation details for reproducible research and future
innovation. Code: https://github.com/GAIR-NLP/ProX
ProX, a novel framework treating data refinement as a programming task, enhances large language model pre-training by generating fine-grained operations for each example, outperforming traditional human-crafted rule-based methods significantly across various benchmarks and domains.
Programming Every Example (ProX) data refinement string normalization LLLM pre-training C4 RedPajama-V2 FineWeb domain-specific continual pre-training OpenWebMath Mistral-7B Llama-2-7B CodeLlama-7B training FLOPs