MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus
  Expansion
Despite the remarkable capabilities of large language models across various
tasks, their continued scaling faces a critical challenge: the scarcity of
high-quality pretraining data. While model architectures continue to evolve,
the natural language data struggles to scale up. To tackle this bottleneck, we
propose MAssive Genre-Audience~(MAGA) reformulation
method, which systematic synthesizes diverse, contextually-rich pretraining
data from existing corpus. This work makes three main contributions: (1) We
propose MAGA reformulation method, a lightweight and scalable approach for
pretraining corpus expansion, and build a 770B tokens MAGACorpus. (2) We
evaluate MAGACorpus with different data budget scaling strategies,
demonstrating consistent improvements across various model sizes (134M-13B),
establishing the necessity for next-generation large-scale synthetic
pretraining language models. (3) Through comprehensive analysis, we investigate
prompt engineering's impact on synthetic training collapse and reveal
limitations in conventional collapse detection metrics using validation losses.
Our work shows that MAGA can substantially expand training datasets while
maintaining quality, offering a reliably pathway for scaling models beyond data
limitations.
67a56af6d7c26c7497a86308
Xintong Hao
extracted_confirmed
2025-02-07T04:41:11.249Z
64b764bffdb702b3d8640610
https://cdn-avatars.huggingface.co/v1/production/uploads/64b764bffdb702b3d8640610/lpHg0AX_NOmzw-ZxeOa1s.png
haoxintong
user
haoxintong
67a56af6d7c26c7497a86309
Ke Shen
claimed_verified
2025-02-07T12:56:30.330Z
645604eebabbbbd3486dc615
/avatars/17a5ca8274e2bfc8f183a4af9878a930.svg
shenke
user
shenke18
67a56af6d7c26c7497a8630a
Chenggang Li
MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus
  Expansion
Despite the remarkable capabilities of large language models across various
tasks, their continued scaling faces a critical challenge: the scarcity of
high-quality pretraining data. While model architectures continue to evolve,
the natural language data struggles to scale up. To tackle this bottleneck, we
propose MAssive Genre-Audience~(MAGA) reformulation
method, which systematic synthesizes diverse, contextually-rich pretraining
data from existing corpus. This work makes three main contributions: (1) We
propose MAGA reformulation method, a lightweight and scalable approach for
pretraining corpus expansion, and build a 770B tokens MAGACorpus. (2) We
evaluate MAGACorpus with different data budget scaling strategies,
demonstrating consistent improvements across various model sizes (134M-13B),
establishing the necessity for next-generation large-scale synthetic
pretraining language models. (3) Through comprehensive analysis, we investigate
prompt engineering's impact on synthetic training collapse and reveal
limitations in conventional collapse detection metrics using validation losses.
Our work shows that MAGA can substantially expand training datasets while
maintaining quality, offering a reliably pathway for scaling models beyond data
limitations.
MAGA reformulation method expands pretraining datasets by synthesizing diverse and contextually-rich data, improving model performance across various sizes and highlighting limitations in collapse detection metrics.
MAssive Genre-Audience MAGA reformulation method MAGACorpus pretraining corpus expansion data budget scaling strategies prompt engineering synthetic training collapse collapse detection metrics