RePro: Training Language Models to Faithfully Recycle the Web for
  Pretraining
High-quality pretraining data is the fossil fuel of large language models
(LLMs), yet its reserves are running low for frontier models. In this paper, we
introduce RePro, a novel web recycling method that trains a relatively small LM
with reinforcement learning to generate effective and faithful rephrasings of
pretraining data. Specifically, we design one quality reward and three
faithfulness rewards, optimizing the LM rephraser to convert organic data into
high-quality rephrasings while maintaining its core semantics and structure. In
our experiment, we train a 4B rephraser to recycle 72B tokens sampled from
DCLM-RefinedWeb. Pretraining results on 400M and 1.4B models demonstrate that
RePro delivers 4.7%-14.0% relative accuracy gains over organic-only baseline on
22 downstream tasks. RePro also outperforms ReWire, the state-of-the-art web
recycling method that prompts a 70B rephraser, as well as the organic baseline
with a 4x larger data pool. Experiments with different amounts of recycled data
highlight that RePro improves organic data efficiency by 2-3x. Individual and
distributional analyses validate that RePro preserves more critical information
and faithfully reflects the characteristics of organic data compared to
prompting-based methods. Together, these results show that RePro provides an
efficient and controllable path to effectively harness the fossil fuel of LLM
pretraining. We open-source our code, rephraser, and recycled data at
https://github.com/cxcscmu/RePro.
68edc950de1fee572713a915
Zichun Yu
68edc950de1fee572713a916
Chenyan Xiong
RePro: Training Language Models to Faithfully Recycle the Web for
  Pretraining
High-quality pretraining data is the fossil fuel of large language models
(LLMs), yet its reserves are running low for frontier models. In this paper, we
introduce RePro, a novel web recycling method that trains a relatively small LM
with reinforcement learning to generate effective and faithful rephrasings of
pretraining data. Specifically, we design one quality reward and three
faithfulness rewards, optimizing the LM rephraser to convert organic data into
high-quality rephrasings while maintaining its core semantics and structure. In
our experiment, we train a 4B rephraser to recycle 72B tokens sampled from
DCLM-RefinedWeb. Pretraining results on 400M and 1.4B models demonstrate that
RePro delivers 4.7%-14.0% relative accuracy gains over organic-only baseline on
22 downstream tasks. RePro also outperforms ReWire, the state-of-the-art web
recycling method that prompts a 70B rephraser, as well as the organic baseline
with a 4x larger data pool. Experiments with different amounts of recycled data
highlight that RePro improves organic data efficiency by 2-3x. Individual and
distributional analyses validate that RePro preserves more critical information
and faithfully reflects the characteristics of organic data compared to
prompting-based methods. Together, these results show that RePro provides an
efficient and controllable path to effectively harness the fossil fuel of LLM
pretraining. We open-source our code, rephraser, and recycled data at
https://github.com/cxcscmu/RePro.
RePro, a reinforcement learning-based method, generates high-quality rephrasings of pretraining data to enhance the efficiency and accuracy of large language models.
reinforcement learning rephraser quality reward faithfulness rewards pretraining data DCLM-RefinedWeb downstream tasks ReWire data efficiency critical information preservation