Front-Loading Reasoning: The Synergy between Pretraining and
  Post-Training Data
The prevailing paradigm for enhancing the reasoning abilities of LLMs
revolves around post-training on high-quality, reasoning-intensive data. While
emerging literature suggests that reasoning data is increasingly incorporated
also during the mid-training stage-a practice that is relatively more
proprietary and less openly characterized-the role of such data in pretraining
remains unclear. In particular, due to the opaqueness of pretraining corpora in
most frontier models, the effect of reasoning data introduced at different
phases of pre- and/or post-training is relatively less reported in the
scientific literature. This raises several important questions: Is adding
reasoning data earlier during pretraining any better than introducing it during
post-training? Could earlier inclusion risk overfitting and harm
generalization, or instead establish durable foundations that later fine-tuning
cannot recover? We conduct the first systematic study of how reasoning
data-varying in scale, diversity, and quality-affects LLM performance when
introduced at different stages of training. We find that front-loading
reasoning data into pretraining is critical (19% avg gain), establishing
foundational capabilities that cannot be fully replicated by later-stage SFT,
even with more data. We uncover an asymmetric principle for optimal data
allocation: pretraining benefits most from broad diversity in reasoning
patterns (11% avg gain), while SFT is more sensitive to data quality (15% avg
gain). We show that high-quality pretraining data has latent effects, activated
only after SFT, and that naively scaling SFT data can be detrimental, washing
away the benefits of early reasoning injection. Our results challenge the
conventional separation of language modeling and reasoning, providing a
principled guide for strategically allocating data across the entire training
pipeline to build more capable models.
68e473a0e4e093a7044e4cd7
Syeda Nahida Akter
claimed_verified
2025-10-07T12:27:15.673Z
6338dd1776421c0543150467
/avatars/4539dcec644e40be33f4a0d419fa66cb.svg
Syeda Nahida Akter
user
SieraL
68e473a0e4e093a7044e4cd8
Shrimai Prabhumoye
68e473a0e4e093a7044e4cd9
Eric Nyberg
68e473a0e4e093a7044e4cda
Mostofa Patwary
68e473a0e4e093a7044e4cdb
Mohammad Shoeybi
68e473a0e4e093a7044e4cdc
Yejin Choi
68e473a0e4e093a7044e4cdd
Bryan Catanzaro
Front-Loading Reasoning: The Synergy between Pretraining and
  Post-Training Data
The prevailing paradigm for enhancing the reasoning abilities of LLMs
revolves around post-training on high-quality, reasoning-intensive data. While
emerging literature suggests that reasoning data is increasingly incorporated
also during the mid-training stage-a practice that is relatively more
proprietary and less openly characterized-the role of such data in pretraining
remains unclear. In particular, due to the opaqueness of pretraining corpora in
most frontier models, the effect of reasoning data introduced at different
phases of pre- and/or post-training is relatively less reported in the
scientific literature. This raises several important questions: Is adding
reasoning data earlier during pretraining any better than introducing it during
post-training? Could earlier inclusion risk overfitting and harm
generalization, or instead establish durable foundations that later fine-tuning
cannot recover? We conduct the first systematic study of how reasoning
data-varying in scale, diversity, and quality-affects LLM performance when
introduced at different stages of training. We find that front-loading
reasoning data into pretraining is critical (19% avg gain), establishing
foundational capabilities that cannot be fully replicated by later-stage SFT,
even with more data. We uncover an asymmetric principle for optimal data
allocation: pretraining benefits most from broad diversity in reasoning
patterns (11% avg gain), while SFT is more sensitive to data quality (15% avg
gain). We show that high-quality pretraining data has latent effects, activated
only after SFT, and that naively scaling SFT data can be detrimental, washing
away the benefits of early reasoning injection. Our results challenge the
conventional separation of language modeling and reasoning, providing a
principled guide for strategically allocating data across the entire training
pipeline to build more capable models.
Introducing reasoning data during pretraining significantly enhances LLM performance compared to post-training, with pretraining benefiting more from diverse data patterns while SFT benefits more from high-quality data.
LLMs reasoning data pretraining post-training mid-training overfitting generalization systematic study SFT data allocation language modeling reasoning