Thinking Augmented Pre-training
This paper introduces a simple and scalable approach to improve the data
efficiency of large language model (LLM) training by augmenting existing text
data with thinking trajectories. The compute for pre-training LLMs has been
growing at an unprecedented rate, while the availability of high-quality data
remains limited. Consequently, maximizing the utility of available data
constitutes a significant research challenge. A primary impediment is that
certain high-quality tokens are difficult to learn given a fixed model
capacity, as the underlying rationale for a single token can be exceptionally
complex and deep. To address this issue, we propose Thinking augmented
Pre-Training (TPT), a universal methodology that augments text with
automatically generated thinking trajectories. Such augmentation effectively
increases the volume of the training data and makes high-quality tokens more
learnable through step-by-step reasoning and decomposition. We apply TPT across
diverse training configurations up to 100B tokens, encompassing pre-training
with both constrained and abundant data, as well as mid-training from strong
open-source checkpoints. Experimental results indicate that our method
substantially improves the performance of LLMs across various model sizes and
families. Notably, TPT enhances the data efficiency of LLM pre-training by a
factor of 3. For a 3B parameter model, it improves the post-training
performance by over 10% on several challenging reasoning benchmarks.
68d563258ccd91bdd39ffc79
Liang Wang
admin_assigned
2025-09-26T16:12:45.469Z
62343594c63d91cec1ca4a8d
/avatars/5a1ee74c2dbe349a6ec9843a1599d281.svg
Liang Wang
user
intfloat
68d563258ccd91bdd39ffc7a
Nan Yang
68d563258ccd91bdd39ffc7b
Shaohan Huang
admin_assigned
2025-09-26T16:13:06.086Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
68d563258ccd91bdd39ffc7c
Li Dong
68d563258ccd91bdd39ffc7d
Furu Wei
admin_assigned
2025-09-26T16:13:12.700Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Thinking Augmented Pre-training
This paper introduces a simple and scalable approach to improve the data
efficiency of large language model (LLM) training by augmenting existing text
data with thinking trajectories. The compute for pre-training LLMs has been
growing at an unprecedented rate, while the availability of high-quality data
remains limited. Consequently, maximizing the utility of available data
constitutes a significant research challenge. A primary impediment is that
certain high-quality tokens are difficult to learn given a fixed model
capacity, as the underlying rationale for a single token can be exceptionally
complex and deep. To address this issue, we propose Thinking augmented
Pre-Training (TPT), a universal methodology that augments text with
automatically generated thinking trajectories. Such augmentation effectively
increases the volume of the training data and makes high-quality tokens more
learnable through step-by-step reasoning and decomposition. We apply TPT across
diverse training configurations up to 100B tokens, encompassing pre-training
with both constrained and abundant data, as well as mid-training from strong
open-source checkpoints. Experimental results indicate that our method
substantially improves the performance of LLMs across various model sizes and
families. Notably, TPT enhances the data efficiency of LLM pre-training by a
factor of 3. For a 3B parameter model, it improves the post-training
performance by over 10% on several challenging reasoning benchmarks.
Thinking augmented pre-training improves data efficiency and performance of large language models by augmenting text with automatically generated thinking trajectories.
large language model thinking trajectories pre-training data efficiency high-quality tokens step-by-step reasoning decomposition constrained data abundant data mid-training open-source checkpoints reasoning benchmarks