Synthetic bootstrapped pretraining
We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM)
pretraining procedure that first learns a model of relations between documents
from the pretraining dataset and then leverages it to synthesize a vast new
corpus for joint training. While the standard pretraining teaches LMs to learn
causal correlations among tokens within a single document, it is not designed
to efficiently model the rich, learnable inter-document correlations that can
potentially lead to better performance. We validate SBP by designing a
compute-matched pretraining setup and pretrain a 3B-parameter model on up to 1T
tokens from scratch. We find SBP consistently improves upon a strong repetition
baseline and delivers a significant fraction of performance improvement
attainable by an oracle upper bound with access to 20x more unique data.
Qualitative analysis reveals that the synthesized documents go beyond mere
paraphrases -- SBP first abstracts a core concept from the seed material and
then crafts a new narration on top of it. Besides strong empirical performance,
SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns
to abstract the latent concepts shared between related documents.
68d209c41ca7156988a8ecd4
Zitong Yang
68d209c41ca7156988a8ecd5
Aonan Zhang
68d209c41ca7156988a8ecd6
Hong Liu
68d209c41ca7156988a8ecd7
Tatsunori Hashimoto
68d209c41ca7156988a8ecd8
Emmanuel Candès
68d209c41ca7156988a8ecd9
Chong Wang
68d209c41ca7156988a8ecda
Ruoming Pang
Synthetic bootstrapped pretraining
We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM)
pretraining procedure that first learns a model of relations between documents
from the pretraining dataset and then leverages it to synthesize a vast new
corpus for joint training. While the standard pretraining teaches LMs to learn
causal correlations among tokens within a single document, it is not designed
to efficiently model the rich, learnable inter-document correlations that can
potentially lead to better performance. We validate SBP by designing a
compute-matched pretraining setup and pretrain a 3B-parameter model on up to 1T
tokens from scratch. We find SBP consistently improves upon a strong repetition
baseline and delivers a significant fraction of performance improvement
attainable by an oracle upper bound with access to 20x more unique data.
Qualitative analysis reveals that the synthesized documents go beyond mere
paraphrases -- SBP first abstracts a core concept from the seed material and
then crafts a new narration on top of it. Besides strong empirical performance,
SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns
to abstract the latent concepts shared between related documents.
Synthetic Bootstrapped Pretraining (SBP) enhances language model performance by learning inter-document correlations and synthesizing new training data, leading to significant improvements over standard pretraining methods.
Synthetic Bootstrapped Pretraining SBP language model pretraining procedure inter-document correlations causal correlations Bayesian interpretation latent concepts