In-Context Pretraining: Language Modeling Beyond Document Boundaries
Large language models (LMs) are currently trained to predict tokens given
document prefixes, enabling them to directly perform long-form generation and
prompting-style tasks which can be reduced to document completion. Existing
pretraining pipelines train LMs by concatenating random sets of short documents
to create input contexts but the prior documents provide no signal for
predicting the next document. We instead present In-Context Pretraining, a new
approach where language models are pretrained on a sequence of related
documents, thereby explicitly encouraging them to read and reason across
document boundaries. We can do In-Context Pretraining by simply changing the
document ordering so that each context contains related documents, and directly
applying existing pretraining pipelines. However, this document sorting problem
is challenging. There are billions of documents and we would like the sort to
maximize contextual similarity for every document without repeating any data.
To do this, we introduce approximate algorithms for finding related documents
with efficient nearest neighbor search and constructing coherent input contexts
with a graph traversal algorithm. Our experiments show In-Context Pretraining
offers a simple and scalable approach to significantly enhance LMs'performance:
we see notable improvements in tasks that require more complex contextual
reasoning, including in-context learning (+8%), reading comprehension (+15%),
faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
retrieval augmentation (+9%).
652e0cceb86e108d0ff3c2aa
Weijia Shi
admin_assigned
2023-10-17T07:32:01.655Z
6400f2ed568dbe30c9161e47
/avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
Weijia Shi
user
swj0419
652e0cceb86e108d0ff3c2ab
Sewon Min
admin_assigned
2023-10-17T07:32:08.918Z
63a76d0de27a6dbd485fe863
https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
Sewon Min
user
sewon
652e0cceb86e108d0ff3c2ac
Maria Lomeli
admin_assigned
2023-10-17T07:32:25.253Z
64f86587e7584abc62545b3e
/avatars/57296279592ad5f77c39d174f6289420.svg
Maria Lomeli
user
marialomeli
652e0cceb86e108d0ff3c2ad
Chunting Zhou
admin_assigned
2023-10-17T07:32:35.939Z
64f8d07dd493d8b0d2c04ecd
/avatars/7112232dcadcd2a7c12ec635d781b3ea.svg
Zhou
user
Chunting
652e0cceb86e108d0ff3c2ae
Margaret Li
admin_assigned
2023-10-17T07:32:54.740Z
64bf0bcfc05a0df0d2d09d5c
/avatars/d24bfb4d6b3d9815e68cca710639ab87.svg
Margaret Li
user
margsli
652e0cceb86e108d0ff3c2af
Victoria Lin
admin_assigned
2023-10-17T07:33:10.059Z
61df349e34b589fbaf2e93c4
/avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
Victoria Lin
user
VictoriaLinML
652e0cceb86e108d0ff3c2b0
Noah A. Smith
652e0cceb86e108d0ff3c2b1
Luke Zettlemoyer
652e0cceb86e108d0ff3c2b2
Scott Yih
652e0cceb86e108d0ff3c2b3
Mike Lewis
admin_assigned
2023-10-17T07:34:54.174Z
5e67eda71009063689407484
/avatars/6d7688faba888d463584eaa02ad011e3.svg
Mike Lewis
user
mikelewis0
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Large language models (LMs) are currently trained to predict tokens given
document prefixes, enabling them to directly perform long-form generation and
prompting-style tasks which can be reduced to document completion. Existing
pretraining pipelines train LMs by concatenating random sets of short documents
to create input contexts but the prior documents provide no signal for
predicting the next document. We instead present In-Context Pretraining, a new
approach where language models are pretrained on a sequence of related
documents, thereby explicitly encouraging them to read and reason across
document boundaries. We can do In-Context Pretraining by simply changing the
document ordering so that each context contains related documents, and directly
applying existing pretraining pipelines. However, this document sorting problem
is challenging. There are billions of documents and we would like the sort to
maximize contextual similarity for every document without repeating any data.
To do this, we introduce approximate algorithms for finding related documents
with efficient nearest neighbor search and constructing coherent input contexts
with a graph traversal algorithm. Our experiments show In-Context Pretraining
offers a simple and scalable approach to significantly enhance LMs'performance:
we see notable improvements in tasks that require more complex contextual
reasoning, including in-context learning (+8%), reading comprehension (+15%),
faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
retrieval augmentation (+9%).
In-Context Pretraining improves large language models by ordering documents to maximize contextual similarity, leading to better performance in complex reasoning tasks.
large language models (LMs) token prediction long-form generation prompting-style tasks document prefixes document completion pretraining pipelines In-Context Pretraining related documents contextual similarity nearest neighbor search graph traversal algorithm in-context learning reading comprehension faithfulness long-context reasoning retrieval augmentation