Data Engineering for Scaling Language Models to 128K Context
We study the continual pretraining recipe for scaling language models'
context lengths to 128K, with a focus on data engineering. We hypothesize that
long context modeling, in particular the ability to utilize information
at arbitrary input locations, is a capability that is mostly already acquired
through large-scale pretraining, and that this capability can be readily
extended to contexts substantially longer than seen during training~(e.g., 4K
to 128K) through lightweight continual pretraining on appropriate data mixture.
We investigate the quantity and quality of the data for
continual pretraining: (1) for quantity, we show that 500 million to 5 billion
tokens are enough to enable the model to retrieve information anywhere within
the 128K context; (2) for quality, our results equally emphasize domain
balance and length upsampling. Concretely, we find that naively
upsampling longer data on certain domains like books, a common practice of
existing work, gives suboptimal performance, and that a balanced domain mixture
is important. We demonstrate that continual pretraining of the full model on
1B-5B tokens of such data is an effective and affordable strategy for scaling
the context length of language models to 128K. Our recipe outperforms strong
open-source long-context models and closes the gap to frontier models like
GPT-4 128K.
65ced3ba2a811b35593b217b
Yao Fu
admin_assigned
2024-02-16T14:29:21.605Z
636c929d5ef5a1a07db4e785
/avatars/1bb48dbba964f7fa84036586032684cf.svg
Yao Fu
user
yaofu
65ced3ba2a811b35593b217c
Rameswar Panda
admin_assigned
2024-02-16T14:29:28.707Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
65ced3ba2a811b35593b217d
Xinyao Niu
admin_assigned
2024-02-16T14:29:34.558Z
627f4c100d2c1c0ba3da8a72
/avatars/404869a87480ef526c1d5fccb583bfa8.svg
Xinyao Niu
user
sirius-ctrl
65ced3ba2a811b35593b217e
Xiang Yue
admin_assigned
2024-02-16T14:29:59.838Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
65ced3ba2a811b35593b217f
Hannaneh Hajishirzi
65ced3ba2a811b35593b2180
Yoon Kim
extracted_pending
2024-02-16T03:17:14.629Z
630ee22c05b5c5280e3cdcdf
/avatars/b282162d6d9ae8512873eba0275f603e.svg
Yoon Kim
user
yoon-kim
65ced3ba2a811b35593b2181
Hao Peng
extracted_pending
2024-04-04T15:24:18.817Z
660ec5a2509153ca49775a7c
/avatars/97570fc245cc8ec7628da9c13bd35b71.svg
Hao Peng
user
haopeng01
Data Engineering for Scaling Language Models to 128K Context
We study the continual pretraining recipe for scaling language models'
context lengths to 128K, with a focus on data engineering. We hypothesize that
long context modeling, in particular the ability to utilize information
at arbitrary input locations, is a capability that is mostly already acquired
through large-scale pretraining, and that this capability can be readily
extended to contexts substantially longer than seen during training~(e.g., 4K
to 128K) through lightweight continual pretraining on appropriate data mixture.
We investigate the quantity and quality of the data for
continual pretraining: (1) for quantity, we show that 500 million to 5 billion
tokens are enough to enable the model to retrieve information anywhere within
the 128K context; (2) for quality, our results equally emphasize domain
balance and length upsampling. Concretely, we find that naively
upsampling longer data on certain domains like books, a common practice of
existing work, gives suboptimal performance, and that a balanced domain mixture
is important. We demonstrate that continual pretraining of the full model on
1B-5B tokens of such data is an effective and affordable strategy for scaling
the context length of language models to 128K. Our recipe outperforms strong
open-source long-context models and closes the gap to frontier models like
GPT-4 128K.
Continual pretraining with domain-balanced and length-upsampled data effectively scales language models' context length to 128K without significant additional computational cost.
continual pretraining context lengths long context modeling data engineering data mixture domain balance length upsampling language models