Data Engineering for Scaling Language Models to 128K Context
We study the continual pretraining recipe for scaling language models'
context lengths to 128K, with a focus on data engineering. We hypothesize that
long context modeling, in particular the ability to utilize information
at arbitrary input locations, is a capability that is mostly already acquired
through large-scale pretraining, and that this capability can be readily
extended to contexts substantially longer than seen during training~(e.g., 4K
to 128K) through lightweight continual pretraining on appropriate data mixture.
We investigate the quantity and quality of the data for
continual pretraining: (1) for quantity, we show that 500 million to 5 billion
tokens are enough to enable the model to retrieve information anywhere within
the 128K context; (2) for quality, our results equally emphasize domain
balance and length upsampling. Concretely, we find that naively
upsampling longer data on certain domains like books, a common practice of
existing work, gives suboptimal performance, and that a balanced domain mixture
is important. We demonstrate that continual pretraining of the full model on
1B-5B tokens of such data is an effective and affordable strategy for scaling
the context length of language models to 128K. Our recipe outperforms strong
open-source long-context models and closes the gap to frontier models like
GPT-4 128K.
65ced3ba2a811b35593b217b
Yao Fu
admin_assigned
2024-02-16T14:29:21.605Z
636c929d5ef5a1a07db4e785
/avatars/1bb48dbba964f7fa84036586032684cf.svg
Yao Fu
user
yaofu
65ced3ba2a811b35593b217c
Rameswar Panda
admin_assigned
2024-02-16T14:29:28.707Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
65ced3ba2a811b35593b217d
Xinyao Niu
admin_assigned
2024-02-16T14:29:34.558Z
627f4c100d2c1c0ba3da8a72
/avatars/404869a87480ef526c1d5fccb583bfa8.svg
Xinyao Niu
user
sirius-ctrl
65ced3ba2a811b35593b217e
Xiang Yue
admin_assigned
2024-02-16T14:29:59.838Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
65ced3ba2a811b35593b217f
Hannaneh Hajishirzi
65ced3ba2a811b35593b2180
Yoon Kim
extracted_pending
2024-02-16T03:17:14.629Z
630ee22c05b5c5280e3cdcdf
/avatars/b282162d6d9ae8512873eba0275f603e.svg
Yoon Kim
user
yoon-kim
65ced3ba2a811b35593b2181
Hao Peng
extracted_pending
2024-04-04T15:24:18.817Z
660ec5a2509153ca49775a7c
/avatars/97570fc245cc8ec7628da9c13bd35b71.svg
Hao Peng
user
haopeng01
Data Engineering for Scaling Language Models to 128K Context
We study the continual pretraining recipe for scaling language models'
context lengths to 128K, with a focus on data engineering. We hypothesize that
long context modeling, in particular the ability to utilize information
at arbitrary input locations, is a capability that is mostly already acquired
through large-scale pretraining, and that this capability can be readily
extended to contexts substantially longer than seen during training~(e.g., 4K
to 128K) through lightweight continual pretraining on appropriate data mixture.
We investigate the quantity and quality of the data for
continual pretraining: (1) for quantity, we show that 500 million to 5 billion
tokens are enough to enable the model to retrieve information anywhere within
the 128K context; (2) for quality, our results equally emphasize domain
balance and length upsampling. Concretely, we find that naively
upsampling longer data on certain domains like books, a common practice of
existing work, gives suboptimal performance, and that a balanced domain mixture
is important. We demonstrate that continual pretraining of the full model on
1B-5B tokens of such data is an effective and affordable strategy for scaling
the context length of language models to 128K. Our recipe outperforms strong
open-source long-context models and closes the gap to frontier models like
GPT-4 128K.