Simple and Scalable Strategies to Continually Pre-train Large Language
  Models
Large language models (LLMs) are routinely pre-trained on billions of tokens,
only to start the process over again once new data becomes available. A much
more efficient solution is to continually pre-train these models, saving
significant compute compared to re-training. However, the distribution shift
induced by new data typically results in degraded performance on previous data
or poor adaptation to the new data. In this work, we show that a simple and
scalable combination of learning rate (LR) re-warming, LR re-decaying, and
replay of previous data is sufficient to match the performance of fully
re-training from scratch on all available data, as measured by final loss and
language model (LM) evaluation benchmarks. Specifically, we show this for a
weak but realistic distribution shift between two commonly used LLM
pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution
shift (EnglishrightarrowGerman) at the 405M parameter model scale with
large dataset sizes (hundreds of billions of tokens). Selecting the weak but
realistic shift for larger-scale experiments, we also find that our continual
learning strategies match the re-training baseline for a 10B parameter LLM. Our
results demonstrate that LLMs can be successfully updated via simple and
scalable continual learning strategies, matching the re-training baseline using
only a fraction of the compute. Finally, inspired by previous work, we propose
alternatives to the cosine learning rate schedule that help circumvent
forgetting induced by LR re-warming and that are not bound to a fixed token
budget.
65f271e1b06ca1ff48a12ff3
Adam Ibrahim
extracted_confirmed
2024-03-14T04:44:57.674Z
648743f05aed4a7b0bf2e3a6
/avatars/e6f3b35085b1b8cb2c231516d1fea70a.svg
Adam Ibrahim
user
adami1
65f271e1b06ca1ff48a12ff4
Benjamin Thérien
admin_assigned
2024-03-14T08:01:31.759Z
62d0e2a3fb896639b2a04ca1
https://cdn-avatars.huggingface.co/v1/production/uploads/62d0e2a3fb896639b2a04ca1/UBZzdldkvCAEHp-t6L2JJ.jpeg
Benjamin Therien
user
btherien
65f271e1b06ca1ff48a12ff5
Kshitij Gupta
claimed_verified
2024-03-14T07:59:44.947Z
625daa0c943e346492b63d21
/avatars/860f8e9bff85325096b6f8e15102def6.svg
Kshitij Gupta
user
kshitijkg
65f271e1b06ca1ff48a12ff6
Mats L. Richter
65f271e1b06ca1ff48a12ff7
Quentin Anthony
extracted_confirmed
2024-06-20T01:09:03.116Z
63b9a261efe99543b34e9579
https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
Quentin Anthony
user
qanthony
65f271e1b06ca1ff48a12ff8
Timothée Lesort
extracted_pending
2024-03-14T03:41:24.077Z
631610b876c93113fb129947
/avatars/fe80d40ac13330c2cb64958d2aee253d.svg
Timothée Lesort
user
TLESORT
65f271e1b06ca1ff48a12ff9
Eugene Belilovsky
65f271e1b06ca1ff48a12ffa
Irina Rish
admin_assigned
2024-03-14T08:02:13.169Z
6227f764236b7b2eefa7f173
/avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
Irina Rish
user
irinarish
Simple and Scalable Strategies to Continually Pre-train Large Language
  Models
Large language models (LLMs) are routinely pre-trained on billions of tokens,
only to start the process over again once new data becomes available. A much
more efficient solution is to continually pre-train these models, saving
significant compute compared to re-training. However, the distribution shift
induced by new data typically results in degraded performance on previous data
or poor adaptation to the new data. In this work, we show that a simple and
scalable combination of learning rate (LR) re-warming, LR re-decaying, and
replay of previous data is sufficient to match the performance of fully
re-training from scratch on all available data, as measured by final loss and
language model (LM) evaluation benchmarks. Specifically, we show this for a
weak but realistic distribution shift between two commonly used LLM
pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution
shift (EnglishrightarrowGerman) at the 405M parameter model scale with
large dataset sizes (hundreds of billions of tokens). Selecting the weak but
realistic shift for larger-scale experiments, we also find that our continual
learning strategies match the re-training baseline for a 10B parameter LLM. Our
results demonstrate that LLMs can be successfully updated via simple and
scalable continual learning strategies, matching the re-training baseline using
only a fraction of the compute. Finally, inspired by previous work, we propose
alternatives to the cosine learning rate schedule that help circumvent
forgetting induced by LR re-warming and that are not bound to a fixed token
budget.
Simple continual learning strategies, including learning rate re-warming, re-decaying, and data replay, can match the performance of full re-training for large language models with much lower computational cost.
learning rate re-warming learning rate re-decaying data replay continual learning distribution shift language model evaluation benchmarks