Instruction Pre-Training: Language Models are Supervised Multitask
  Learners
Unsupervised multitask pre-training has been the critical method behind the
recent success of language models (LMs). However, supervised multitask learning
still holds significant promise, as scaling it in the post-training stage
trends towards better generalization. In this paper, we explore supervised
multitask pre-training by proposing Instruction Pre-Training, a framework that
scalably augments massive raw corpora with instruction-response pairs to
pre-train LMs. The instruction-response pairs are generated by an efficient
instruction synthesizer built on open-source models. In our experiments, we
synthesize 200M instruction-response pairs covering 40+ task categories to
verify the effectiveness of Instruction Pre-Training. In pre-training from
scratch, Instruction Pre-Training not only consistently enhances pre-trained
base models but also benefits more from further instruction tuning. In
continual pre-training, Instruction Pre-Training enables Llama3-8B to be
comparable to or even outperform Llama3-70B. Our model, code, and data are
available at https://github.com/microsoft/LMOps.
6674ef905f7d5c8af70b5607
Daixuan Cheng
claimed_verified
2024-06-21T07:05:43.020Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
6674ef905f7d5c8af70b5608
Yuxian Gu
admin_assigned
2024-06-21T11:12:42.585Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
6674ef905f7d5c8af70b5609
Shaohan Huang
admin_assigned
2024-06-21T11:12:55.699Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
6674ef905f7d5c8af70b560a
Junyu Bi
6674ef905f7d5c8af70b560b
Minlie Huang
6674ef905f7d5c8af70b560c
Furu Wei
admin_assigned
2024-06-21T11:13:45.390Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Instruction Pre-Training: Language Models are Supervised Multitask
  Learners
Unsupervised multitask pre-training has been the critical method behind the
recent success of language models (LMs). However, supervised multitask learning
still holds significant promise, as scaling it in the post-training stage
trends towards better generalization. In this paper, we explore supervised
multitask pre-training by proposing Instruction Pre-Training, a framework that
scalably augments massive raw corpora with instruction-response pairs to
pre-train LMs. The instruction-response pairs are generated by an efficient
instruction synthesizer built on open-source models. In our experiments, we
synthesize 200M instruction-response pairs covering 40+ task categories to
verify the effectiveness of Instruction Pre-Training. In pre-training from
scratch, Instruction Pre-Training not only consistently enhances pre-trained
base models but also benefits more from further instruction tuning. In
continual pre-training, Instruction Pre-Training enables Llama3-8B to be
comparable to or even outperform Llama3-70B. Our model, code, and data are
available at https://github.com/microsoft/LMOps.
Instruction Pre-Training enhances language models by generating and incorporating instruction-response pairs into unsupervised multitask pre-training.
Instruction Pre-Training instruction-response pairs instruction synthesizer open-source models pre-training continual pre-training Llama3-8B Llama3-70B