Pre-training Distillation for Large Language Models: A Design Space
  Exploration
Knowledge distillation (KD) aims to transfer knowledge from a large teacher
model to a smaller student model. Previous work applying KD in the field of
large language models (LLMs) typically focused on the post-training phase,
where the student LLM learns directly from instructions and corresponding
responses generated by the teacher model. In this paper, we extend KD to the
pre-training phase of LLMs, named pre-training distillation (PD). We first
conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a
1.9B parameter student LLM, validating the effectiveness of PD. Considering the
key impact factors of distillation, we systematically explore the design space
of pre-training distillation across four aspects: logits processing, loss
selection, scaling law, and offline or online logits. We conduct extensive
experiments to explore the design space of pre-training distillation and find
better configurations and interesting conclusions, such as larger student LLMs
generally benefiting more from pre-training distillation, while a larger
teacher LLM does not necessarily guarantee better results. We hope our
exploration of the design space will inform future practices in pre-training
distillation.
67173a397994589e3f441da8
Hao Peng
claimed_verified
2024-10-22T09:48:33.508Z
625a5446f1063e7085d5178a
/avatars/5e78186f13f74b14e01583e06ff6c4dc.svg
Hao Peng
user
Wesleythu
67173a397994589e3f441da9
Xin Lv
67173a397994589e3f441daa
Yushi Bai
claimed_verified
2024-10-22T07:55:46.101Z
64ed568ccf6118a9379a61b8
/avatars/6d040cbcb4a9b624cbe64c9d01cd5c88.svg
Yushi Bai
user
bys0318
67173a397994589e3f441dab
Zijun Yao
67173a397994589e3f441dac
Jiajie Zhang
claimed_verified
2024-10-22T09:14:57.280Z
66cdd285c51a915bd5f2d017
/avatars/14e5794307e4672b1b51d26b31227e0f.svg
Jiajie Zhang
user
NeoZ123
67173a397994589e3f441dad
Lei Hou
67173a397994589e3f441dae
Juanzi Li
admin_assigned
2024-10-22T09:13:58.645Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
Pre-training Distillation for Large Language Models: A Design Space
  Exploration
Knowledge distillation (KD) aims to transfer knowledge from a large teacher
model to a smaller student model. Previous work applying KD in the field of
large language models (LLMs) typically focused on the post-training phase,
where the student LLM learns directly from instructions and corresponding
responses generated by the teacher model. In this paper, we extend KD to the
pre-training phase of LLMs, named pre-training distillation (PD). We first
conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a
1.9B parameter student LLM, validating the effectiveness of PD. Considering the
key impact factors of distillation, we systematically explore the design space
of pre-training distillation across four aspects: logits processing, loss
selection, scaling law, and offline or online logits. We conduct extensive
experiments to explore the design space of pre-training distillation and find
better configurations and interesting conclusions, such as larger student LLMs
generally benefiting more from pre-training distillation, while a larger
teacher LLM does not necessarily guarantee better results. We hope our
exploration of the design space will inform future practices in pre-training
distillation.
The study extends knowledge distillation to the pre-training phase of large language models, demonstrating its effectiveness and exploring key design factors like logits processing, loss selection, scaling laws, and offline/online logits.
knowledge distillation large language models pre-training distillation GLM-4-9B logits processing loss selection scaling law offline or online logits