Knowledge Distillation of Large Language Models
Knowledge Distillation (KD) is a promising technique for reducing the high
computational demand of large language models (LLMs). However, previous KD
methods are primarily applied to white-box classification models or training
small models to imitate black-box model APIs like ChatGPT. How to effectively
distill the knowledge from white-box generative LLMs is still under-explored,
which becomes more and more important with the prosperity of LLMs. In this
work, we propose MiniLLM that distills smaller language models from generative
larger language models. We first replace the forward Kullback-Leibler
divergence (KLD) objective in the standard KD approaches with reverse KLD,
which is more suitable for KD on generative language models, to prevent the
student model from overestimating the low-probability regions of the teacher
distribution. Then, we derive an effective optimization approach to learn this
objective. Extensive experiments in the instruction-following setting show that
the MiniLLM models generate more precise responses with the higher overall
quality, lower exposure bias, better calibration, and higher long-text
generation performance. Our method is also scalable for different model
families with 120M to 13B parameters. We will release our code and model
checkpoints at https://aka.ms/MiniLLM.
648bc4a8d60140c4bbca5278
Yuxian Gu
claimed_verified
2023-06-16T17:38:47.105Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
648bc4a8d60140c4bbca5279
Li Dong
claimed_verified
2024-10-08T15:20:13.208Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
648bc4a8d60140c4bbca527a
Furu Wei
admin_assigned
2023-06-19T12:03:25.107Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
648bc4a8d60140c4bbca527b
Minlie Huang
Knowledge Distillation of Large Language Models
Knowledge Distillation (KD) is a promising technique for reducing the high
computational demand of large language models (LLMs). However, previous KD
methods are primarily applied to white-box classification models or training
small models to imitate black-box model APIs like ChatGPT. How to effectively
distill the knowledge from white-box generative LLMs is still under-explored,
which becomes more and more important with the prosperity of LLMs. In this
work, we propose MiniLLM that distills smaller language models from generative
larger language models. We first replace the forward Kullback-Leibler
divergence (KLD) objective in the standard KD approaches with reverse KLD,
which is more suitable for KD on generative language models, to prevent the
student model from overestimating the low-probability regions of the teacher
distribution. Then, we derive an effective optimization approach to learn this
objective. Extensive experiments in the instruction-following setting show that
the MiniLLM models generate more precise responses with the higher overall
quality, lower exposure bias, better calibration, and higher long-text
generation performance. Our method is also scalable for different model
families with 120M to 13B parameters. We will release our code and model
checkpoints at https://aka.ms/MiniLLM.
MiniLLM distills knowledge from large generative language models to smaller models using reverse KLD for better precision, quality, and performance.
Knowledge Distillation (KD) large language models (LLMs) white-box generative LLMs forward Kullback-Leibler divergence (KLD) reverse KLD optimization approach instruction-following exposure bias calibration long-text generation