Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Knowledge distillation can be a cost-effective technique to distill knowledge
in Large Language Models, if the teacher output logits can be pre-computed and
cached. However, successfully applying this to pre-training remains largely
unexplored. In this work, we prove that naive approaches for sparse knowledge
distillation such as caching Top-K probabilities, while intuitive, provide
biased estimates of teacher probability distribution to the student, resulting
in suboptimal performance and calibration. We propose an
importance-sampling-based method `Random Sampling Knowledge Distillation',
which provides unbiased estimates, preserves the gradient in expectation, and
requires storing significantly sparser logits. Our method enables faster
training of student models with marginal overhead (<10%) compared to
cross-entropy based training, while maintaining competitive performance
compared to full distillation, across a range of model sizes from 300M to 3B.
67e4c860136c8a867191e52e
Anshumann
admin_assigned
2025-03-27T10:03:45.930Z
6627ff2a3b4fbc8420a416c3
/avatars/de3aafdaf5563fe25edcdb92b394474f.svg
AR
user
Anshumann
67e4c860136c8a867191e52f
Mohd Abbas Zaidi
admin_assigned
2025-03-27T10:03:57.163Z
61765fe0b0715831eab6d465
/avatars/830406cebdd7ddf66799bf71f6bbff4b.svg
Mohd Abbas Zaidi
user
ya-mehdi
67e4c860136c8a867191e530
Akhil Kedia
67e4c860136c8a867191e531
Jinwoo Ahn
admin_assigned
2025-03-27T10:04:07.687Z
65d61eebee922bc7a777d5a6
/avatars/180efd503ef412b4dc728e6aa477c16e.svg
Jinwoo Ahn
user
AndrewAhn
67e4c860136c8a867191e532
Taehwak Kwon
admin_assigned
2025-03-27T10:04:23.731Z
629059cdb90dde28ef5cbb30
/avatars/451b0e8999447b3a2f03378fe98c0661.svg
Taehwak Kwon
user
Rock222
67e4c860136c8a867191e533
Kangwook Lee
67e4c860136c8a867191e534
Haejun Lee
admin_assigned
2025-03-27T10:04:39.155Z
653b6e7232bd4db35d981615
/avatars/0fbf0c5d502b840bf26baf8c420c7593.svg
Haejun Lee
user
haejunlee
67e4c860136c8a867191e535
Joohyung Lee
admin_assigned
2025-03-27T10:04:46.702Z
64b4be0665a7e15eac085878
/avatars/dd19ec2987fb0735457c6492b53aacfe.svg
Joo-hyung Lee
user
snrbs17
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Knowledge distillation can be a cost-effective technique to distill knowledge
in Large Language Models, if the teacher output logits can be pre-computed and
cached. However, successfully applying this to pre-training remains largely
unexplored. In this work, we prove that naive approaches for sparse knowledge
distillation such as caching Top-K probabilities, while intuitive, provide
biased estimates of teacher probability distribution to the student, resulting
in suboptimal performance and calibration. We propose an
importance-sampling-based method `Random Sampling Knowledge Distillation',
which provides unbiased estimates, preserves the gradient in expectation, and
requires storing significantly sparser logits. Our method enables faster
training of student models with marginal overhead (<10%) compared to
cross-entropy based training, while maintaining competitive performance
compared to full distillation, across a range of model sizes from 300M to 3B.
Knowledge distillation Large Language Models teacher output logits pre-computed cached sparse knowledge distillation Top-K probabilities biased estimates teacher probability distribution student suboptimal performance calibration importance-sampling-based method Random Sampling Knowledge Distillation unbiased estimates gradient in expectation storing significantly sparser logits cross-entropy based training competitive performance model sizes