PLaD: Preference-based Large Language Model Distillation with
  Pseudo-Preference Pairs
Large Language Models (LLMs) have exhibited impressive capabilities in
various tasks, yet their vast parameter sizes restrict their applicability in
resource-constrained settings. Knowledge distillation (KD) offers a viable
solution by transferring expertise from large teacher models to compact student
models. However, traditional KD techniques face specific challenges when
applied to LLMs, including restricted access to LLM outputs, significant
teacher-student capacity gaps, and the inherited mis-calibration issue. In this
work, we present PLaD, a novel preference-based LLM distillation framework.
PLaD exploits the teacher-student capacity discrepancy to generate
pseudo-preference pairs where teacher outputs are preferred over student
outputs. Then, PLaD leverages a ranking loss to re-calibrate student's
estimation of sequence likelihood, which steers the student's focus towards
understanding the relative quality of outputs instead of simply imitating the
teacher. PLaD bypasses the need for access to teacher LLM's internal states,
tackles the student's expressivity limitations, and mitigates the student
mis-calibration issue. Through extensive experiments on two sequence generation
tasks and with various LLMs, we demonstrate the effectiveness of our proposed
PLaD framework.
66612600fb3dfd49f05b2215
Rongzhi Zhang
extracted_confirmed
2025-05-16T06:19:21.355Z
644a2c4e9a1c5faef7a5dbd8
/avatars/fbbbc1347f8e423b2477e2506fdb43d9.svg
Rongzhi Zhang
user
Solute
66612600fb3dfd49f05b2216
Jiaming Shen
admin_assigned
2024-06-06T10:08:16.564Z
5f7d1ecdb1a525442ff96ee6
/avatars/cbc7840a6fb318dd37ce91347a7f96e4.svg
Jiaming Shen
user
jmshen
66612600fb3dfd49f05b2217
Tianqi Liu
admin_assigned
2024-06-06T10:08:45.814Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
66612600fb3dfd49f05b2218
Haorui Wang
admin_assigned
2024-06-06T10:08:58.413Z
63476afcdbe3f8b009e2f758
https://cdn-avatars.huggingface.co/v1/production/uploads/1665624822880-noauth.jpeg
Wang Haorui
user
JerrySkywalker
66612600fb3dfd49f05b2219
Zhen Qin
claimed_verified
2024-06-18T08:29:42.402Z
64a2bd9245dde1d6ac7b4d81
/avatars/8a238a832bfcf14e7704a2152b254a0e.svg
Zhen Qin
user
paulqin
66612600fb3dfd49f05b221a
Feng Han
66612600fb3dfd49f05b221b
Jialu Liu
admin_assigned
2024-06-06T10:10:23.991Z
64acb90a5d48838462da09ae
/avatars/dc87416dcb2806a6e0d60bd636024978.svg
liujialu
user
Jialuliu
66612600fb3dfd49f05b221c
Simon Baumgartner
admin_assigned
2024-06-06T10:10:32.982Z
6560f697e0a7720b6ae377bc
https://cdn-avatars.huggingface.co/v1/production/uploads/6560f697e0a7720b6ae377bc/8b-p-lQ7KV_VygBJ6Vf3_.jpeg
Simon Baumgartner
user
sens3
66612600fb3dfd49f05b221d
Michael Bendersky
66612600fb3dfd49f05b221e
Chao Zhang
PLaD: Preference-based Large Language Model Distillation with
  Pseudo-Preference Pairs
Large Language Models (LLMs) have exhibited impressive capabilities in
various tasks, yet their vast parameter sizes restrict their applicability in
resource-constrained settings. Knowledge distillation (KD) offers a viable
solution by transferring expertise from large teacher models to compact student
models. However, traditional KD techniques face specific challenges when
applied to LLMs, including restricted access to LLM outputs, significant
teacher-student capacity gaps, and the inherited mis-calibration issue. In this
work, we present PLaD, a novel preference-based LLM distillation framework.
PLaD exploits the teacher-student capacity discrepancy to generate
pseudo-preference pairs where teacher outputs are preferred over student
outputs. Then, PLaD leverages a ranking loss to re-calibrate student's
estimation of sequence likelihood, which steers the student's focus towards
understanding the relative quality of outputs instead of simply imitating the
teacher. PLaD bypasses the need for access to teacher LLM's internal states,
tackles the student's expressivity limitations, and mitigates the student
mis-calibration issue. Through extensive experiments on two sequence generation
tasks and with various LLMs, we demonstrate the effectiveness of our proposed
PLaD framework.
PLaD is a preference-based distillation framework that re-calibrates compact student LLMs by comparing them to larger teacher models without direct access to internal states.
Large Language Models (LLMs) Knowledge distillation (KD) pseudo-preference pairs ranking loss sequence likelihood sequence generation tasks