PLD: A Choice-Theoretic List-Wise Knowledge Distillation
Ejafa Bassam, Dawei Zhu, Kaigui Bian
Knowledge distillation is a model compression technique in which a compact "student" network is trained to replicate the predictive behavior of a larger "teacher" network. In logit-based knowledge distillation, it has become the de facto approach to augment cross-entropy with a distillation term. Typically, this term is either a KL divergence-matching marginal probabilities or a correlation-based loss-capturing intra and inter-class relationships-but in every case it sits as an "add-on" to cross-entropy, with its own weight that must be carefully tuned. In this paper, we adopt a choice‐theoretic perspective and recast knowledge distillation under the Plackett-Luce model by interpreting teacher logits as "worth" scores. We introduce *Plackett–Luce Distillation (PLD)*, a weighted list‐wise ranking loss in which the teacher model transfers knowledge of its full ranking of classes, weighting each ranked choice by its own confidence. PLD directly optimizes a single "teacher‐optimal" ranking-true label first, followed by the remaining classes in descending teacher confidence-yielding a convex, translation-invariant surrogate that subsumes weighted cross-entropy. Empirically, PLD is robust with default settings and consistently outperforms both classical KL‐based distillation and recent correlation-based method (DIST) across diverse student-teacher architectures and model sizes on standard image classification benchmarks.