SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization
Tongtong Su, Yun Liao, Fengbo Zheng
Self-knowledge distillation (SKD) enables single-model training by distilling knowledge from the model's own outputs, eliminating the need for a separate teacher network required in conventional distillation methods. However, current SKD methods mainly focus on replicating common features in the student model, neglecting the extraction of key features that significantly enhance student learning. Inspired by this, we devise a self-knowledge distillation framework entitled Self-Distillation training via Proximal Gradient Optimization or SDPGO, which utilizes gradient information to identify and assign greater weight to features that significantly impact classification performance, enabling the network to learn the most relevant features during training. Additionally, this manner reduces the computational cost while enhancing the model's feature representation capability. Specifically, the proposed framework refines gradient information into a dynamically changing weighting factor to evaluate the distillation knowledge via the dynamic weight adjustment scheme. Our SDPGO technique uses the gradient information to extract the features most relevant to its own learning, and assigns more weights to the features that significantly affect the classification loss, which improves the representation learning of the student model. Extensive experimental results on five image classification benchmarks, including small-scale, large-scale, and fine-grained datasets, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.