Simple Semi-supervised Knowledge Distillation from Vision-Language
  Models via texttt{D}ual-texttt{H}ead
  texttt{O}ptimization
Vision-language models (VLMs) have achieved remarkable success across diverse
tasks by leveraging rich textual information with minimal labeled data.
However, deploying such large models remains challenging, particularly in
resource-constrained environments. Knowledge distillation (KD) offers a
well-established solution to this problem; however, recent KD approaches from
VLMs often involve multi-stage training or additional tuning, increasing
computational overhead and optimization complexity. In this paper, we propose
texttt{D}ual-texttt{H}ead
texttt{O}ptimization (texttt{DHO}) -- a simple yet
effective KD framework that transfers knowledge from VLMs to compact,
task-specific models in semi-supervised settings. Specifically, we introduce
dual prediction heads that independently learn from labeled data and teacher
predictions, and propose to linearly combine their outputs during inference. We
observe that DHO mitigates gradient conflicts between supervised and
distillation signals, enabling more effective feature learning than single-head
KD baselines. As a result, extensive experiments show that DHO
consistently outperforms baselines across multiple domains and fine-grained
datasets. Notably, on ImageNet, it achieves state-of-the-art performance,
improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively,
while using fewer parameters.
6829dcab0daa5ccc817e6ec8
Seongjae Kang
admin_assigned
2025-05-19T17:47:06.348Z
6357a08f8ed056fa1ccd3b38
/avatars/07d4ca8f3197a6945ad71e6150801135.svg
Seongjae Kang
user
erjui
6829dcab0daa5ccc817e6ec9
Dong Bok Lee
claimed_verified
2025-05-19T06:42:58.152Z
64f000769e7770db74d44bba
/avatars/d015820380ffb823b1b35df64dcd3457.svg
Dong-Bok Lee
user
dongboklee
6829dcab0daa5ccc817e6eca
Hyungjoon Jang
admin_assigned
2025-05-19T14:20:23.560Z
633ad9e4976a7d6910e84d15
/avatars/bb464e7503d6e865d1c351430982f7dd.svg
Hyungjoon Jang
user
hjoon
6829dcab0daa5ccc817e6ecb
Sung Ju Hwang
Simple Semi-supervised Knowledge Distillation from Vision-Language
  Models via texttt{D}ual-texttt{H}ead
  texttt{O}ptimization
Vision-language models (VLMs) have achieved remarkable success across diverse
tasks by leveraging rich textual information with minimal labeled data.
However, deploying such large models remains challenging, particularly in
resource-constrained environments. Knowledge distillation (KD) offers a
well-established solution to this problem; however, recent KD approaches from
VLMs often involve multi-stage training or additional tuning, increasing
computational overhead and optimization complexity. In this paper, we propose
texttt{D}ual-texttt{H}ead
texttt{O}ptimization (texttt{DHO}) -- a simple yet
effective KD framework that transfers knowledge from VLMs to compact,
task-specific models in semi-supervised settings. Specifically, we introduce
dual prediction heads that independently learn from labeled data and teacher
predictions, and propose to linearly combine their outputs during inference. We
observe that DHO mitigates gradient conflicts between supervised and
distillation signals, enabling more effective feature learning than single-head
KD baselines. As a result, extensive experiments show that DHO
consistently outperforms baselines across multiple domains and fine-grained
datasets. Notably, on ImageNet, it achieves state-of-the-art performance,
improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively,
while using fewer parameters.
Dual-Head Optimization (DHO) framework improves knowledge distillation from vision-language models to compact task-specific models by using dual prediction heads, achieving state-of-the-art performance with fewer parameters.
vision-language models knowledge distillation dual prediction heads gradient conflicts ImageNet