LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks
  for Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have achieved strong performance on
general visual benchmarks but struggle with out-of-distribution (OOD) tasks in
specialized domains such as medical imaging, where labeled data is limited and
expensive. We introduce LEAML, a label-efficient adaptation framework that
leverages both scarce labeled VQA samples and abundant unlabeled images. Our
approach generates domain-relevant pseudo question-answer pairs for unlabeled
data using a QA generator regularized by caption distillation. Importantly, we
selectively update only those neurons most relevant to question-answering,
enabling the QA Generator to efficiently acquire domain-specific knowledge
during distillation. Experiments on gastrointestinal endoscopy and sports VQA
demonstrate that LEAML consistently outperforms standard fine-tuning under
minimal supervision, highlighting the effectiveness of our proposed LEAML
framework.
68e385ab73e20ab577842165
Ci-Siang Lin
68e385ab73e20ab577842166
Min-Hung Chen
claimed_verified
2025-10-06T12:48:24.797Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
68e385ab73e20ab577842167
Yu-Yang Sheng
68e385ab73e20ab577842168
Yu-Chiang Frank Wang
LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks
  for Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have achieved strong performance on
general visual benchmarks but struggle with out-of-distribution (OOD) tasks in
specialized domains such as medical imaging, where labeled data is limited and
expensive. We introduce LEAML, a label-efficient adaptation framework that
leverages both scarce labeled VQA samples and abundant unlabeled images. Our
approach generates domain-relevant pseudo question-answer pairs for unlabeled
data using a QA generator regularized by caption distillation. Importantly, we
selectively update only those neurons most relevant to question-answering,
enabling the QA Generator to efficiently acquire domain-specific knowledge
during distillation. Experiments on gastrointestinal endoscopy and sports VQA
demonstrate that LEAML consistently outperforms standard fine-tuning under
minimal supervision, highlighting the effectiveness of our proposed LEAML
framework.
LEAML, a label-efficient adaptation framework, enhances MLLMs for specialized domains by generating pseudo question-answer pairs and selectively updating relevant neurons, outperforming standard fine-tuning with minimal supervision.
Multimodal Large Language Models MLLMs out-of-distribution OOD medical imaging labeled data LEAML label-efficient adaptation QA generator caption distillation domain-relevant pseudo question-answer pairs unlabeled images selective update neurons question-answering domain-specific knowledge gastrointestinal endoscopy sports VQA standard fine-tuning