Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal
  Representations
Prior research on out-of-distribution detection (OoDD) has primarily focused
on single-modality models. Recently, with the advent of large-scale pretrained
vision-language models such as CLIP, OoDD methods utilizing such multi-modal
representations through zero-shot and prompt learning strategies have emerged.
However, these methods typically involve either freezing the pretrained weights
or only partially tuning them, which can be suboptimal for downstream datasets.
In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve
notable OoDD performance. Despite some recent works demonstrating the impact of
fine-tuning methods for OoDD, there remains significant potential for
performance improvement. We investigate the limitation of na\"ive fine-tuning
methods, examining why they fail to fully leverage the pretrained knowledge.
Our empirical analysis suggests that this issue could stem from the modality
gap within in-distribution (ID) embeddings. To address this, we propose a
training objective that enhances cross-modal alignment by regularizing the
distances between image and text embeddings of ID data. This adjustment helps
in better utilizing pretrained textual information by aligning similar
semantics from different modalities (i.e., text and image) more closely in the
hyperspherical representation space. We theoretically demonstrate that the
proposed regularization corresponds to the maximum likelihood estimation of an
energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark
datasets, we show that our method, combined with post-hoc OoDD approaches
leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms
existing methods, achieving state-of-the-art OoDD performance and leading ID
accuracy.
67ede492bdd88c72dc99fbd7
Jeonghyeon Kim
claimed_verified
2025-04-03T07:52:31.201Z
654b4c9cfabd2cc66874806c
/avatars/c48b423d3657ee27eed0d3d83738ec2e.svg
jeonghyeon kim
user
mawjdgus
67ede492bdd88c72dc99fbd8
Sangheum Hwang
admin_assigned
2025-04-03T14:07:05.202Z
64e6ff6b09375662071ae8c8
/avatars/4e1086476a29733eb4680eb8771d2f68.svg
Sangheum Hwang
user
beopst
Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal
  Representations
Prior research on out-of-distribution detection (OoDD) has primarily focused
on single-modality models. Recently, with the advent of large-scale pretrained
vision-language models such as CLIP, OoDD methods utilizing such multi-modal
representations through zero-shot and prompt learning strategies have emerged.
However, these methods typically involve either freezing the pretrained weights
or only partially tuning them, which can be suboptimal for downstream datasets.
In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve
notable OoDD performance. Despite some recent works demonstrating the impact of
fine-tuning methods for OoDD, there remains significant potential for
performance improvement. We investigate the limitation of na\"ive fine-tuning
methods, examining why they fail to fully leverage the pretrained knowledge.
Our empirical analysis suggests that this issue could stem from the modality
gap within in-distribution (ID) embeddings. To address this, we propose a
training objective that enhances cross-modal alignment by regularizing the
distances between image and text embeddings of ID data. This adjustment helps
in better utilizing pretrained textual information by aligning similar
semantics from different modalities (i.e., text and image) more closely in the
hyperspherical representation space. We theoretically demonstrate that the
proposed regularization corresponds to the maximum likelihood estimation of an
energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark
datasets, we show that our method, combined with post-hoc OoDD approaches
leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms
existing methods, achieving state-of-the-art OoDD performance and leading ID
accuracy.
A novel training objective that enhances cross-modal alignment through regularization of embedding distances improves out-of-distribution detection and in-distribution accuracy using multi-modal fine-tuning.
out-of-distribution detection OoDD single-modality models vision-language models CLIP zero-shot learning prompt learning multi-modal fine-tuning MMFT na\"ive fine-tuning modality gap in-distribution embeddings cross-modal alignment energy-based model hyperspherical representation space ImageNet-1k NegLabel