Decoupled Global-Local Alignment for Improving Compositional
  Understanding
Contrastive Language-Image Pre-training (CLIP) has achieved success on
multiple downstream tasks by aligning image and text modalities. However, the
nature of global contrastive learning limits CLIP's ability to comprehend
compositional concepts, such as relations and attributes. Although recent
studies employ global hard negative samples to improve compositional
understanding, these methods significantly compromise the model's inherent
general capabilities by forcibly distancing textual negative samples from
images in the embedding space. To overcome this limitation, we introduce a
Decoupled Global-Local Alignment (DeGLA) framework that improves compositional
understanding while substantially mitigating losses in general capabilities. To
optimize the retention of the model's inherent capabilities, we incorporate a
self-distillation mechanism within the global alignment process, aligning the
learnable image-text encoder with a frozen teacher model derived from an
exponential moving average. Under the constraint of self-distillation, it
effectively mitigates the catastrophic forgetting of pretrained knowledge
during fine-tuning. To improve compositional understanding, we first leverage
the in-context learning capability of Large Language Models (LLMs) to construct
about 2M high-quality negative captions across five types. Subsequently, we
propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC)
loss to enhance vision-language compositionally. Extensive experimental results
demonstrate the effectiveness of the DeGLA framework. Compared to previous
state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across
the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average
performance improvement of 13.0% on zero-shot classification tasks across
eleven datasets. Our code will be released at
https://github.com/xiaoxing2001/DeGLA
6809bebd0f6dfd7bd5159b76
Xiaoxing Hu
claimed_verified
2025-04-28T12:50:22.326Z
67223563fa69c82e19d2232c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1z_axjIty3uB4UDYa9JK4.png
xiaoxinghhh
user
wsdwJohn1231
6809bebd0f6dfd7bd5159b77
Kaicheng Yang
claimed_verified
2025-04-24T09:10:22.302Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
6809bebd0f6dfd7bd5159b78
Jun Wang
6809bebd0f6dfd7bd5159b79
Haoran Xu
admin_assigned
2025-04-24T11:26:36.470Z
61384b860317b0a5c10877d3
https://cdn-avatars.huggingface.co/v1/production/uploads/1631080954171-61384b860317b0a5c10877d3.jpeg
Haoran Xu
user
haoranxu
6809bebd0f6dfd7bd5159b7a
Ziyong Feng
6809bebd0f6dfd7bd5159b7b
Yupei Wang
Decoupled Global-Local Alignment for Improving Compositional
  Understanding
Contrastive Language-Image Pre-training (CLIP) has achieved success on
multiple downstream tasks by aligning image and text modalities. However, the
nature of global contrastive learning limits CLIP's ability to comprehend
compositional concepts, such as relations and attributes. Although recent
studies employ global hard negative samples to improve compositional
understanding, these methods significantly compromise the model's inherent
general capabilities by forcibly distancing textual negative samples from
images in the embedding space. To overcome this limitation, we introduce a
Decoupled Global-Local Alignment (DeGLA) framework that improves compositional
understanding while substantially mitigating losses in general capabilities. To
optimize the retention of the model's inherent capabilities, we incorporate a
self-distillation mechanism within the global alignment process, aligning the
learnable image-text encoder with a frozen teacher model derived from an
exponential moving average. Under the constraint of self-distillation, it
effectively mitigates the catastrophic forgetting of pretrained knowledge
during fine-tuning. To improve compositional understanding, we first leverage
the in-context learning capability of Large Language Models (LLMs) to construct
about 2M high-quality negative captions across five types. Subsequently, we
propose the Image-Grounded Contrast (IGC) loss and Text-Grounded Contrast (TGC)
loss to enhance vision-language compositionally. Extensive experimental results
demonstrate the effectiveness of the DeGLA framework. Compared to previous
state-of-the-art methods, DeGLA achieves an average enhancement of 3.5% across
the VALSE, SugarCrepe, and ARO benchmarks. Concurrently, it obtains an average
performance improvement of 13.0% on zero-shot classification tasks across
eleven datasets. Our code will be released at
https://github.com/xiaoxing2001/DeGLA
The DeGLA framework enhances compositional understanding in vision-language models through decoupled global-local alignment, self-distillation, and image-text grounded contrastive losses, improving performance on benchmarks and zero-shot classification tasks.
Contrastive Language-Image Pre-training (CLIP) global contrastive learning compositional concepts global hard negative samples Decoupled Global-Local Alignment (DeGLA) self-distillation frozen teacher model exponential moving average catastrophic forgetting Large Language Models (LLMs) negative captions Image-Grounded Contrast (IGC) loss Text-Grounded Contrast (TGC) loss VALSE SugarCrepe ARO benchmarks zero-shot classification tasks