CLoVe: Encoding Compositional Language in Contrastive Vision-Language
  Models
Recent years have witnessed a significant increase in the performance of
Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as
CLIP, have been leveraged in multiple settings and demonstrated remarkable
performance across several tasks. Such models excel at object-centric
recognition yet learn text representations that seem invariant to word order,
failing to compose known concepts in novel ways. However, no evidence exists
that any VLM, including large-scale single-stream models such as GPT-4V,
identifies compositions successfully. In this paper, we introduce a framework
to significantly improve the ability of existing models to encode compositional
language, with over 10% absolute improvement on compositionality benchmarks,
while maintaining or improving the performance on standard object-recognition
and retrieval benchmarks. Our code and pre-trained models are publicly
available at https://github.com/netflix/clove.
65dbf75b594bab313c713f68
Santiago Castro
extracted_confirmed
2024-02-26T04:03:04.956Z
5e7b8bd930dc073f817a2ba2
https://cdn-avatars.huggingface.co/v1/production/uploads/1628132691344-5e7b8bd930dc073f817a2ba2.jpeg
Santiago Castro
user
bryant1410
65dbf75b594bab313c713f69
Amir Ziai
admin_assigned
2024-02-26T10:44:24.809Z
63a38a7f3618ae17d6649165
/avatars/1bf374f038f3cf27415f14d07b03d5b7.svg
Amir
user
amirziai
65dbf75b594bab313c713f6a
Avneesh Saluja
65dbf75b594bab313c713f6b
Zhuoning Yuan
65dbf75b594bab313c713f6c
Rada Mihalcea
admin_assigned
2024-02-26T11:05:13.981Z
64153d8f95fb6f824b22d960
/avatars/e189e792dbdb0e2a8a04eda3a8c2bcfc.svg
Rada Mihalcea
user
mihalcea
CLoVe: Encoding Compositional Language in Contrastive Vision-Language
  Models
Recent years have witnessed a significant increase in the performance of
Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as
CLIP, have been leveraged in multiple settings and demonstrated remarkable
performance across several tasks. Such models excel at object-centric
recognition yet learn text representations that seem invariant to word order,
failing to compose known concepts in novel ways. However, no evidence exists
that any VLM, including large-scale single-stream models such as GPT-4V,
identifies compositions successfully. In this paper, we introduce a framework
to significantly improve the ability of existing models to encode compositional
language, with over 10% absolute improvement on compositionality benchmarks,
while maintaining or improving the performance on standard object-recognition
and retrieval benchmarks. Our code and pre-trained models are publicly
available at https://github.com/netflix/clove.
A new framework enhances compositional language encoding in Vision-Language Models while maintaining or improving object recognition, achieving significant gains on benchmark tests.
foundational vision-language models VLMs CLIP GPT-4V compositional language compositionality benchmarks object-recognition retrieval benchmarks