VisualGPTScore: Visio-Linguistic Reasoning with Multimodal Generative
  Pre-Training Scores
Vision-language models (VLMs) discriminatively pre-trained with contrastive
image-text matching losses such as P(match|text, image)
have been criticized for lacking compositional understanding. This means they
might output similar scores even if the original caption is rearranged into a
different semantic statement. To address this, we propose to use the {bf
V}isual {bf G}enerative {bf P}re-{bf T}raining Score ({bf
VisualGPTScore}) of P(text|image), a multimodal
generative score that captures the likelihood of a text caption conditioned
on an image using an image-conditioned language model. Contrary to the belief
that VLMs are mere bag-of-words models, our off-the-shelf VisualGPTScore
demonstrates top-tier performance on recently proposed image-text retrieval
benchmarks like ARO and Crepe that assess compositional reasoning. Furthermore,
we factorize VisualGPTScore into a product of the marginal P(text)
and the Pointwise Mutual Information (PMI). This helps to (a)
diagnose datasets with strong language bias, and (b) debias results on other
benchmarks like Winoground using an information-theoretic framework.
VisualGPTScore provides valuable insights and serves as a strong baseline for
future evaluation of visio-linguistic compositionality.
647ea243770c299e56fcebf7
Zhiqiu Lin
claimed_verified
2024-06-14T07:27:29.827Z
64c170190bfb901b04399295
/avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
Zhiqiu Lin
user
zhiqiulin
647ea243770c299e56fcebf8
Xinyue Chen
647ea243770c299e56fcebf9
Deepak Pathak
647ea243770c299e56fcebfa
Pengchuan Zhang
647ea243770c299e56fcebfb
Deva Ramanan
admin_assigned
2023-06-06T09:00:23.979Z
6337151b0267ebcf02640eb6
/avatars/14a723cafc5587043bdfb19304fc202d.svg
Deva Ramanan
user
devakramanan
VisualGPTScore: Visio-Linguistic Reasoning with Multimodal Generative
  Pre-Training Scores
Vision-language models (VLMs) discriminatively pre-trained with contrastive
image-text matching losses such as P(match|text, image)
have been criticized for lacking compositional understanding. This means they
might output similar scores even if the original caption is rearranged into a
different semantic statement. To address this, we propose to use the {bf
V}isual {bf G}enerative {bf P}re-{bf T}raining Score ({bf
VisualGPTScore}) of P(text|image), a multimodal
generative score that captures the likelihood of a text caption conditioned
on an image using an image-conditioned language model. Contrary to the belief
that VLMs are mere bag-of-words models, our off-the-shelf VisualGPTScore
demonstrates top-tier performance on recently proposed image-text retrieval
benchmarks like ARO and Crepe that assess compositional reasoning. Furthermore,
we factorize VisualGPTScore into a product of the marginal P(text)
and the Pointwise Mutual Information (PMI). This helps to (a)
diagnose datasets with strong language bias, and (b) debias results on other
benchmarks like Winoground using an information-theoretic framework.
VisualGPTScore provides valuable insights and serves as a strong baseline for
future evaluation of visio-linguistic compositionality.
VisualGPTScore, a multimodal generative score using an image-conditioned language model, enhances compositional understanding in vision-language models and outperforms discriminatively pre-trained models on image-text retrieval benchmarks.
contrastive image-text matching vision-language models VisualGPTScore multimodal generative image-conditioned language model compositional reasoning ARO Crepe Winoground Pointwise Mutual Information (PMI)