Diffusion Feedback Helps CLIP See Better
Contrastive Language-Image Pre-training (CLIP), which excels at abstracting
open-world representations across domains and modalities, has become a
foundation for a variety of vision and multimodal tasks. However, recent
studies reveal that CLIP has severe visual shortcomings, such as which can
hardly distinguish orientation, quantity, color, structure, etc. These visual
shortcomings also limit the perception capabilities of multimodal large
language models (MLLMs) built on CLIP. The main reason could be that the
image-text pairs used to train CLIP are inherently biased, due to the lack of
the distinctiveness of the text and the diversity of images. In this work, we
present a simple post-training approach for CLIP models, which largely
overcomes its visual shortcomings via a self-supervised diffusion process. We
introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP.
Specifically, DIVA leverages generative feedback from text-to-image diffusion
models to optimize CLIP representations, with only images (without
corresponding text). We demonstrate that DIVA improves CLIP's performance on
the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities
to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and
vision models on multimodal understanding and segmentation tasks. Extensive
evaluation on 29 image classification and retrieval benchmarks confirms that
our framework preserves CLIP's strong zero-shot capabilities. The code will be
available at https://github.com/baaivision/DIVA.
66a85e795f395fc0c6c7afac
Wenxuan Wang
claimed_verified
2024-07-30T09:51:07.296Z
656428b5462e5ebcbf537d4e
/avatars/cbedecc9c6f2afee2ca6b72efb593561.svg
Wenxuan Wang
user
Rookielion
66a85e795f395fc0c6c7afad
Quan Sun
admin_assigned
2024-07-30T08:42:05.741Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
66a85e795f395fc0c6c7afae
Fan Zhang
claimed_verified
2024-10-08T07:22:37.841Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
66a85e795f395fc0c6c7afaf
Yepeng Tang
admin_assigned
2024-07-30T08:42:34.745Z
64dee07b3d3a7519f18ebbe2
/avatars/8f113938715b094435e014509630aa68.svg
tang
user
yepeng98
66a85e795f395fc0c6c7afb0
Jing Liu
66a85e795f395fc0c6c7afb1
Xinlong Wang
admin_assigned
2024-07-30T08:43:46.103Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
Diffusion Feedback Helps CLIP See Better
Contrastive Language-Image Pre-training (CLIP), which excels at abstracting
open-world representations across domains and modalities, has become a
foundation for a variety of vision and multimodal tasks. However, recent
studies reveal that CLIP has severe visual shortcomings, such as which can
hardly distinguish orientation, quantity, color, structure, etc. These visual
shortcomings also limit the perception capabilities of multimodal large
language models (MLLMs) built on CLIP. The main reason could be that the
image-text pairs used to train CLIP are inherently biased, due to the lack of
the distinctiveness of the text and the diversity of images. In this work, we
present a simple post-training approach for CLIP models, which largely
overcomes its visual shortcomings via a self-supervised diffusion process. We
introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP.
Specifically, DIVA leverages generative feedback from text-to-image diffusion
models to optimize CLIP representations, with only images (without
corresponding text). We demonstrate that DIVA improves CLIP's performance on
the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities
to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and
vision models on multimodal understanding and segmentation tasks. Extensive
evaluation on 29 image classification and retrieval benchmarks confirms that
our framework preserves CLIP's strong zero-shot capabilities. The code will be
available at https://github.com/baaivision/DIVA.
DIVA enhances CLIP's performance through a self-supervised diffusion process, improving visual capabilities and multimodal understanding without additional text labels.
Contrastive Language-Image Pre-training CLIP visual shortcomings multimodal large language models MLLMs image-text pairs self-supervised diffusion DIffusion model Visual Assistant generative feedback text-to-image diffusion MMVP-VLM benchmark image classification image retrieval zero-shot capabilities