FG-CLIP: Fine-Grained Visual and Textual Alignment
Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks
such as image-text retrieval and zero-shot classification but struggles with
fine-grained understanding due to its focus on coarse-grained short captions.
To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances
fine-grained understanding through three key innovations. First, we leverage
large multimodal models to generate 1.6 billion long caption-image pairs for
capturing global-level semantic details. Second, a high-quality dataset is
constructed with 12 million images and 40 million region-specific bounding
boxes aligned with detailed captions to ensure precise, context-rich
representations. Third, 10 million hard fine-grained negative samples are
incorporated to improve the model's ability to distinguish subtle semantic
differences. Corresponding training methods are meticulously designed for these
data. Extensive experiments demonstrate that FG-CLIP outperforms the original
CLIP and other state-of-the-art methods across various downstream tasks,
including fine-grained understanding, open-vocabulary object detection,
image-text retrieval, and general multimodal benchmarks. These results
highlight FG-CLIP's effectiveness in capturing fine-grained image details and
improving overall model performance. The related data, code, and models are
available at https://github.com/360CVGroup/FG-CLIP.
681da6375f701833274a0d21
Chunyu Xie
claimed_verified
2025-05-09T07:21:03.296Z
6621e591c50869c1e91a1639
https://cdn-avatars.huggingface.co/v1/production/uploads/6621e591c50869c1e91a1639/L_PoEn2BRAJcnWZX-JebR.jpeg
Chunyu Xie
user
xiechunyu
681da6375f701833274a0d22
Bin Wang
681da6375f701833274a0d23
Fanjing Kong
admin_assigned
2025-05-09T08:58:42.881Z
632c098b456c31252774e7c5
/avatars/e3720d2fcb69d93c8d5aa5f50aab5f0e.svg
kong
user
fanjing
681da6375f701833274a0d24
Jincheng Li
admin_assigned
2025-05-09T08:58:50.538Z
65b793b374698ba5a815bf4f
/avatars/44a7e694a5089dbc773018111270ac26.svg
Jincheng Li
user
jinchenglijc
681da6375f701833274a0d25
Dawei Liang
admin_assigned
2025-05-09T08:58:56.445Z
659b8576999b82db2ad8a398
/avatars/2ec7663e25e4a0238819818e69d9a5bd.svg
Liang
user
DaweiLiang
681da6375f701833274a0d26
Gengshen Zhang
681da6375f701833274a0d27
Dawei Leng
extracted_confirmed
2025-05-09T07:00:10.249Z
649935abbe8fd92c27ab1ed8
https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
David Leon
user
DavidLeon
681da6375f701833274a0d28
Yuhui Yin
FG-CLIP: Fine-Grained Visual and Textual Alignment
Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks
such as image-text retrieval and zero-shot classification but struggles with
fine-grained understanding due to its focus on coarse-grained short captions.
To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances
fine-grained understanding through three key innovations. First, we leverage
large multimodal models to generate 1.6 billion long caption-image pairs for
capturing global-level semantic details. Second, a high-quality dataset is
constructed with 12 million images and 40 million region-specific bounding
boxes aligned with detailed captions to ensure precise, context-rich
representations. Third, 10 million hard fine-grained negative samples are
incorporated to improve the model's ability to distinguish subtle semantic
differences. Corresponding training methods are meticulously designed for these
data. Extensive experiments demonstrate that FG-CLIP outperforms the original
CLIP and other state-of-the-art methods across various downstream tasks,
including fine-grained understanding, open-vocabulary object detection,
image-text retrieval, and general multimodal benchmarks. These results
highlight FG-CLIP's effectiveness in capturing fine-grained image details and
improving overall model performance. The related data, code, and models are
available at https://github.com/360CVGroup/FG-CLIP.
FG-CLIP enhances fine-grained understanding in multimodal tasks by leveraging large multimodal models, a high-quality dataset with detailed captions, and hard fine-grained negative samples.
Contrastive Language-Image Pre-training (CLIP) Fine-Grained CLIP (FG-CLIP) multimodal models caption-image pairs global-level semantic details high-quality dataset region-specific bounding boxes detailed captions fine-grained negative samples fine-grained understanding open-vocabulary object detection image-text retrieval multimodal benchmarks