FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model
Fine-grained vision-language understanding requires precise alignment between
visual content and linguistic descriptions, a capability that remains limited
in current models, particularly in non-English settings. While models like CLIP
perform well on global alignment, they often struggle to capture fine-grained
details in object attributes, spatial relations, and linguistic expressions,
with limited support for bilingual comprehension. To address these challenges,
we introduce FG-CLIP 2, a bilingual vision-language model designed to advance
fine-grained alignment for both English and Chinese. Our approach leverages
rich fine-grained supervision, including region-text matching and long-caption
modeling, alongside multiple discriminative objectives. We further introduce
the Textual Intra-modal Contrastive (TIC) loss to better distinguish
semantically similar captions. Trained on a carefully curated mixture of
large-scale English and Chinese data, FG-CLIP 2 achieves powerful bilingual
performance. To enable rigorous evaluation, we present a new benchmark for
Chinese multimodal understanding, featuring long-caption retrieval and bounding
box classification. Extensive experiments on 29 datasets across 8 tasks show
that FG-CLIP 2 outperforms existing methods, achieving state-of-the-art results
in both languages. We release the model, code, and benchmark to facilitate
future research on bilingual fine-grained alignment.
68edbbe5de1fee572713a84e
Chunyu Xie
68edbbe5de1fee572713a84f
Bin Wang
68edbbe5de1fee572713a850
Fanjing Kong
68edbbe5de1fee572713a851
Jincheng Li
68edbbe5de1fee572713a852
Dawei Liang
68edbbe5de1fee572713a853
Ji Ao
68edbbe5de1fee572713a854
Dawei Leng
68edbbe5de1fee572713a855
Yuhui Yin
FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model
Fine-grained vision-language understanding requires precise alignment between
visual content and linguistic descriptions, a capability that remains limited
in current models, particularly in non-English settings. While models like CLIP
perform well on global alignment, they often struggle to capture fine-grained
details in object attributes, spatial relations, and linguistic expressions,
with limited support for bilingual comprehension. To address these challenges,
we introduce FG-CLIP 2, a bilingual vision-language model designed to advance
fine-grained alignment for both English and Chinese. Our approach leverages
rich fine-grained supervision, including region-text matching and long-caption
modeling, alongside multiple discriminative objectives. We further introduce
the Textual Intra-modal Contrastive (TIC) loss to better distinguish
semantically similar captions. Trained on a carefully curated mixture of
large-scale English and Chinese data, FG-CLIP 2 achieves powerful bilingual
performance. To enable rigorous evaluation, we present a new benchmark for
Chinese multimodal understanding, featuring long-caption retrieval and bounding
box classification. Extensive experiments on 29 datasets across 8 tasks show
that FG-CLIP 2 outperforms existing methods, achieving state-of-the-art results
in both languages. We release the model, code, and benchmark to facilitate
future research on bilingual fine-grained alignment.
FG-CLIP 2, a bilingual vision-language model, enhances fine-grained alignment for English and Chinese through rich supervision and a new TIC loss, achieving state-of-the-art performance across multiple datasets and tasks.
CLIP FG-CLIP 2 fine-grained supervision region-text matching long-caption modeling multiple discriminative objectives Textual Intra-modal Contrastive (TIC) loss bilingual performance long-caption retrieval bounding box classification