Improving fine-grained understanding in image-text pre-training
We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
method for pretraining more fine-grained multimodal representations from
image-text pairs. Given that multiple image patches often correspond to single
words, we propose to learn a grouping of image patches for every token in the
caption. To achieve this, we use a sparse similarity metric between image
patches and language tokens and compute for each token a language-grouped
vision embedding as the weighted average of patches. The token and
language-grouped vision embeddings are then contrasted through a fine-grained
sequence-wise loss that only depends on individual samples and does not require
other batch samples as negatives. This enables more detailed information to be
learned in a computationally inexpensive manner. SPARC combines this
fine-grained loss with a contrastive loss between global image and text
embeddings to learn representations that simultaneously encode global and local
information. We thoroughly evaluate our proposed method and show improved
performance over competing approaches both on image-level tasks relying on
coarse-grained information, e.g. classification, as well as region-level tasks
relying on fine-grained information, e.g. retrieval, object detection, and
segmentation. Moreover, SPARC improves model faithfulness and captioning in
foundational vision-language models.
65a9e02ae15948f4c6b25dc7
Ioana Bica
65a9e02ae15948f4c6b25dc8
Anastasija Ilić
65a9e02ae15948f4c6b25dc9
Matthias Bauer
65a9e02ae15948f4c6b25dca
Goker Erdogan
65a9e02ae15948f4c6b25dcb
Matko Bošnjak
65a9e02ae15948f4c6b25dcc
Christos Kaplanis
65a9e02ae15948f4c6b25dcd
Alexey A. Gritsenko
65a9e02ae15948f4c6b25dce
Matthias Minderer
admin_assigned
2024-01-19T13:27:42.915Z
649ac6e57c36fc2dc6e6b0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
Matthias Minderer
user
mjlm
65a9e02ae15948f4c6b25dcf
Charles Blundell
65a9e02ae15948f4c6b25dd0
Razvan Pascanu
admin_assigned
2024-01-19T13:27:54.801Z
64b9310403124195cd9778ec
/avatars/57c594d3d0f97d3010b15b6a0806451c.svg
Razvan Pascanu
user
razp
65a9e02ae15948f4c6b25dd1
Jovana Mitrović
Improving fine-grained understanding in image-text pre-training
We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
method for pretraining more fine-grained multimodal representations from
image-text pairs. Given that multiple image patches often correspond to single
words, we propose to learn a grouping of image patches for every token in the
caption. To achieve this, we use a sparse similarity metric between image
patches and language tokens and compute for each token a language-grouped
vision embedding as the weighted average of patches. The token and
language-grouped vision embeddings are then contrasted through a fine-grained
sequence-wise loss that only depends on individual samples and does not require
other batch samples as negatives. This enables more detailed information to be
learned in a computationally inexpensive manner. SPARC combines this
fine-grained loss with a contrastive loss between global image and text
embeddings to learn representations that simultaneously encode global and local
information. We thoroughly evaluate our proposed method and show improved
performance over competing approaches both on image-level tasks relying on
coarse-grained information, e.g. classification, as well as region-level tasks
relying on fine-grained information, e.g. retrieval, object detection, and
segmentation. Moreover, SPARC improves model faithfulness and captioning in
foundational vision-language models.
SPARC pretrains fine-grained multimodal representations using sparse contrastive alignment, improving both global and local information in vision-language models and enhancing performance on classification and region-level tasks.
SPARC fine-grained multimodal representations image-text pairs image patches language tokens sparse similarity metric vision embedding fine-grained sequence-wise loss contrastive loss global image embeddings text embeddings faithfulness captioning