Three Towers: Flexible Contrastive Learning with Pretrained Image Models
We introduce Three Towers (3T), a flexible method to improve the contrastive
learning of vision-language models by incorporating pretrained image
classifiers. While contrastive models are usually trained from scratch, LiT
(Zhai et al., 2022) has recently shown performance gains from using pretrained
classifier embeddings. However, LiT directly replaces the image tower with the
frozen embeddings, excluding any potential benefits of contrastively training
the image tower. With 3T, we propose a more flexible strategy that allows the
image tower to benefit from both pretrained embeddings and contrastive
training. To achieve this, we introduce a third tower that contains the frozen
pretrained embeddings, and we encourage alignment between this third tower and
the main image-text towers. Empirically, 3T consistently improves over LiT and
the CLIP-style from-scratch baseline for retrieval tasks. For classification,
3T reliably improves over the from-scratch baseline, and while it underperforms
relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
and Places365 pretraining.
64741f210da364bd0d00d0fb
Jannik Kossen
claimed_verified
2024-06-14T15:40:37.533Z
64df30f2e672e4d2c57049e8
/avatars/8c9503e43f403d655b11ea5c6e23ae79.svg
Jannik
user
jlko
64741f210da364bd0d00d0fc
Mark Collier
64741f210da364bd0d00d0fd
Basil Mustafa
admin_assigned
2023-05-31T09:13:52.460Z
63cfcad6e23b90128c66685c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
Basil Mustafa
user
BasilMustafa
64741f210da364bd0d00d0fe
Xiao Wang
64741f210da364bd0d00d0ff
Xiaohua Zhai
64741f210da364bd0d00d100
Lucas Beyer
admin_assigned
2023-05-31T09:14:24.187Z
642d334ff65714b4585f2de4
https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
Lucas Beyer
user
giffmana
64741f210da364bd0d00d101
Andreas Steiner
64741f210da364bd0d00d102
Jesse Berent
64741f210da364bd0d00d103
Rodolphe Jenatton
64741f210da364bd0d00d104
Efi Kokiopoulou
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
We introduce Three Towers (3T), a flexible method to improve the contrastive
learning of vision-language models by incorporating pretrained image
classifiers. While contrastive models are usually trained from scratch, LiT
(Zhai et al., 2022) has recently shown performance gains from using pretrained
classifier embeddings. However, LiT directly replaces the image tower with the
frozen embeddings, excluding any potential benefits of contrastively training
the image tower. With 3T, we propose a more flexible strategy that allows the
image tower to benefit from both pretrained embeddings and contrastive
training. To achieve this, we introduce a third tower that contains the frozen
pretrained embeddings, and we encourage alignment between this third tower and
the main image-text towers. Empirically, 3T consistently improves over LiT and
the CLIP-style from-scratch baseline for retrieval tasks. For classification,
3T reliably improves over the from-scratch baseline, and while it underperforms
relative to LiT for JFT-pretrained models, it outperforms LiT for ImageNet-21k
and Places365 pretraining.
Three Towers combines pretrained image classifiers with contrastive learning to enhance vision-language models, offering better performance across multiple retrieval and classification tasks.
Three Towers 3T contrastive learning vision-language models pretrained image classifiers image tower frozen embeddings main image-text towers retrieval tasks classification JFT-pretrained models ImageNet-21k Places365