An Inverse Scaling Law for CLIP Training
CLIP, the first foundation model that connects images and text, has enabled
many recent breakthroughs in computer vision. However, its associated training
cost is prohibitively high, imposing a significant barrier to its widespread
exploration. In this paper, we present a surprising finding that there exists
an inverse scaling law for CLIP training, whereby the larger the image/text
encoders used, the shorter the sequence length of image/text tokens that can be
applied in training. Moreover, we showcase that the strategy for reducing
image/text token length plays a crucial role in determining the quality of this
scaling law.
  As a result of this finding, we are able to successfully train CLIP even by
using academic resources. For example, on an A100 eight-GPU server, our CLIP
models achieve zero-shot top-1 ImageNet accuracies of 63.2% in ~2 days, 67.8%
in ~3 days, and 69.3% in ~4 days. By reducing the computation barrier
associated with CLIP, we hope to inspire more research in this field,
particularly from academics. Our code is available at
https://github.com/UCSC-VLAA/CLIPA.
645ddb74363a956edb13b363
Xianhang Li
claimed_verified
2023-05-12T17:13:25.211Z
645e72d4b5c9a8666d0ddc1c
/avatars/f0f1684731eb5ae148eadf841cceb18e.svg
xhl
user
Xianhang
645ddb74363a956edb13b364
Zeyu Wang
claimed_verified
2023-05-16T09:55:45.516Z
6462879777f63463fc5f75f9
/avatars/f19d14a009e940e3b565a6c62251599b.svg
Zeyu Wang
user
zw123
645ddb74363a956edb13b365
Cihang Xie
claimed_verified
2023-05-14T17:50:57.213Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
An Inverse Scaling Law for CLIP Training
CLIP, the first foundation model that connects images and text, has enabled
many recent breakthroughs in computer vision. However, its associated training
cost is prohibitively high, imposing a significant barrier to its widespread
exploration. In this paper, we present a surprising finding that there exists
an inverse scaling law for CLIP training, whereby the larger the image/text
encoders used, the shorter the sequence length of image/text tokens that can be
applied in training. Moreover, we showcase that the strategy for reducing
image/text token length plays a crucial role in determining the quality of this
scaling law.
  As a result of this finding, we are able to successfully train CLIP even by
using academic resources. For example, on an A100 eight-GPU server, our CLIP
models achieve zero-shot top-1 ImageNet accuracies of 63.2% in ~2 days, 67.8%
in ~3 days, and 69.3% in ~4 days. By reducing the computation barrier
associated with CLIP, we hope to inspire more research in this field,
particularly from academics. Our code is available at
https://github.com/UCSC-VLAA/CLIPA.
Reducing the token length in CLIP training improves scaling efficiency, allowing academic researchers to train high-performing models with limited resources.
CLIP foundation model inverse scaling law image/text encoders image/text tokens sequence length zero-shot ImageNet accuracy