Leveraging Unpaired Data for Vision-Language Generative Models via Cycle
  Consistency
Current vision-language generative models rely on expansive corpora of paired
image-text data to attain optimal performance and generalization capabilities.
However, automatically collecting such data (e.g. via large-scale web scraping)
leads to low quality and poor image-text correlation, while human annotation is
more accurate but requires significant manual effort and expense. We introduce
ITIT (InTegrating Image
Text): an innovative training paradigm grounded in the concept of
cycle consistency which allows vision-language training on unpaired image and
text data. ITIT is comprised of a joint image-text encoder with disjoint image
and text decoders that enable bidirectional image-to-text and text-to-image
generation in a single framework. During training, ITIT leverages a small set
of paired image-text data to ensure its output matches the input reasonably
well in both directions. Simultaneously, the model is also trained on much
larger datasets containing only images or texts. This is achieved by enforcing
cycle consistency between the original unpaired samples and the cycle-generated
counterparts. For instance, it generates a caption for a given input image and
then uses the caption to create an output image, and enforces similarity
between the input and output images. Our experiments show that ITIT with
unpaired datasets exhibits similar scaling behavior as using high-quality
paired data. We demonstrate image generation and captioning performance on par
with state-of-the-art text-to-image and image-to-text models with orders of
magnitude fewer (only 3M) paired image-text data.
651f6664063e54a50cb5f182
Tianhong Li
extracted_confirmed
2023-10-09T18:18:56.166Z
65244352e0823fbfef8c6475
https://cdn-avatars.huggingface.co/v1/production/uploads/65244352e0823fbfef8c6475/d6YipDIYHdhP_B6RJACNc.jpeg
Tianhong Li
user
LTH14
651f6664063e54a50cb5f183
Sangnie Bhardwaj
claimed_verified
2023-10-11T07:34:46.710Z
6525a62969946cd5aa21717a
/avatars/81d382110a427b5541455ff947000a7f.svg
Sangnie Bhardwaj
user
sangnie
651f6664063e54a50cb5f184
Yonglong Tian
admin_assigned
2023-10-09T08:52:23.092Z
64793086ac77bb7603842456
/avatars/5891292138d372a9cc063f86c08b8174.svg
Yonglong Tian
user
tyl5566
651f6664063e54a50cb5f185
Han Zhang
651f6664063e54a50cb5f186
Jarred Barber
admin_assigned
2023-10-09T08:52:44.963Z
63136d5154e6e5d9f0f7643b
/avatars/6f3c8a0539da5964ca7e494178bd4f09.svg
Jarred
user
jarredbarber
651f6664063e54a50cb5f187
Dina Katabi
651f6664063e54a50cb5f188
Guillaume Lajoie
admin_assigned
2023-10-09T08:52:59.050Z
64909d1d6f217a26d91c6d9b
/avatars/28eeff49dd5c2f01b576e4a050551e17.svg
guillaume lajoie
user
glajoie
651f6664063e54a50cb5f189
Huiwen Chang
651f6664063e54a50cb5f18a
Dilip Krishnan
admin_assigned
2023-10-09T08:53:17.295Z
6501603c12c1442d99618aa5
/avatars/9c3d04da7fdb8b13b00b4486b75c9788.svg
Dilip Krishnan
user
dilipk8
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle
  Consistency
Current vision-language generative models rely on expansive corpora of paired
image-text data to attain optimal performance and generalization capabilities.
However, automatically collecting such data (e.g. via large-scale web scraping)
leads to low quality and poor image-text correlation, while human annotation is
more accurate but requires significant manual effort and expense. We introduce
ITIT (InTegrating Image
Text): an innovative training paradigm grounded in the concept of
cycle consistency which allows vision-language training on unpaired image and
text data. ITIT is comprised of a joint image-text encoder with disjoint image
and text decoders that enable bidirectional image-to-text and text-to-image
generation in a single framework. During training, ITIT leverages a small set
of paired image-text data to ensure its output matches the input reasonably
well in both directions. Simultaneously, the model is also trained on much
larger datasets containing only images or texts. This is achieved by enforcing
cycle consistency between the original unpaired samples and the cycle-generated
counterparts. For instance, it generates a caption for a given input image and
then uses the caption to create an output image, and enforces similarity
between the input and output images. Our experiments show that ITIT with
unpaired datasets exhibits similar scaling behavior as using high-quality
paired data. We demonstrate image generation and captioning performance on par
with state-of-the-art text-to-image and image-to-text models with orders of
magnitude fewer (only 3M) paired image-text data.
The ITIT model uses cycle consistency to train vision-language generative models on unpaired data, achieving performance comparable to those trained on large labeled datasets with significantly less paired data.
cycle consistency joint image-text encoder disjoint image and text decoders bidirectional image-to-text and text-to-image generation unpaired datasets similarity enforcement image generation captioning performance