COSMO: COntrastive Streamlined MultimOdal Model with Interleaved
  Pre-Training
In the evolution of Vision-Language Pre-training, shifting from short-text
comprehension to encompassing extended textual contexts is pivotal. Recent
autoregressive vision-language models like flamingo, palme, leveraging
the long-context capability of Large Language Models, have excelled in few-shot
text generation tasks but face challenges in alignment tasks. Addressing this
gap, we introduce the contrastive loss into text generation models, presenting
the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically
partitioning the language model into dedicated unimodal text processing and
adept multimodal data handling components. \ModelName, our unified framework,
merges unimodal and multimodal elements, enhancing model performance for tasks
involving textual and visual data while notably reducing learnable parameters.
However, these models demand extensive long-text datasets, yet the availability
of high-quality long-text video datasets remains limited. To bridge this gap,
this work introduces \VideoDatasetName, an inaugural interleaved video-text
dataset featuring comprehensive captions, marking a significant step forward.
Demonstrating its impact, we illustrate how  enhances model
performance in image-text tasks. With 34% learnable parameters and utilizing
72\% of the available data, our model demonstrates significant superiority over
OpenFlamingo~openflamingo. For instance, in the 4-shot flickr captioning
task, performance notably improves from 57.2% to 65.\%. The contributions of
 and  are underscored by notable performance
gains across 14 diverse downstream datasets encompassing both image-text and
video-text tasks.
6593966b3533188375b72db1
Alex Jinpeng Wang
6593966b3533188375b72db2
Linjie Li
6593966b3533188375b72db3
Kevin Qinghong Lin
claimed_verified
2024-01-04T09:34:33.745Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
6593966b3533188375b72db4
Jianfeng Wang
6593966b3533188375b72db5
Kevin Lin
6593966b3533188375b72db6
Zhengyuan Yang
6593966b3533188375b72db7
Lijuan Wang
6593966b3533188375b72db8
Mike Zheng Shou
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved
  Pre-Training
In the evolution of Vision-Language Pre-training, shifting from short-text
comprehension to encompassing extended textual contexts is pivotal. Recent
autoregressive vision-language models like flamingo, palme, leveraging
the long-context capability of Large Language Models, have excelled in few-shot
text generation tasks but face challenges in alignment tasks. Addressing this
gap, we introduce the contrastive loss into text generation models, presenting
the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically
partitioning the language model into dedicated unimodal text processing and
adept multimodal data handling components. \ModelName, our unified framework,
merges unimodal and multimodal elements, enhancing model performance for tasks
involving textual and visual data while notably reducing learnable parameters.
However, these models demand extensive long-text datasets, yet the availability
of high-quality long-text video datasets remains limited. To bridge this gap,
this work introduces \VideoDatasetName, an inaugural interleaved video-text
dataset featuring comprehensive captions, marking a significant step forward.
Demonstrating its impact, we illustrate how  enhances model
performance in image-text tasks. With 34% learnable parameters and utilizing
72\% of the available data, our model demonstrates significant superiority over
OpenFlamingo~openflamingo. For instance, in the 4-shot flickr captioning
task, performance notably improves from 57.2% to 65.\%. The contributions of
 and  are underscored by notable performance
gains across 14 diverse downstream datasets encompassing both image-text and
video-text tasks.
A contrastive loss and unified multimodal framework improve vision-language pre-training models, particularly in tasks involving long textual contexts and extended video data, outperforming existing models and utilizing a novel interleaved video-text dataset.
autoregressive vision-language models contrastive loss multimodal framework unimodal text processing multimodal data handling interleaved video-text dataset learned parameters flickr captioning task