Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
  Foundation Models
Recent advancements in multimodal models highlight the value of rewritten
captions for improving performance, yet key challenges remain. For example,
while synthetic captions often provide superior quality and image-text
alignment, it is not clear whether they can fully replace AltTexts: the role of
synthetic captions and their interaction with original web-crawled AltTexts in
pre-training is still not well understood. Moreover, different multimodal
foundation models may have unique preferences for specific caption formats, but
efforts to identify the optimal captions for each model remain limited. In this
work, we propose a novel, controllable, and scalable captioning pipeline
designed to generate diverse caption formats tailored to various multimodal
models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
Captions (DSC+) as case studies, we systematically explore their effects and
interactions with AltTexts across models such as CLIP, multimodal LLMs, and
diffusion models. Our findings reveal that a hybrid approach that keeps both
synthetic captions and AltTexts can outperform the use of synthetic captions
alone, improving both alignment and performance, with each model demonstrating
preferences for particular caption formats. This comprehensive analysis
provides valuable insights into optimizing captioning strategies, thereby
advancing the pre-training of multimodal foundation models.
66ff559f1e0b212adc0da561
Zhengfeng Lai
claimed_verified
2024-10-04T07:31:00.892Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
66ff559f1e0b212adc0da562
Vasileios Saveris
66ff559f1e0b212adc0da563
Chen Chen
claimed_verified
2025-03-16T21:23:46.017Z
65409e53d07194a81ecdd427
/avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
Chen Chen
user
ultra7chen
66ff559f1e0b212adc0da564
Hong-You Chen
claimed_verified
2024-10-04T07:31:04.373Z
66fb920d63d612c629bfea36
/avatars/86f5ccd4a15e8394792b422fefa64b69.svg
Chen
user
Hong-You
66ff559f1e0b212adc0da565
Haotian Zhang
claimed_verified
2024-10-04T07:31:02.875Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
66ff559f1e0b212adc0da566
Bowen Zhang
66ff559f1e0b212adc0da567
Juan Lao Tebar
66ff559f1e0b212adc0da568
Wenze Hu
66ff559f1e0b212adc0da569
Zhe Gan
66ff559f1e0b212adc0da56a
Peter Grasch
66ff559f1e0b212adc0da56b
Meng Cao
66ff559f1e0b212adc0da56c
Yinfei Yang
Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
  Foundation Models
Recent advancements in multimodal models highlight the value of rewritten
captions for improving performance, yet key challenges remain. For example,
while synthetic captions often provide superior quality and image-text
alignment, it is not clear whether they can fully replace AltTexts: the role of
synthetic captions and their interaction with original web-crawled AltTexts in
pre-training is still not well understood. Moreover, different multimodal
foundation models may have unique preferences for specific caption formats, but
efforts to identify the optimal captions for each model remain limited. In this
work, we propose a novel, controllable, and scalable captioning pipeline
designed to generate diverse caption formats tailored to various multimodal
models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
Captions (DSC+) as case studies, we systematically explore their effects and
interactions with AltTexts across models such as CLIP, multimodal LLMs, and
diffusion models. Our findings reveal that a hybrid approach that keeps both
synthetic captions and AltTexts can outperform the use of synthetic captions
alone, improving both alignment and performance, with each model demonstrating
preferences for particular caption formats. This comprehensive analysis
provides valuable insights into optimizing captioning strategies, thereby
advancing the pre-training of multimodal foundation models.
A study examines the use of synthetic and AltText captions in multimodal models, finding that a hybrid approach using both types improves alignment and performance across various models.
multimodal models Short Synthetic Captions (SSC) Dense Synthetic Captions (DSC+) CLIP multimodal LLMs diffusion models