CapsFusion: Rethinking Image-Text Data at Scale
Large multimodal models demonstrate remarkable generalist ability to perform
diverse multimodal tasks in a zero-shot manner. Large-scale web-based
image-text pairs contribute fundamentally to this success, but suffer from
excessive noise. Recent studies use alternative captions synthesized by
captioning models and have achieved notable benchmark performance. However, our
experiments reveal significant Scalability Deficiency and World Knowledge Loss
issues in models trained with synthetic captions, which have been largely
obscured by their initial benchmark success. Upon closer examination, we
identify the root cause as the overly-simplified language structure and lack of
knowledge details in existing synthetic captions. To provide higher-quality and
more scalable multimodal pretraining data, we propose CapsFusion, an advanced
framework that leverages large language models to consolidate and refine
information from both web-based image-text pairs and synthetic captions.
Extensive experiments show that CapsFusion captions exhibit remarkable
all-round superiority over existing captions in terms of model performance
(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
efficiency (requiring 11-16 times less computation than baselines), world
knowledge depth, and scalability. These effectiveness, efficiency and
scalability advantages position CapsFusion as a promising candidate for future
scaling of LMM training.
6541c65111e4836f27d46e88
Qiying Yu
admin_assigned
2023-11-02T14:26:16.051Z
6341375be5071b0c5cc946a3
/avatars/c26a32b54a3f7731854059d247b5c523.svg
Qiying Yu
user
qiying
6541c65111e4836f27d46e89
Quan Sun
admin_assigned
2023-11-02T14:57:06.789Z
630d7a8f81ef9b1772b67f4c
/avatars/00757abd6e548ccebb5bfb233be129a2.svg
Quan Sun
user
QuanSun
6541c65111e4836f27d46e8a
Xiaosong Zhang
admin_assigned
2023-11-02T14:55:59.849Z
64e473f0b78bc92221ab1883
/avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
Xiaosong Zhang
user
zhangxiaosong18
6541c65111e4836f27d46e8b
Yufeng Cui
admin_assigned
2023-11-02T14:56:06.913Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
6541c65111e4836f27d46e8c
Fan Zhang
claimed_verified
2024-10-08T07:23:09.979Z
640ed40dc025ddf618950af7
/avatars/a99abd4346fa649ad4144f284ebcc972.svg
Fan Zhang
user
ryanzhangfan
6541c65111e4836f27d46e8d
Xinlong Wang
admin_assigned
2023-11-02T14:56:41.820Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
6541c65111e4836f27d46e8e
Jingjing Liu
admin_assigned
2023-11-02T14:56:50.187Z
650bf18678d2dfbad85c0db1
/avatars/07ac9d726dc2ccf433746ee858a8044f.svg
jin liu
user
jingjingliu
CapsFusion: Rethinking Image-Text Data at Scale
Large multimodal models demonstrate remarkable generalist ability to perform
diverse multimodal tasks in a zero-shot manner. Large-scale web-based
image-text pairs contribute fundamentally to this success, but suffer from
excessive noise. Recent studies use alternative captions synthesized by
captioning models and have achieved notable benchmark performance. However, our
experiments reveal significant Scalability Deficiency and World Knowledge Loss
issues in models trained with synthetic captions, which have been largely
obscured by their initial benchmark success. Upon closer examination, we
identify the root cause as the overly-simplified language structure and lack of
knowledge details in existing synthetic captions. To provide higher-quality and
more scalable multimodal pretraining data, we propose CapsFusion, an advanced
framework that leverages large language models to consolidate and refine
information from both web-based image-text pairs and synthetic captions.
Extensive experiments show that CapsFusion captions exhibit remarkable
all-round superiority over existing captions in terms of model performance
(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
efficiency (requiring 11-16 times less computation than baselines), world
knowledge depth, and scalability. These effectiveness, efficiency and
scalability advantages position CapsFusion as a promising candidate for future
scaling of LMM training.
CapsFusion is an advanced framework that improves multimodal pretraining data by combining web-based image-text pairs and synthetic captions, leading to enhanced model performance, sample efficiency, and scalability.
multimodal models zero-shot image-text pairs captioning models Scalability Deficiency World Knowledge Loss large language models CapsFusion CIDEr score sample efficiency LMM training