What If We Recaption Billions of Web Images with LLaMA-3?
Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
that semantically aligning and enriching textual descriptions of these pairs
can significantly enhance model training across various vision-language tasks,
particularly text-to-image generation. However, large-scale investigations in
this area remain predominantly closed-source. Our paper aims to bridge this
community effort, leveraging the powerful and open-sourced LLaMA-3, a
GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
from the DataComp-1B dataset. Our empirical results confirm that this enhanced
dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
vision-language models. For discriminative models like CLIP, we observe
enhanced zero-shot performance in cross-modal retrieval tasks. For generative
models like text-to-image Diffusion Transformers, the generated images exhibit
a significant improvement in alignment with users' text instructions,
especially in following complex queries. Our project page is
https://www.haqtu.me/Recap-Datacomp-1B/
666a525aad291b6fa2567224
Xianhang Li
claimed_verified
2024-06-14T07:23:55.019Z
645e72d4b5c9a8666d0ddc1c
/avatars/f0f1684731eb5ae148eadf841cceb18e.svg
xhl
user
Xianhang
666a525aad291b6fa2567225
Haoqin Tu
admin_assigned
2024-06-13T19:53:20.158Z
604ae011caabafacfa48e3de
https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
Haoqin Tu
user
PahaII
666a525aad291b6fa2567226
Mude Hui
admin_assigned
2024-06-13T19:53:12.102Z
659f6f0de06dc8fc0e411224
/avatars/e16170f36b949c8697f277f6b52c51c3.svg
MudeHui
user
MudeHui
666a525aad291b6fa2567227
Zeyu Wang
claimed_verified
2024-06-14T07:23:52.589Z
6462879777f63463fc5f75f9
/avatars/f19d14a009e940e3b565a6c62251599b.svg
Zeyu Wang
user
zw123
666a525aad291b6fa2567228
Bingchen Zhao
admin_assigned
2024-06-13T19:52:43.007Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
666a525aad291b6fa2567229
Junfei Xiao
admin_assigned
2024-06-13T19:52:36.151Z
64b5ba6060274cbb296d6288
/avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
Junfei Xiao
user
lambertxiao
666a525aad291b6fa256722a
Sucheng Ren
claimed_verified
2025-02-28T12:18:12.526Z
65317ea1501804124f011950
/avatars/b055c3aba0c65d5377c69472e4576480.svg
Ren
user
OliverRen
666a525aad291b6fa256722b
Jieru Mei
admin_assigned
2024-06-13T19:52:22.423Z
652809fa7ad7a3460213ac0f
/avatars/5fc01ac91df2f03d213d05fa48109c67.svg
Jieru Mei
user
meijieru
666a525aad291b6fa256722c
Qing Liu
666a525aad291b6fa256722d
Huangjie Zheng
admin_assigned
2024-06-13T19:52:05.940Z
62f6dc90d278a8f3e7877dc0
/avatars/bce37487397278d5daf6d95f4dc25221.svg
Huangjie Zheng
user
hjzheng
666a525aad291b6fa256722e
Yuyin Zhou
666a525aad291b6fa256722f
Cihang Xie
admin_assigned
2024-06-13T19:51:47.209Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
What If We Recaption Billions of Web Images with LLaMA-3?
Web-crawled image-text pairs are inherently noisy. Prior studies demonstrate
that semantically aligning and enriching textual descriptions of these pairs
can significantly enhance model training across various vision-language tasks,
particularly text-to-image generation. However, large-scale investigations in
this area remain predominantly closed-source. Our paper aims to bridge this
community effort, leveraging the powerful and open-sourced LLaMA-3, a
GPT-4 level LLM. Our recaptioning pipeline is simple: first, we fine-tune a
LLaMA-3-8B powered LLaVA-1.5 and then employ it to recaption 1.3 billion images
from the DataComp-1B dataset. Our empirical results confirm that this enhanced
dataset, Recap-DataComp-1B, offers substantial benefits in training advanced
vision-language models. For discriminative models like CLIP, we observe
enhanced zero-shot performance in cross-modal retrieval tasks. For generative
models like text-to-image Diffusion Transformers, the generated images exhibit
a significant improvement in alignment with users' text instructions,
especially in following complex queries. Our project page is
https://www.haqtu.me/Recap-Datacomp-1B/
Enhancing image-text pairs through recaptioning with LLaMA-3 improves model performance in vision-language tasks, including zero-shot retrieval and text-to-image generation.
LLaMA-3 LLaVA-1.5 DataComp-1B Recap-DataComp-1B CLIP cross-modal retrieval text-to-image Diffusion Transformers