Should VLMs be Pre-trained with Image Data?
Pre-trained LLMs that are further trained with image data perform well on
vision-language tasks. While adding images during a second training phase
effectively unlocks this capability, it is unclear how much of a gain or loss
this two-step pipeline gives over VLMs which integrate images earlier into the
training process. To investigate this, we train models spanning various
datasets, scales, image-text ratios, and amount of pre-training done before
introducing vision tokens. We then fine-tune these models and evaluate their
downstream performance on a suite of vision-language and text-only tasks. We
find that pre-training with a mixture of image and text data allows models to
perform better on vision-language tasks while maintaining strong performance on
text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B
model, introducing visual tokens 80% of the way through pre-training results in
a 2% average improvement over introducing visual tokens to a fully pre-trained
model.
67cfc310f2b1fe815dc24ebf
Sedrick Keh
admin_assigned
2025-03-11T14:52:53.915Z
6365c1158b6bd0d9f6605d3d
/avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
Sedrick Keh
user
sedrickkeh
67cfc310f2b1fe815dc24ec0
Jean Mercat
admin_assigned
2025-03-11T14:53:00.187Z
62be399f01dc22b4d22fc990
/avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
Jean Mercat
user
jmercat
67cfc310f2b1fe815dc24ec1
Samir Yitzhak Gadre
admin_assigned
2025-03-11T14:53:10.097Z
62be1cd77c0388414f012f23
https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
Samir Gadre
user
samirg
67cfc310f2b1fe815dc24ec2
Kushal Arora
67cfc310f2b1fe815dc24ec3
Igor Vasiljevic
admin_assigned
2025-03-11T14:54:33.031Z
633b2d317af633cbcd03a53e
https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
Igor Vasiljevic
user
ivas-tri
67cfc310f2b1fe815dc24ec4
Benjamin Burchfiel
67cfc310f2b1fe815dc24ec5
Shuran Song
67cfc310f2b1fe815dc24ec6
Russ Tedrake
admin_assigned
2025-03-11T14:53:34.722Z
668924108c440fe1951e2d2a
/avatars/97e60f1ecb0b7d8f7238a4e92944e370.svg
Russ Tedrake
user
RussTedrake
67cfc310f2b1fe815dc24ec7
Thomas Kollar
admin_assigned
2025-03-11T14:53:47.408Z
64358e662d0ed796668cb438
/avatars/6f86547d8691e061e289dd906deedfdb.svg
Thomas Kollar
user
thomas-kollar
67cfc310f2b1fe815dc24ec8
Ludwig Schmidt
admin_assigned
2025-03-11T14:54:00.333Z
6252375c818a5dc29ab9d945
/avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
Ludwig Schmidt
user
ludwigschmidt
67cfc310f2b1fe815dc24ec9
Achal Dave
admin_assigned
2025-03-11T14:54:08.423Z
6488e29b78570cedad71eb2d
/avatars/0d8bb52c1e2be8d6aa7c732cf92fc5a9.svg
Achal Dave
user
achal-tri
Should VLMs be Pre-trained with Image Data?
Pre-trained LLMs that are further trained with image data perform well on
vision-language tasks. While adding images during a second training phase
effectively unlocks this capability, it is unclear how much of a gain or loss
this two-step pipeline gives over VLMs which integrate images earlier into the
training process. To investigate this, we train models spanning various
datasets, scales, image-text ratios, and amount of pre-training done before
introducing vision tokens. We then fine-tune these models and evaluate their
downstream performance on a suite of vision-language and text-only tasks. We
find that pre-training with a mixture of image and text data allows models to
perform better on vision-language tasks while maintaining strong performance on
text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B
model, introducing visual tokens 80% of the way through pre-training results in
a 2% average improvement over introducing visual tokens to a fully pre-trained
model.
Pre-training large language models with both image and text data improves vision-language performance without degrading text-only performance.
pre-trained LLMs vision-language tasks image-text ratios vision tokens downstream performance