VILA^2: VILA Augmented VILA
Visual language models (VLMs) have rapidly progressed, driven by the success
of large language models (LLMs). While model architectures and training
infrastructures advance rapidly, data curation remains under-explored. When
data quantity and quality become a bottleneck, existing work either directly
crawls more raw data from the Internet that does not have a guarantee of data
quality or distills from black-box commercial models (e.g., GPT-4V / Gemini)
causing the performance upper bounded by that model. In this work, we introduce
a novel approach that includes a self-augment step and a specialist-augment
step to iteratively improve data quality and model performance. In the
self-augment step, a VLM recaptions its own pretraining data to enhance data
quality, and then retrains from scratch using this refined dataset to improve
model performance. This process can iterate for several rounds. Once
self-augmentation saturates, we employ several specialist VLMs finetuned from
the self-augmented VLM with domain-specific expertise, to further infuse
specialist knowledge into the generalist VLM through task-oriented recaptioning
and retraining. With the combined self-augmented and specialist-augmented
training, we introduce VILA^2 (VILA-augmented-VILA), a VLM family that
consistently improves the accuracy on a wide range of tasks over prior art, and
achieves new state-of-the-art results on MMMU leaderboard among open-sourced
models.
66a1bbb7ca8ee359d60e5118
Yunhao Fang
claimed_verified
2024-07-25T07:38:26.772Z
640d68938aee167ccda391da
/avatars/8e3fbf6ca10fe4e9b04a3a84d4e3e255.svg
Yunhao Fang
user
Seerkfang
66a1bbb7ca8ee359d60e5119
Ligeng Zhu
admin_assigned
2024-07-25T08:19:26.721Z
62b4b5beb25cb80fcf278354
https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
Ligeng Zhu
user
Ligeng-Zhu
66a1bbb7ca8ee359d60e511a
Yao Lu
admin_assigned
2024-07-25T08:19:39.594Z
64119ace518736207a51625a
https://cdn-avatars.huggingface.co/v1/production/uploads/64119ace518736207a51625a/bJ-hyscgoU7a9gs-br0H7.jpeg
Yao Lu
user
yaolu
66a1bbb7ca8ee359d60e511b
Yan Wang
claimed_verified
2024-07-26T07:40:37.179Z
64b9c32c2ab555ea562c736c
/avatars/b22f6d77aaeb1e6dd34aa3c4d53f1e3e.svg
Yan Wang
user
yyyanw
66a1bbb7ca8ee359d60e511c
Pavlo Molchanov
claimed_verified
2024-09-30T08:08:12.215Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
66a1bbb7ca8ee359d60e511d
Jang Hyun Cho
admin_assigned
2024-07-25T08:20:49.312Z
647a4ae6c6077fb66fbd0036
/avatars/ee08791366311da83bd5395012f3b1c2.svg
Jang Hyun (Vincent) Cho
user
janghyuncho7
66a1bbb7ca8ee359d60e511e
Marco Pavone
66a1bbb7ca8ee359d60e511f
Song Han
admin_assigned
2024-07-25T08:21:16.081Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
66a1bbb7ca8ee359d60e5120
Hongxu Yin
admin_assigned
2024-07-25T08:21:26.595Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
VILA^2: VILA Augmented VILA
Visual language models (VLMs) have rapidly progressed, driven by the success
of large language models (LLMs). While model architectures and training
infrastructures advance rapidly, data curation remains under-explored. When
data quantity and quality become a bottleneck, existing work either directly
crawls more raw data from the Internet that does not have a guarantee of data
quality or distills from black-box commercial models (e.g., GPT-4V / Gemini)
causing the performance upper bounded by that model. In this work, we introduce
a novel approach that includes a self-augment step and a specialist-augment
step to iteratively improve data quality and model performance. In the
self-augment step, a VLM recaptions its own pretraining data to enhance data
quality, and then retrains from scratch using this refined dataset to improve
model performance. This process can iterate for several rounds. Once
self-augmentation saturates, we employ several specialist VLMs finetuned from
the self-augmented VLM with domain-specific expertise, to further infuse
specialist knowledge into the generalist VLM through task-oriented recaptioning
and retraining. With the combined self-augmented and specialist-augmented
training, we introduce VILA^2 (VILA-augmented-VILA), a VLM family that
consistently improves the accuracy on a wide range of tasks over prior art, and
achieves new state-of-the-art results on MMMU leaderboard among open-sourced
models.
A novel data augmentation approach iteratively improves visual language model data quality and performance using self-augmentation and specialist-augmentation, leading to state-of-the-art results on MMMU tasks.
visual language models large language models data curation self-augment specialist-augment recaptions domain-specific expertise task-oriented recaptioning