PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong
  Vision-language Adapter
This paper demonstrates that a progressively aligned language model can
effectively bridge frozen vision encoders and large language models (LLMs).
While the fundamental architecture and pre-training methods of vision encoders
and LLMs have been extensively studied, the architecture and training strategy
of vision-language adapters vary significantly across recent works. Our
research undertakes a thorough exploration of the state-of-the-art perceiver
resampler architecture and builds a strong baseline. However, we observe that
the vision-language alignment with perceiver resampler exhibits slow
convergence and limited scalability with a lack of direct supervision. To
address this issue, we propose PaLM2-VAdapter, employing a progressively
aligned language model as the vision-language adapter. Compared to the strong
baseline with perceiver resampler, our method empirically shows faster
convergence, higher performance, and stronger scalability. Extensive
experiments across various Visual Question Answering (VQA) and captioning tasks
on both images and videos demonstrate that our model exhibits state-of-the-art
visual understanding and multi-modal reasoning capabilities. Notably, our
method achieves these advancements with 30~70% fewer parameters than the
state-of-the-art large vision-language models, marking a significant efficiency
improvement.
65d2e7b5ff101ee25e170c66
Junfei Xiao
admin_assigned
2024-02-19T15:03:15.371Z
64b5ba6060274cbb296d6288
/avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
Junfei Xiao
user
lambertxiao
65d2e7b5ff101ee25e170c67
Zheng Xu
65d2e7b5ff101ee25e170c68
Alan Yuille
65d2e7b5ff101ee25e170c69
Shen Yan
65d2e7b5ff101ee25e170c6a
Boyu Wang
PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong
  Vision-language Adapter
This paper demonstrates that a progressively aligned language model can
effectively bridge frozen vision encoders and large language models (LLMs).
While the fundamental architecture and pre-training methods of vision encoders
and LLMs have been extensively studied, the architecture and training strategy
of vision-language adapters vary significantly across recent works. Our
research undertakes a thorough exploration of the state-of-the-art perceiver
resampler architecture and builds a strong baseline. However, we observe that
the vision-language alignment with perceiver resampler exhibits slow
convergence and limited scalability with a lack of direct supervision. To
address this issue, we propose PaLM2-VAdapter, employing a progressively
aligned language model as the vision-language adapter. Compared to the strong
baseline with perceiver resampler, our method empirically shows faster
convergence, higher performance, and stronger scalability. Extensive
experiments across various Visual Question Answering (VQA) and captioning tasks
on both images and videos demonstrate that our model exhibits state-of-the-art
visual understanding and multi-modal reasoning capabilities. Notably, our
method achieves these advancements with 30~70% fewer parameters than the
state-of-the-art large vision-language models, marking a significant efficiency
improvement.
A progressively aligned language model improves vision-language alignment, achieving faster convergence and higher performance with fewer parameters than state-of-the-art models.
progressively aligned language model vision encoders large language models LLMs perceiver resampler vision-language adapter Visual Question Answering VQA captioning tasks visual understanding multi-modal reasoning parameter-efficient