Video-LLaVA: Learning United Visual Representation by Alignment Before
  Projection
The Large Vision-Language Model (LVLM) has enhanced the performance of
various downstream tasks in visual-language understanding. Most existing
approaches encode images and videos into separate feature spaces, which are
then fed as inputs to large language models. However, due to the lack of
unified tokenization for images and videos, namely misalignment before
projection, it becomes challenging for a Large Language Model (LLM) to learn
multi-modal interactions from several poor projection layers. In this work, we
unify visual representation into the language feature space to advance the
foundational LLM towards a unified LVLM. As a result, we establish a simple but
robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images
and videos, mutually enhancing each other. Video-LLaVA achieves superior
performances on a broad range of 9 image benchmarks across 5 image
question-answering datasets and 4 image benchmark toolkits. Additionally, our
Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on
MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive
experiments demonstrate that Video-LLaVA mutually benefits images and videos
within a unified visual representation, outperforming models designed
specifically for images or videos.
655abd747d187b097e1176f6
Bin Lin
admin_assigned
2023-11-20T12:58:28.101Z
6367a8175bb06007ea099b8f
https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
linbin
user
LanguageBind
655abd747d187b097e1176f7
Bin Zhu
655abd747d187b097e1176f8
Yang Ye
655abd747d187b097e1176f9
Munan Ning
admin_assigned
2023-11-20T12:53:35.582Z
62a9748708a7ea93ff6d48ab
/avatars/c6358f81e31f7d31e14899011ce67e09.svg
Munan Ning
user
MasonNing
655abd747d187b097e1176fa
Peng Jin
claimed_verified
2024-10-17T11:32:00.414Z
651585ea18cfed0d30bee586
/avatars/579e468334102472d870875fe40302e6.svg
Peng Jin
user
Chat-UniVi
655abd747d187b097e1176fb
Li Yuan
claimed_verified
2025-04-04T07:13:23.198Z
66135a5e50350afe76beebce
/avatars/370a4b83949355feb050c2cb0425c264.svg
yl2488
user
yl2488
Video-LLaVA: Learning United Visual Representation by Alignment Before
  Projection
The Large Vision-Language Model (LVLM) has enhanced the performance of
various downstream tasks in visual-language understanding. Most existing
approaches encode images and videos into separate feature spaces, which are
then fed as inputs to large language models. However, due to the lack of
unified tokenization for images and videos, namely misalignment before
projection, it becomes challenging for a Large Language Model (LLM) to learn
multi-modal interactions from several poor projection layers. In this work, we
unify visual representation into the language feature space to advance the
foundational LLM towards a unified LVLM. As a result, we establish a simple but
robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images
and videos, mutually enhancing each other. Video-LLaVA achieves superior
performances on a broad range of 9 image benchmarks across 5 image
question-answering datasets and 4 image benchmark toolkits. Additionally, our
Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on
MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive
experiments demonstrate that Video-LLaVA mutually benefits images and videos
within a unified visual representation, outperforming models designed
specifically for images or videos.
Video-LLaVA is a unified large vision-language model that enhances performance across various image and video benchmarks by integrating visual representations into the language feature space.
Large Vision-Language Model LVLM unified tokenization multi-modal interactions Video-LLaVA mutual enhancement image benchmarks image question-answering datasets image benchmark toolkits Video-ChatGPT MSRVTT MSVD TGIF ActivityNet