VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio
  Understanding in Video-LLMs
In this paper, we present the VideoLLaMA 2, a set of Video Large Language
Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio
understanding in video and audio-oriented tasks. Building upon its predecessor,
VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC)
connector, which effectively captures the intricate spatial and temporal
dynamics of video data. Additionally, we integrate an Audio Branch into the
model through joint training, thereby enriching the multimodal understanding
capabilities of the model by seamlessly incorporating audio cues. Comprehensive
evaluations on multiple-choice video question answering (MC-VQA), open-ended
video question answering (OE-VQA), and video captioning (VC) tasks demonstrate
that VideoLLaMA 2 consistently achieves competitive results among open-source
models and even gets close to some proprietary models on several benchmarks.
Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and
audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models.
These advancements underline VideoLLaMA 2's superior performance in multimodal
comprehension, setting a new standard for intelligent video analysis systems.
All models are public to facilitate further research.
66692323d993003c25001828
Zesen Cheng
admin_assigned
2024-06-13T13:05:24.153Z
65b2529285b6c21448a10d65
/avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
Zesen Cheng
user
ClownRat
66692323d993003c25001829
Sicong Leng
admin_assigned
2024-06-13T13:05:34.956Z
609115c79a8bcaa437b234a9
/avatars/1631a91030703d8397133363cf82c863.svg
Leng Sicong
user
Sicong
66692323d993003c2500182a
Hang Zhang
66692323d993003c2500182b
Yifei Xin
66692323d993003c2500182c
Xin Li
claimed_verified
2024-06-13T09:31:47.064Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
66692323d993003c2500182d
Guanzheng Chen
66692323d993003c2500182e
Yongxin Zhu
claimed_verified
2024-07-12T07:41:53.168Z
637c7d8a88699fba70e1e1ff
/avatars/e7f60fab6ca9ad5ecd320bd7cd51ce2e.svg
yongxinzhu
user
youngsheen
66692323d993003c2500182f
Wenqi Zhang
66692323d993003c25001830
Ziyang Luo
admin_assigned
2024-06-13T13:07:20.220Z
6090ff099a8bcaa437b234a4
https://cdn-avatars.huggingface.co/v1/production/uploads/6090ff099a8bcaa437b234a4/iUvw7JXT-ngI7rGk1x-io.jpeg
Ziyang Luo
user
Ziyang
66692323d993003c25001831
Deli Zhao
66692323d993003c25001832
Lidong Bing
admin_assigned
2024-06-13T13:07:33.945Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio
  Understanding in Video-LLMs
In this paper, we present the VideoLLaMA 2, a set of Video Large Language
Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio
understanding in video and audio-oriented tasks. Building upon its predecessor,
VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC)
connector, which effectively captures the intricate spatial and temporal
dynamics of video data. Additionally, we integrate an Audio Branch into the
model through joint training, thereby enriching the multimodal understanding
capabilities of the model by seamlessly incorporating audio cues. Comprehensive
evaluations on multiple-choice video question answering (MC-VQA), open-ended
video question answering (OE-VQA), and video captioning (VC) tasks demonstrate
that VideoLLaMA 2 consistently achieves competitive results among open-source
models and even gets close to some proprietary models on several benchmarks.
Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and
audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models.
These advancements underline VideoLLaMA 2's superior performance in multimodal
comprehension, setting a new standard for intelligent video analysis systems.
All models are public to facilitate further research.
VideoLLaMA 2, an enhanced Video Large Language Model, incorporates Spatial-Temporal Convolution and audio cues to achieve competitive performance in multimodal video and audio tasks.
Spatial-Temporal Convolution Video Large Language Models Video-LLMs multimodal understanding multiple-choice video question answering open-ended video question answering video captioning audio-only question answering audio-video question-answering