MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with
  Interleaved Visual-Textual Tokens
This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
designed specifically for video understanding. The model is capable of
processing both temporal visual and textual data, making it adept at
understanding the complexities of videos. Building upon the success of
MiniGPT-v2, which excelled in translating visual features into the LLM space
for single images and achieved impressive results on various image-text
benchmarks, this paper extends the model's capabilities to process a sequence
of frames, enabling it to comprehend videos. MiniGPT4-video does not only
consider visual content but also incorporates textual conversations, allowing
the model to effectively answer queries involving both visual and text
components. The proposed model outperforms existing state-of-the-art methods,
registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF,
and TVQA benchmarks respectively. Our models and code have been made publicly
available here https://vision-cair.github.io/MiniGPT4-video/
660f6878ccc61eea055c67ee
Kirolos Ataallah
660f6878ccc61eea055c67ef
Xiaoqian Shen
claimed_verified
2024-10-25T09:31:59.671Z
620c5b20c33ca69c4c82145b
/avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
Xiaoqian Shen
user
shenxq
660f6878ccc61eea055c67f0
Eslam Abdelrahman
660f6878ccc61eea055c67f1
Essam Sleiman
660f6878ccc61eea055c67f2
Deyao Zhu
660f6878ccc61eea055c67f3
Jian Ding
660f6878ccc61eea055c67f4
Mohamed Elhoseiny
MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with
  Interleaved Visual-Textual Tokens
This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
designed specifically for video understanding. The model is capable of
processing both temporal visual and textual data, making it adept at
understanding the complexities of videos. Building upon the success of
MiniGPT-v2, which excelled in translating visual features into the LLM space
for single images and achieved impressive results on various image-text
benchmarks, this paper extends the model's capabilities to process a sequence
of frames, enabling it to comprehend videos. MiniGPT4-video does not only
consider visual content but also incorporates textual conversations, allowing
the model to effectively answer queries involving both visual and text
components. The proposed model outperforms existing state-of-the-art methods,
registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF,
and TVQA benchmarks respectively. Our models and code have been made publicly
available here https://vision-cair.github.io/MiniGPT4-video/
MiniGPT4-Video, an extension of MiniGPT-v2, processes sequences of video frames and textual conversations, outperforming state-of-the-art models on multiple video-text benchmarks.
multimodal Large Language Model video understanding MiniGPT-v2 temporal visual data textual data MSVD MSRVTT TGIF TVQA