Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
Most current LLM-based models for video understanding can process videos
within minutes. However, they struggle with lengthy videos due to challenges
such as "noise and redundancy", as well as "memory and computation"
constraints. In this paper, we present Goldfish, a methodology tailored for
comprehending videos of arbitrary lengths. We also introduce the TVQA-long
benchmark, specifically designed to evaluate models' capabilities in
understanding long videos with questions in both vision and text content.
Goldfish approaches these challenges with an efficient retrieval mechanism that
initially gathers the top-k video clips relevant to the instruction before
proceeding to provide the desired response. This design of the retrieval
mechanism enables the Goldfish to efficiently process arbitrarily long video
sequences, facilitating its application in contexts such as movies or
television series. To facilitate the retrieval process, we developed
MiniGPT4-Video that generates detailed descriptions for the video clips. In
addressing the scarcity of benchmarks for long video evaluation, we adapted the
TVQA short video benchmark for extended content analysis by aggregating
questions from entire episodes, thereby shifting the evaluation from partial to
full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long
benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows
exceptional performance in short video comprehension, exceeding existing
state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT,
TGIF, and TVQA short video benchmarks, respectively. These results indicate
that our models have significant improvements in both long and short-video
understanding. Our models and code have been made publicly available at
https://vision-cair.github.io/Goldfish_website/
669881c42f9cd07f44a94334
Kirolos Ataallah
669881c42f9cd07f44a94335
Xiaoqian Shen
claimed_verified
2024-10-25T09:31:44.007Z
620c5b20c33ca69c4c82145b
/avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
Xiaoqian Shen
user
shenxq
669881c42f9cd07f44a94336
Eslam Abdelrahman
669881c42f9cd07f44a94337
Essam Sleiman
669881c42f9cd07f44a94338
Mingchen Zhuge
669881c42f9cd07f44a94339
Jian Ding
669881c42f9cd07f44a9433a
Deyao Zhu
669881c42f9cd07f44a9433b
Jürgen Schmidhuber
669881c42f9cd07f44a9433c
Mohamed Elhoseiny
Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
Most current LLM-based models for video understanding can process videos
within minutes. However, they struggle with lengthy videos due to challenges
such as "noise and redundancy", as well as "memory and computation"
constraints. In this paper, we present Goldfish, a methodology tailored for
comprehending videos of arbitrary lengths. We also introduce the TVQA-long
benchmark, specifically designed to evaluate models' capabilities in
understanding long videos with questions in both vision and text content.
Goldfish approaches these challenges with an efficient retrieval mechanism that
initially gathers the top-k video clips relevant to the instruction before
proceeding to provide the desired response. This design of the retrieval
mechanism enables the Goldfish to efficiently process arbitrarily long video
sequences, facilitating its application in contexts such as movies or
television series. To facilitate the retrieval process, we developed
MiniGPT4-Video that generates detailed descriptions for the video clips. In
addressing the scarcity of benchmarks for long video evaluation, we adapted the
TVQA short video benchmark for extended content analysis by aggregating
questions from entire episodes, thereby shifting the evaluation from partial to
full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long
benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows
exceptional performance in short video comprehension, exceeding existing
state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT,
TGIF, and TVQA short video benchmarks, respectively. These results indicate
that our models have significant improvements in both long and short-video
understanding. Our models and code have been made publicly available at
https://vision-cair.github.io/Goldfish_website/
Goldfish introduces an efficient retrieval mechanism and MiniGPT4-Video for accurate and efficient understanding of both long and short videos.
LLM-based models video understanding noisy and redundant videos memory and computation constraints retrieval mechanism MiniGPT4-Video TVQA-long benchmark TVQA short video benchmark full episode comprehension short video comprehension