TVBench: Redesigning Video-Language Evaluation
Large language models have demonstrated impressive performance when
integrated with vision models even enabling video understanding. However,
evaluating these video models presents its own unique challenges, for which
several benchmarks have been proposed. In this paper, we show that the
currently most used video-language benchmarks can be solved without requiring
much temporal reasoning. We identified three main issues in existing datasets:
(i) static information from single frames is often sufficient to solve the
tasks (ii) the text of the questions and candidate answers is overly
informative, allowing models to answer correctly without relying on any visual
input (iii) world knowledge alone can answer many of the questions, making the
benchmarks a test of knowledge replication rather than visual reasoning. In
addition, we found that open-ended question-answering benchmarks for video
understanding suffer from similar issues while the automatic evaluation process
with LLMs is unreliable, making it an unsuitable alternative. As a solution, we
propose TVBench, a novel open-source video multiple-choice question-answering
benchmark, and demonstrate through extensive evaluations that it requires a
high level of temporal understanding. Surprisingly, we find that most recent
state-of-the-art video-language models perform similarly to random performance
on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.
6708e94c1f9b09c1ab200e8a
Daniel Cores
claimed_verified
2024-10-15T09:11:20.594Z
6616922bff43b65a8ece7ccc
/avatars/983799cedcad7c75a57f215630e7ab4a.svg
Daniel Cores
user
dcores
6708e94c1f9b09c1ab200e8b
Michael Dorkenwald
claimed_verified
2024-10-14T12:16:07.727Z
6566436e3d526e3297c1b4a1
/avatars/e433ffccb0f9c2d91649f9988655fedd.svg
Michael Dorkenwald
user
mdorkenw
6708e94c1f9b09c1ab200e8c
Manuel Mucientes
6708e94c1f9b09c1ab200e8d
Cees G. M. Snoek
6708e94c1f9b09c1ab200e8e
Yuki M. Asano
claimed_verified
2024-10-15T09:11:22.379Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
TVBench: Redesigning Video-Language Evaluation
Large language models have demonstrated impressive performance when
integrated with vision models even enabling video understanding. However,
evaluating these video models presents its own unique challenges, for which
several benchmarks have been proposed. In this paper, we show that the
currently most used video-language benchmarks can be solved without requiring
much temporal reasoning. We identified three main issues in existing datasets:
(i) static information from single frames is often sufficient to solve the
tasks (ii) the text of the questions and candidate answers is overly
informative, allowing models to answer correctly without relying on any visual
input (iii) world knowledge alone can answer many of the questions, making the
benchmarks a test of knowledge replication rather than visual reasoning. In
addition, we found that open-ended question-answering benchmarks for video
understanding suffer from similar issues while the automatic evaluation process
with LLMs is unreliable, making it an unsuitable alternative. As a solution, we
propose TVBench, a novel open-source video multiple-choice question-answering
benchmark, and demonstrate through extensive evaluations that it requires a
high level of temporal understanding. Surprisingly, we find that most recent
state-of-the-art video-language models perform similarly to random performance
on TVBench, with only Gemini-Pro and Tarsier clearly surpassing this baseline.
A new video-language benchmark, TVBench, is introduced to require high-level temporal understanding in video question-answering, showing that most state-of-the-art models do not perform well on this task.
video-language models video understanding temporal reasoning TVBench open-source benchmark multiple-choice question-answering Gemini-Pro Tarsier