LongVideoBench: A Benchmark for Long-context Interleaved Video-Language
  Understanding
Large multimodal models (LMMs) are processing increasingly longer and richer
inputs. Albeit the progress, few public benchmark is available to measure such
development. To mitigate this gap, we introduce LongVideoBench, a
question-answering benchmark that features video-language interleaved inputs up
to an hour long. Our benchmark includes 3,763 varying-length web-collected
videos with their subtitles across diverse themes, designed to comprehensively
evaluate LMMs on long-term multimodal understanding. To achieve this, we
interpret the primary challenge as to accurately retrieve and reason over
detailed multimodal information from long inputs. As such, we formulate a novel
video question-answering task termed referring reasoning. Specifically, as part
of the question, it contains a referring query that references related video
contexts, called referred context. The model is then required to reason over
relevant video details from the referred context. Following the paradigm of
referring reasoning, we curate 6,678 human-annotated multiple-choice questions
in 17 fine-grained categories, establishing one of the most comprehensive
benchmarks for long-form video understanding. Evaluations suggest that the
LongVideoBench presents significant challenges even for the most advanced
proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their
open-source counterparts show an even larger performance gap. In addition, our
results indicate that model performance on the benchmark improves only when
they are capable of processing more frames, positioning LongVideoBench as a
valuable benchmark for evaluating future-generation long-context LMMs.
669f43ba04a09709a6b378f8
Haoning Wu
claimed_verified
2024-07-23T07:47:21.755Z
63047ed2412a1b9d381b09c9
https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
Haoning Wu, Teo
user
teowu
669f43ba04a09709a6b378f9
Dongxu Li
admin_assigned
2024-07-23T08:22:56.086Z
6220388b354ac333eb998319
/avatars/e04ebeb57f9dc9aa2f6d4732737f6cb8.svg
Dongxu Li
user
dxli
669f43ba04a09709a6b378fa
Bei Chen
669f43ba04a09709a6b378fb
Junnan Li
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language
  Understanding
Large multimodal models (LMMs) are processing increasingly longer and richer
inputs. Albeit the progress, few public benchmark is available to measure such
development. To mitigate this gap, we introduce LongVideoBench, a
question-answering benchmark that features video-language interleaved inputs up
to an hour long. Our benchmark includes 3,763 varying-length web-collected
videos with their subtitles across diverse themes, designed to comprehensively
evaluate LMMs on long-term multimodal understanding. To achieve this, we
interpret the primary challenge as to accurately retrieve and reason over
detailed multimodal information from long inputs. As such, we formulate a novel
video question-answering task termed referring reasoning. Specifically, as part
of the question, it contains a referring query that references related video
contexts, called referred context. The model is then required to reason over
relevant video details from the referred context. Following the paradigm of
referring reasoning, we curate 6,678 human-annotated multiple-choice questions
in 17 fine-grained categories, establishing one of the most comprehensive
benchmarks for long-form video understanding. Evaluations suggest that the
LongVideoBench presents significant challenges even for the most advanced
proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their
open-source counterparts show an even larger performance gap. In addition, our
results indicate that model performance on the benchmark improves only when
they are capable of processing more frames, positioning LongVideoBench as a
valuable benchmark for evaluating future-generation long-context LMMs.
LongVideoBench is a question-answering benchmark featuring interleaved video-language inputs up to an hour long, designed to evaluate long-term multimodal understanding and referring reasoning.
multimodal models multimodal understanding referring reasoning multiple-choice questions fine-grained categories long-form video understanding frames