Video SimpleQA: Towards Factuality Evaluation in Large Video Language
  Models
Recent advancements in Large Video Language Models (LVLMs) have highlighted
their potential for multi-modal understanding, yet evaluating their factual
grounding in video contexts remains a critical unsolved challenge. To address
this gap, we introduce Video SimpleQA, the first comprehensive benchmark
tailored for factuality evaluation of LVLMs. Our work distinguishes from
existing video benchmarks through the following key features: 1) Knowledge
required: demanding integration of external knowledge beyond the explicit
narrative; 2) Fact-seeking question: targeting objective, undisputed events or
relationships, avoiding subjective interpretation; 3) Definitive & short-form
answer: Answers are crafted as unambiguous and definitively correct in a short
format, enabling automated evaluation through LLM-as-a-judge frameworks with
minimal scoring variance; 4) External-source verified: All annotations undergo
rigorous validation against authoritative external references to ensure the
reliability; 5) Temporal reasoning required: The annotated question types
encompass both static single-frame understanding and dynamic temporal
reasoning, explicitly evaluating LVLMs factuality under the long-context
dependencies. We extensively evaluate 41 state-of-the-art LVLMs and summarize
key findings as follows: 1) Current LVLMs exhibit notable deficiencies in
factual adherence, particularly for open-source models. The best-performing
model Gemini-1.5-Pro achieves merely an F-score of 54.4%; 2) Test-time compute
paradigms show insignificant performance gains, revealing fundamental
constraints for enhancing factuality through post-hoc computation; 3)
Retrieval-Augmented Generation demonstrates consistent improvements at the cost
of additional inference time overhead, presenting a critical
efficiency-performance trade-off.
67e226f401cdb8cf3a1c7cd8
Meng Cao
admin_assigned
2025-03-25T09:22:38.738Z
640222f83e3d0f2745b097b2
/avatars/c5dbac84734855369a7f57b051f16caa.svg
Meng Cao
user
mengcao
67e226f401cdb8cf3a1c7cd9
Pengfei Hu
claimed_verified
2025-03-31T08:15:11.513Z
67597dfd8cd6382952537ddb
https://cdn-avatars.huggingface.co/v1/production/uploads/67597dfd8cd6382952537ddb/veH232xoIvxep7vpTtR8m.jpeg
Pengfei Hu
user
PengfeiHu
67e226f401cdb8cf3a1c7cda
Yingyao Wang
67e226f401cdb8cf3a1c7cdb
Jihao Gu
admin_assigned
2025-03-25T09:22:23.718Z
65733c1b244aefdfc45cc771
/avatars/7223cedbeed065c28a400e130cea30ae.svg
Jihao Guo
user
grejioh
67e226f401cdb8cf3a1c7cdc
Haoran Tang
67e226f401cdb8cf3a1c7cdd
Haoze Zhao
67e226f401cdb8cf3a1c7cde
Jiahua Dong
67e226f401cdb8cf3a1c7cdf
Wangbo Yu
admin_assigned
2025-03-25T09:23:02.455Z
63f095be6309c84d5f48848a
https://cdn-avatars.huggingface.co/v1/production/uploads/63f095be6309c84d5f48848a/pL2CKi-r-0mMfIGhYSAsm.jpeg
Wangbo Yu
user
Drexubery
67e226f401cdb8cf3a1c7ce0
Ge Zhang
admin_assigned
2025-03-25T09:22:08.334Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67e226f401cdb8cf3a1c7ce1
Ian Reid
67e226f401cdb8cf3a1c7ce2
Xiaodan Liang
Video SimpleQA: Towards Factuality Evaluation in Large Video Language
  Models
Recent advancements in Large Video Language Models (LVLMs) have highlighted
their potential for multi-modal understanding, yet evaluating their factual
grounding in video contexts remains a critical unsolved challenge. To address
this gap, we introduce Video SimpleQA, the first comprehensive benchmark
tailored for factuality evaluation of LVLMs. Our work distinguishes from
existing video benchmarks through the following key features: 1) Knowledge
required: demanding integration of external knowledge beyond the explicit
narrative; 2) Fact-seeking question: targeting objective, undisputed events or
relationships, avoiding subjective interpretation; 3) Definitive & short-form
answer: Answers are crafted as unambiguous and definitively correct in a short
format, enabling automated evaluation through LLM-as-a-judge frameworks with
minimal scoring variance; 4) External-source verified: All annotations undergo
rigorous validation against authoritative external references to ensure the
reliability; 5) Temporal reasoning required: The annotated question types
encompass both static single-frame understanding and dynamic temporal
reasoning, explicitly evaluating LVLMs factuality under the long-context
dependencies. We extensively evaluate 41 state-of-the-art LVLMs and summarize
key findings as follows: 1) Current LVLMs exhibit notable deficiencies in
factual adherence, particularly for open-source models. The best-performing
model Gemini-1.5-Pro achieves merely an F-score of 54.4%; 2) Test-time compute
paradigms show insignificant performance gains, revealing fundamental
constraints for enhancing factuality through post-hoc computation; 3)
Retrieval-Augmented Generation demonstrates consistent improvements at the cost
of additional inference time overhead, presenting a critical
efficiency-performance trade-off.
Video SimpleQA evaluates the factual grounding of LVLMs in videos through a benchmark requiring external knowledge integration, fact-seeking questions, and temporal reasoning.
Large Video Language Models LVLMs Video SimpleQA factual grounding knowledge integration fact-seeking questions short-form answers external-source verification temporal reasoning LLM-as-a-judge Gemini-1.5-Pro F-score Retrieval-Augmented Generation