VidText: Towards Comprehensive Evaluation for Video Text Understanding
Visual texts embedded in videos carry rich semantic information, which is
crucial for both holistic video understanding and fine-grained reasoning about
local human actions. However, existing video understanding benchmarks largely
overlook textual information, while OCR-specific benchmarks are constrained to
static images, limiting their ability to capture the interaction between text
and dynamic visual contexts. To address this gap, we propose VidText, a new
benchmark designed for comprehensive and in-depth evaluation of video text
understanding. VidText offers the following key features: 1) It covers a wide
range of real-world scenarios and supports multilingual content, encompassing
diverse settings where video text naturally appears. 2) It introduces a
hierarchical evaluation framework with video-level, clip-level, and
instance-level tasks, enabling assessment of both global summarization and
local retrieval capabilities. 3) The benchmark also introduces a set of paired
perception reasoning tasks, ranging from visual text perception to cross-modal
reasoning between textual and visual information. Extensive experiments on 18
state-of-the-art Large Multimodal Models (LMMs) reveal that current models
struggle across most tasks, with significant room for improvement. Further
analysis highlights the impact of both model-intrinsic factors, such as input
resolution and OCR capability, and external factors, including the use of
auxiliary information and Chain-of-Thought reasoning strategies. We hope
VidText will fill the current gap in video understanding benchmarks and serve
as a foundation for future research on multimodal reasoning with video text in
dynamic environments.
68395c867f983113faf29005
Zhoufaran Yang
68395c867f983113faf29006
Yan Shu
68395c867f983113faf29007
Zhifei Yang
claimed_verified
2025-06-02T07:46:17.645Z
6464c4ef92773d5eeb588525
/avatars/65fc839453d892016640249cc1acf277.svg
Zhifei Yang
user
yangzhifei
68395c867f983113faf29008
Yan Zhang
68395c867f983113faf29009
Yu Li
68395c867f983113faf2900a
Keyang Lu
68395c867f983113faf2900b
Gangyan Zeng
68395c867f983113faf2900c
Shaohui Liu
68395c867f983113faf2900d
Yu Zhou
68395c867f983113faf2900e
Nicu Sebe
VidText: Towards Comprehensive Evaluation for Video Text Understanding
Visual texts embedded in videos carry rich semantic information, which is
crucial for both holistic video understanding and fine-grained reasoning about
local human actions. However, existing video understanding benchmarks largely
overlook textual information, while OCR-specific benchmarks are constrained to
static images, limiting their ability to capture the interaction between text
and dynamic visual contexts. To address this gap, we propose VidText, a new
benchmark designed for comprehensive and in-depth evaluation of video text
understanding. VidText offers the following key features: 1) It covers a wide
range of real-world scenarios and supports multilingual content, encompassing
diverse settings where video text naturally appears. 2) It introduces a
hierarchical evaluation framework with video-level, clip-level, and
instance-level tasks, enabling assessment of both global summarization and
local retrieval capabilities. 3) The benchmark also introduces a set of paired
perception reasoning tasks, ranging from visual text perception to cross-modal
reasoning between textual and visual information. Extensive experiments on 18
state-of-the-art Large Multimodal Models (LMMs) reveal that current models
struggle across most tasks, with significant room for improvement. Further
analysis highlights the impact of both model-intrinsic factors, such as input
resolution and OCR capability, and external factors, including the use of
auxiliary information and Chain-of-Thought reasoning strategies. We hope
VidText will fill the current gap in video understanding benchmarks and serve
as a foundation for future research on multimodal reasoning with video text in
dynamic environments.
VidText is a new benchmark that evaluates video text understanding across various tasks, covering global summarization and local retrieval, and highlights challenges for current multimodal models.
LMMs video text perception cross-modal reasoning Chain-of-Thought reasoning