Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose
  Video Hallucination by Fine-grained Spatial-Temporal Grounding
Recent advancements in large video models (LVMs) have significantly enhance
video understanding. However, these models continue to suffer from
hallucinations, producing content that conflicts with input videos. To address
this issue, we propose Dr.V, a hierarchical framework covering perceptive,
temporal, and cognitive levels to diagnose video hallucination by fine-grained
spatial-temporal grounding. Dr.V comprises of two key components: a benchmark
dataset Dr.V-Bench and a satellite video agent Dr.V-Agent. Dr.V-Bench includes
10k instances drawn from 4,974 videos spanning diverse tasks, each enriched
with detailed spatial-temporal annotation. Dr.V-Agent detects hallucinations in
LVMs by systematically applying fine-grained spatial-temporal grounding at the
perceptive and temporal levels, followed by cognitive level reasoning. This
step-by-step pipeline mirrors human-like video comprehension and effectively
identifies hallucinations. Extensive experiments demonstrate that Dr.V-Agent is
effective in diagnosing hallucination while enhancing interpretability and
reliability, offering a practical blueprint for robust video understanding in
real-world scenarios. All our data and code are available at
https://github.com/Eurekaleo/Dr.V.
68c954bf4b9025f9f1e0c188
Meng Luo
68c954bf4b9025f9f1e0c189
Shengqiong Wu
68c954bf4b9025f9f1e0c18a
Liqiang Jing
68c954bf4b9025f9f1e0c18b
Tianjie Ju
68c954bf4b9025f9f1e0c18c
Li Zheng
68c954bf4b9025f9f1e0c18d
Jinxiang Lai
68c954bf4b9025f9f1e0c18e
Tianlong Wu
68c954bf4b9025f9f1e0c18f
Xinya Du
68c954bf4b9025f9f1e0c190
Jian Li
68c954bf4b9025f9f1e0c191
Siyuan Yan
68c954bf4b9025f9f1e0c192
Jiebo Luo
68c954bf4b9025f9f1e0c193
William Yang Wang
68c954bf4b9025f9f1e0c194
Hao Fei
68c954bf4b9025f9f1e0c195
Mong-Li Lee
68c954bf4b9025f9f1e0c196
Wynne Hsu
Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose
  Video Hallucination by Fine-grained Spatial-Temporal Grounding
Recent advancements in large video models (LVMs) have significantly enhance
video understanding. However, these models continue to suffer from
hallucinations, producing content that conflicts with input videos. To address
this issue, we propose Dr.V, a hierarchical framework covering perceptive,
temporal, and cognitive levels to diagnose video hallucination by fine-grained
spatial-temporal grounding. Dr.V comprises of two key components: a benchmark
dataset Dr.V-Bench and a satellite video agent Dr.V-Agent. Dr.V-Bench includes
10k instances drawn from 4,974 videos spanning diverse tasks, each enriched
with detailed spatial-temporal annotation. Dr.V-Agent detects hallucinations in
LVMs by systematically applying fine-grained spatial-temporal grounding at the
perceptive and temporal levels, followed by cognitive level reasoning. This
step-by-step pipeline mirrors human-like video comprehension and effectively
identifies hallucinations. Extensive experiments demonstrate that Dr.V-Agent is
effective in diagnosing hallucination while enhancing interpretability and
reliability, offering a practical blueprint for robust video understanding in
real-world scenarios. All our data and code are available at
https://github.com/Eurekaleo/Dr.V.
Dr.V, a hierarchical framework with Dr.V-Bench and Dr.V-Agent, addresses video hallucinations through fine-grained spatial-temporal grounding and cognitive reasoning, enhancing video understanding.
large video models LVMs video hallucinations hierarchical framework perceptive levels temporal levels cognitive levels fine-grained spatial-temporal grounding benchmark dataset satellite video agent Dr.V-Bench Dr.V-Agent spatial-temporal annotation human-like video comprehension interpretability reliability