ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long
  Video Understanding
Video multimodal large language models (Video-MLLMs) have achieved remarkable
progress in video understanding. However, they remain vulnerable to
hallucination-producing content inconsistent with or unrelated to video inputs.
Previous video hallucination benchmarks primarily focus on short-videos. They
attribute hallucinations to factors such as strong language priors, missing
frames, or vision-language biases introduced by the visual encoder. While these
causes indeed account for most hallucinations in short videos, they still
oversimplify the cause of hallucinations. Sometimes, models generate incorrect
outputs but with correct frame-level semantics. We refer to this type of
hallucination as Semantic Aggregation Hallucination (SAH), which arises during
the process of aggregating frame-level semantics into event-level semantic
groups. Given that SAH becomes particularly critical in long videos due to
increased semantic complexity across multiple events, it is essential to
separate and thoroughly investigate the causes of this type of hallucination.
To address the above issues, we introduce ELV-Halluc, the first benchmark
dedicated to long-video hallucination, enabling a systematic investigation of
SAH. Our experiments confirm the existence of SAH and show that it increases
with semantic complexity. Additionally, we find that models are more prone to
SAH on rapidly changing semantics. Moreover, we discuss potential approaches to
mitigate SAH. We demonstrate that positional encoding strategy contributes to
alleviating SAH, and further adopt DPO strategy to enhance the model's ability
to distinguish semantics within and across events. To support this, we curate a
dataset of 8K adversarial data pairs and achieve improvements on both
ELV-Halluc and Video-MME, including a substantial 27.7% reduction in SAH ratio.
68b6d3de5aa065c912612750
Hao Lu
claimed_verified
2025-09-03T08:28:39.009Z
65c2bef1ce1be49b569745b8
/avatars/c3709a914220b353fad170ee12ec3172.svg
luhao
user
HLSv
68b6d3de5aa065c912612751
Jiahao Wang
claimed_verified
2025-09-03T08:28:36.907Z
66bc7862aa7cdcb1c31a1efb
/avatars/4c2ab907247fe071ff5cdd71c404ca7c.svg
wang jiahao
user
datamonkey
68b6d3de5aa065c912612752
Yaolun Zhang
claimed_verified
2025-09-03T08:28:34.641Z
648d2e2e514bf0ce32ba729f
https://cdn-avatars.huggingface.co/v1/production/uploads/648d2e2e514bf0ce32ba729f/VPL1rehLxkvixz5oRD6u_.jpeg
Yaolun Zhang
user
Mercury7353
68b6d3de5aa065c912612753
Ruohui Wang
68b6d3de5aa065c912612754
Xuanyu Zheng
68b6d3de5aa065c912612755
Yepeng Tang
68b6d3de5aa065c912612756
Dahua Lin
68b6d3de5aa065c912612757
Lewei Lu
ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long
  Video Understanding
Video multimodal large language models (Video-MLLMs) have achieved remarkable
progress in video understanding. However, they remain vulnerable to
hallucination-producing content inconsistent with or unrelated to video inputs.
Previous video hallucination benchmarks primarily focus on short-videos. They
attribute hallucinations to factors such as strong language priors, missing
frames, or vision-language biases introduced by the visual encoder. While these
causes indeed account for most hallucinations in short videos, they still
oversimplify the cause of hallucinations. Sometimes, models generate incorrect
outputs but with correct frame-level semantics. We refer to this type of
hallucination as Semantic Aggregation Hallucination (SAH), which arises during
the process of aggregating frame-level semantics into event-level semantic
groups. Given that SAH becomes particularly critical in long videos due to
increased semantic complexity across multiple events, it is essential to
separate and thoroughly investigate the causes of this type of hallucination.
To address the above issues, we introduce ELV-Halluc, the first benchmark
dedicated to long-video hallucination, enabling a systematic investigation of
SAH. Our experiments confirm the existence of SAH and show that it increases
with semantic complexity. Additionally, we find that models are more prone to
SAH on rapidly changing semantics. Moreover, we discuss potential approaches to
mitigate SAH. We demonstrate that positional encoding strategy contributes to
alleviating SAH, and further adopt DPO strategy to enhance the model's ability
to distinguish semantics within and across events. To support this, we curate a
dataset of 8K adversarial data pairs and achieve improvements on both
ELV-Halluc and Video-MME, including a substantial 27.7% reduction in SAH ratio.
A benchmark for long-video hallucination identifies and investigates Semantic Aggregation Hallucination (SAH), showing its prevalence in complex and rapidly changing semantic contexts, and proposes strategies to mitigate it.
Video-MLLMs video understanding hallucination short-videos language priors missing frames vision-language biases visual encoder Semantic Aggregation Hallucination (SAH) frame-level semantics event-level semantic groups long videos ELV-Halluc positional encoding DPO strategy adversarial data pairs Video-MME