When and What: Diffusion-Grounded VideoLLM with Entity Aware
  Segmentation for Long Video Understanding
Understanding videos requires more than answering open ended questions, it
demands the ability to pinpoint when events occur and how entities interact
across time. While recent Video LLMs have achieved remarkable progress in
holistic reasoning, they remain coarse in temporal perception: timestamps are
encoded only implicitly, frame level features are weak in capturing continuity,
and language vision alignment often drifts from the entities of interest. In
this paper, we present Grounded VideoDiT, a Video LLM designed to overcome
these limitations by introducing three key innovations. First, a Diffusion
Temporal Latent (DTL) encoder enhances boundary sensitivity and maintains
temporal consistency. Second, object grounded representations explicitly bind
query entities to localized visual evidence, strengthening alignment. Third, a
mixed token scheme with discrete temporal tokens provides explicit timestamp
modeling, enabling fine grained temporal reasoning. Together, these designs
equip Grounded VideoDiT with robust grounding capabilities, as validated by
state of the art results on Charades STA, NExT GQA, and multiple VideoQA
benchmarks.
68a7f23039413c456c05b177
Pengcheng Fang
admin_assigned
2025-08-22T13:07:54.546Z
64eef308ec50d5e55376d394
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KaLwu_Lg25Xr6l3wxLoHm.png
Pengcheng Fang
user
AmberJar
68a7f23039413c456c05b178
Yuxia Chen
admin_assigned
2025-08-22T13:08:01.497Z
66c6f2f91ca3a3e821dba3b2
/avatars/69f6362636be4c2c9926cc56614ff847.svg
Yuxia Chen
user
ElioChen
68a7f23039413c456c05b179
Rui Guo
When and What: Diffusion-Grounded VideoLLM with Entity Aware
  Segmentation for Long Video Understanding
Understanding videos requires more than answering open ended questions, it
demands the ability to pinpoint when events occur and how entities interact
across time. While recent Video LLMs have achieved remarkable progress in
holistic reasoning, they remain coarse in temporal perception: timestamps are
encoded only implicitly, frame level features are weak in capturing continuity,
and language vision alignment often drifts from the entities of interest. In
this paper, we present Grounded VideoDiT, a Video LLM designed to overcome
these limitations by introducing three key innovations. First, a Diffusion
Temporal Latent (DTL) encoder enhances boundary sensitivity and maintains
temporal consistency. Second, object grounded representations explicitly bind
query entities to localized visual evidence, strengthening alignment. Third, a
mixed token scheme with discrete temporal tokens provides explicit timestamp
modeling, enabling fine grained temporal reasoning. Together, these designs
equip Grounded VideoDiT with robust grounding capabilities, as validated by
state of the art results on Charades STA, NExT GQA, and multiple VideoQA
benchmarks.
Grounded VideoDiT addresses temporal perception and entity interaction in videos through a Diffusion Temporal Latent encoder, object grounded representations, and mixed token scheme, achieving state-of-the-art results on VideoQA benchmarks.
Diffusion Temporal Latent object grounded representations mixed token scheme discrete temporal tokens Charades STA NExT GQA VideoQA benchmarks