Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
  Large Language Models
Video Large Language Models (Video-LLMs) have demonstrated remarkable
capabilities in coarse-grained video understanding, however, they struggle with
fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
a novel Video-LLM adept at perceiving and reasoning over specific video moments
in a fine-grained manner. We identify that current Video-LLMs have limitations
for fine-grained video understanding since they lack effective temporal
modeling and timestamp representation. In light of this, we sharpen our model
by incorporating (1) an additional temporal stream to encode the relationships
between frames and (2) discrete temporal tokens enriched with specific time
knowledge to represent timestamps. To optimize the training of
Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
simple video-captioning tasks and progressively introducing video temporal
grounding tasks of increasing complexity. To further enhance
Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
VideoQA dataset by an automatic annotation pipeline. Extensive experiments
demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
tasks such as temporal sentence grounding, dense video captioning, and grounded
VideoQA, but also shows great potential as a versatile video assistant for
general video understanding.
6703aeb7400ad7197be3fcdf
Haibo Wang
claimed_verified
2024-10-09T09:08:43.190Z
63fee47352441fe3e87b5088
/avatars/c1df1899e3925aa6fdfc8ee0049fa8a7.svg
WANG HAIBO
user
WHB139426
6703aeb7400ad7197be3fce0
Zhiyang Xu
admin_assigned
2024-10-09T13:07:21.689Z
64b6c686cf5117d7962d8f62
/avatars/96ed7a9602aa4c21b3a3d89608e76dc8.svg
Zhiyang Xu
user
Zhiyang03
6703aeb7400ad7197be3fce1
Yu Cheng
claimed_verified
2024-10-09T16:26:25.964Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
6703aeb7400ad7197be3fce2
Shizhe Diao
admin_assigned
2024-10-09T13:07:42.547Z
633bd54b00732349209a18fe
/avatars/150aa5b8d9bcca24366402932bf2d049.svg
Shizhe Diao
user
shizhediao
6703aeb7400ad7197be3fce3
Yufan Zhou
admin_assigned
2024-10-09T13:08:13.416Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
6703aeb7400ad7197be3fce4
Yixin Cao
6703aeb7400ad7197be3fce5
Qifan Wang
admin_assigned
2024-10-09T13:09:34.090Z
64b65c84c991daa61404abd7
/avatars/3592aa8774ea4e2836099e0701b4b78d.svg
WangQifan
user
Bayees
6703aeb7400ad7197be3fce6
Weifeng Ge
6703aeb7400ad7197be3fce7
Lifu Huang
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video
  Large Language Models
Video Large Language Models (Video-LLMs) have demonstrated remarkable
capabilities in coarse-grained video understanding, however, they struggle with
fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM,
a novel Video-LLM adept at perceiving and reasoning over specific video moments
in a fine-grained manner. We identify that current Video-LLMs have limitations
for fine-grained video understanding since they lack effective temporal
modeling and timestamp representation. In light of this, we sharpen our model
by incorporating (1) an additional temporal stream to encode the relationships
between frames and (2) discrete temporal tokens enriched with specific time
knowledge to represent timestamps. To optimize the training of
Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with
simple video-captioning tasks and progressively introducing video temporal
grounding tasks of increasing complexity. To further enhance
Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded
VideoQA dataset by an automatic annotation pipeline. Extensive experiments
demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding
tasks such as temporal sentence grounding, dense video captioning, and grounded
VideoQA, but also shows great potential as a versatile video assistant for
general video understanding.
Grounded-VideoLLM enhances fine-grained temporal understanding in videos by adding a temporal stream and discrete temporal tokens, demonstrating superior performance in tasks like temporal sentence grounding and grounded VideoQA.
Video Large Language Models Grounded-VideoLLM temporal modeling timestamp representation temporal stream discrete temporal tokens multi-stage training grounded VideoQA