Number it: Temporal Grounding Videos like Flipping Manga
Video Large Language Models (Vid-LLMs) have made remarkable advancements in
comprehending video content for QA dialogue. However, they struggle to extend
this visual understanding to tasks requiring precise temporal localization,
known as Video Temporal Grounding (VTG). To address this gap, we introduce
Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual
comprehension with temporal grounding by adding unique numerical identifiers to
each video frame. Treating a video as a sequence of numbered frame images,
NumPro transforms VTG into an intuitive process: flipping through manga panels
in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking
visual content with corresponding temporal information. Our experiments
demonstrate that NumPro significantly boosts VTG performance of top-tier
Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a
NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing
previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and
8.5\% in mAP for highlight detection. The code will be available at
https://github.com/yongliang-wu/NumPro.
673ad833abddf84949fd2952
Yongliang Wu
claimed_verified
2024-11-18T07:47:14.506Z
66f6bc97980d52c75c300511
/avatars/f7c23c4b09701580b533212ec9b6e306.svg
Yongliang Wu
user
Liang0223
673ad833abddf84949fd2953
Xinting Hu
admin_assigned
2024-11-18T07:51:51.989Z
652845524def261e0febbf5b
/avatars/92743e595cbce4e37a8fe7400b5051ba.svg
Xinting Hu
user
kaleidudu
673ad833abddf84949fd2954
Yuyang Sun
admin_assigned
2024-11-18T07:51:58.153Z
655540892da2e4f12e6c743c
/avatars/790a80f7f9c21e3bf09addb323a6467a.svg
Yuyang Sun
user
tms28k
673ad833abddf84949fd2955
Yizhou Zhou
admin_assigned
2024-11-18T07:52:13.463Z
667ea7f396704b4d8f95ba58
/avatars/a3ec12d590e99cad07d3ecb618e5aacf.svg
Zhouyi Zhou
user
zhouzhouyi
673ad833abddf84949fd2956
Wenbo Zhu
claimed_verified
2024-11-18T07:47:04.587Z
6434e89e7b82474801105f06
/avatars/25d23d564d35e2ecb5ed6b1bbd0aa489.svg
Vito Zhu
user
Vito328
673ad833abddf84949fd2957
Fengyun Rao
673ad833abddf84949fd2958
Bernt Schiele
673ad833abddf84949fd2959
Xu Yang
Number it: Temporal Grounding Videos like Flipping Manga
Video Large Language Models (Vid-LLMs) have made remarkable advancements in
comprehending video content for QA dialogue. However, they struggle to extend
this visual understanding to tasks requiring precise temporal localization,
known as Video Temporal Grounding (VTG). To address this gap, we introduce
Number-Prompt (NumPro), a novel method that empowers Vid-LLMs to bridge visual
comprehension with temporal grounding by adding unique numerical identifiers to
each video frame. Treating a video as a sequence of numbered frame images,
NumPro transforms VTG into an intuitive process: flipping through manga panels
in sequence. This allows Vid-LLMs to "read" event timelines, accurately linking
visual content with corresponding temporal information. Our experiments
demonstrate that NumPro significantly boosts VTG performance of top-tier
Vid-LLMs without additional computational cost. Furthermore, fine-tuning on a
NumPro-enhanced dataset defines a new state-of-the-art for VTG, surpassing
previous top-performing methods by up to 6.9\% in mIoU for moment retrieval and
8.5\% in mAP for highlight detection. The code will be available at
https://github.com/yongliang-wu/NumPro.
Number-Prompt (NumPro) enhances Video Large Language Models (Vid-LLMs) for Video Temporal Grounding (VTG) by adding numerical identifiers to video frames, improving performance significantly without increased computational costs.
Vid-LLMs Video Temporal Grounding VTG Number-Prompt NumPro moment retrieval highlight detection