Video ReCap: Recursive Captioning of Hour-Long Videos
Most video captioning models are designed to process short video clips of few
seconds and output text describing low-level visual concepts (e.g., objects,
scenes, atomic actions). However, most real-world videos last for minutes or
hours and have a complex hierarchical structure spanning different temporal
granularities. We propose Video ReCap, a recursive video captioning model that
can process video inputs of dramatically different lengths (from 1 second to 2
hours) and output video captions at multiple hierarchy levels. The recursive
video-language architecture exploits the synergy between different video
hierarchies and can process hour-long videos efficiently. We utilize a
curriculum learning training scheme to learn the hierarchical structure of
videos, starting from clip-level captions describing atomic actions, then
focusing on segment-level descriptions, and concluding with generating
summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
recursive model can flexibly generate captions at different hierarchy levels
while also being useful for other complex video understanding tasks, such as
VideoQA on EgoSchema. Data, code, and models are available at:
https://sites.google.com/view/vidrecap
65d571b9882e75a4e90c6714
Md Mohaiminul Islam
claimed_verified
2024-02-21T16:29:31.755Z
64e424dbd2af12910d916ff7
/avatars/d763c32bc4ee2daffa42311fa9e912da.svg
Md Mohaiminul Islam
user
mmiemon
65d571b9882e75a4e90c6715
Ngan Ho
65d571b9882e75a4e90c6716
Xitong Yang
65d571b9882e75a4e90c6717
Tushar Nagarajan
65d571b9882e75a4e90c6718
Lorenzo Torresani
65d571b9882e75a4e90c6719
Gedas Bertasius
claimed_verified
2024-02-21T16:29:35.306Z
65d60993c2c6418adc99206a
/avatars/fb1f9d5af7022bddf4cb090cd477108d.svg
Gedas Bertasius
user
gberta
Video ReCap: Recursive Captioning of Hour-Long Videos
Most video captioning models are designed to process short video clips of few
seconds and output text describing low-level visual concepts (e.g., objects,
scenes, atomic actions). However, most real-world videos last for minutes or
hours and have a complex hierarchical structure spanning different temporal
granularities. We propose Video ReCap, a recursive video captioning model that
can process video inputs of dramatically different lengths (from 1 second to 2
hours) and output video captions at multiple hierarchy levels. The recursive
video-language architecture exploits the synergy between different video
hierarchies and can process hour-long videos efficiently. We utilize a
curriculum learning training scheme to learn the hierarchical structure of
videos, starting from clip-level captions describing atomic actions, then
focusing on segment-level descriptions, and concluding with generating
summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
recursive model can flexibly generate captions at different hierarchy levels
while also being useful for other complex video understanding tasks, such as
VideoQA on EgoSchema. Data, code, and models are available at:
https://sites.google.com/view/vidrecap
Video ReCap is a recursive model for video captioning that handles videos of varying lengths and outputs captions at multiple hierarchical levels using curriculum learning.
recursive video captioning video hierarchies curriculum learning clip-level captions segment-level descriptions video summaries Ego4D-HCap dataset VideoQA EgoSchema