VLog: Video-Language Models by Generative Retrieval of Narration
  Vocabulary
Human daily activities can be concisely narrated as sequences of routine
events (e.g., turning off an alarm) in video streams, forming an event
vocabulary. Motivated by this, we introduce VLog, a novel video understanding
framework that define video narrations as vocabulary, going beyond the typical
subword vocabularies in existing generative video-language models. Built on the
lightweight language model GPT-2, VLog feature three key innovations: (i) A
generative retrieval model, marrying language model's complex reasoning
capabilities with contrastive retrieval's efficient similarity search. (ii) A
hierarchical vocabulary derived from large-scale video narrations using our
narration pair encoding algorithm, enabling efficient indexing of specific
events (e.g., cutting a tomato) by identifying broader scenarios (e.g.,
kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary
update strategy leveraging generative models to extend the vocabulary for novel
events encountered during inference. To validate our approach, we introduce
VidCap-Eval, a development set requiring concise narrations with reasoning
relationships (e.g., before and after). Experiments on EgoSchema, COIN, and
HiREST further demonstrate the effectiveness of VLog, highlighting its ability
to generate concise, contextually accurate, and efficient narrations, offering
a novel perspective on video understanding. Codes are released at
https://github.com/showlab/VLog.
67d27a9117d5cbff7c2519f6
Kevin Qinghong Lin
claimed_verified
2025-03-13T08:23:53.060Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
67d27a9117d5cbff7c2519f7
Mike Zheng Shou
VLog: Video-Language Models by Generative Retrieval of Narration
  Vocabulary
Human daily activities can be concisely narrated as sequences of routine
events (e.g., turning off an alarm) in video streams, forming an event
vocabulary. Motivated by this, we introduce VLog, a novel video understanding
framework that define video narrations as vocabulary, going beyond the typical
subword vocabularies in existing generative video-language models. Built on the
lightweight language model GPT-2, VLog feature three key innovations: (i) A
generative retrieval model, marrying language model's complex reasoning
capabilities with contrastive retrieval's efficient similarity search. (ii) A
hierarchical vocabulary derived from large-scale video narrations using our
narration pair encoding algorithm, enabling efficient indexing of specific
events (e.g., cutting a tomato) by identifying broader scenarios (e.g.,
kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary
update strategy leveraging generative models to extend the vocabulary for novel
events encountered during inference. To validate our approach, we introduce
VidCap-Eval, a development set requiring concise narrations with reasoning
relationships (e.g., before and after). Experiments on EgoSchema, COIN, and
HiREST further demonstrate the effectiveness of VLog, highlighting its ability
to generate concise, contextually accurate, and efficient narrations, offering
a novel perspective on video understanding. Codes are released at
https://github.com/showlab/VLog.
VLog, a generative video understanding framework, enhances vocabulary-based models by integrating generative retrieval, hierarchical scenario-based vocabulary, and dynamic vocabulary updates, improving narration accuracy and efficiency.
generative retrieval contrastive retrieval hierarchical vocabulary narration pair encoding vocabulary update generative models VidCap-Eval EgoSchema COIN HiREST