VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
We explore how reconciling several foundation models (large language models
and vision-language models) with a novel unified memory mechanism could tackle
the challenging video understanding problem, especially capturing the long-term
temporal relations in lengthy videos. In particular, the proposed multimodal
agent VideoAgent: 1) constructs a structured memory to store both the generic
temporal event descriptions and object-centric tracking states of the video; 2)
given an input task query, it employs tools including video segment
localization and object memory querying along with other visual foundation
models to interactively solve the task, utilizing the zero-shot tool-use
ability of LLMs. VideoAgent demonstrates impressive performances on several
long-horizon video understanding benchmarks, an average increase of 6.6% on
NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between
open-sourced models and private counterparts including Gemini 1.5 Pro.
65f9134a1780bc3711558221
Yue Fan
65f9134a1780bc3711558222
Xiaojian Ma
admin_assigned
2024-03-19T09:59:01.858Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
65f9134a1780bc3711558223
Rujie Wu
65f9134a1780bc3711558224
Yuntao Du
admin_assigned
2024-03-19T09:59:26.151Z
64c33dc31920bd98208a244a
/avatars/a83b691f64c851324336334d0b7b8cc6.svg
du
user
yuntao
65f9134a1780bc3711558225
Jiaqi Li
65f9134a1780bc3711558226
Zhi Gao
65f9134a1780bc3711558227
Qing Li
claimed_verified
2024-08-03T08:26:21.818Z
6455f8c2c8f569b995d603c9
/avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
Qing Li
user
li-qing
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
We explore how reconciling several foundation models (large language models
and vision-language models) with a novel unified memory mechanism could tackle
the challenging video understanding problem, especially capturing the long-term
temporal relations in lengthy videos. In particular, the proposed multimodal
agent VideoAgent: 1) constructs a structured memory to store both the generic
temporal event descriptions and object-centric tracking states of the video; 2)
given an input task query, it employs tools including video segment
localization and object memory querying along with other visual foundation
models to interactively solve the task, utilizing the zero-shot tool-use
ability of LLMs. VideoAgent demonstrates impressive performances on several
long-horizon video understanding benchmarks, an average increase of 6.6% on
NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between
open-sourced models and private counterparts including Gemini 1.5 Pro.
A novel unified memory mechanism in VideoAgent enhances video understanding by combining foundation models, improving performance on long-horizon video benchmarks.
foundation models large language models vision-language models unified memory mechanism video understanding temporal relations structured memory temporal event descriptions object-centric tracking video segment localization object memory querying zero-shot tool-use NExT-QA EgoSchema