VideoAgent: Long-form Video Understanding with Large Language Model as
  Agent
Long-form video understanding represents a significant challenge within
computer vision, demanding a model capable of reasoning over long multi-modal
sequences. Motivated by the human cognitive process for long-form video
understanding, we emphasize interactive reasoning and planning over the ability
to process lengthy visual inputs. We introduce a novel agent-based system,
VideoAgent, that employs a large language model as a central agent to
iteratively identify and compile crucial information to answer a question, with
vision-language foundation models serving as tools to translate and retrieve
visual information. Evaluated on the challenging EgoSchema and NExT-QA
benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only
8.4 and 8.2 frames used on average. These results demonstrate superior
effectiveness and efficiency of our method over the current state-of-the-art
methods, highlighting the potential of agent-based approaches in advancing
long-form video understanding.
65f7b20c26f86cf3378c022e
Xiaohan Wang
claimed_verified
2024-12-16T09:44:05.420Z
65703fab7f50602340d23704
/avatars/324c45f5fba9cd8c38a89b30427c06b4.svg
Xiaohan Wang
user
nicholswang
65f7b20c26f86cf3378c022f
Yuhui Zhang
admin_assigned
2024-03-18T09:02:09.951Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
user
yuhuizhang
65f7b20c26f86cf3378c0230
Orr Zohar
admin_assigned
2024-03-18T09:02:16.560Z
648c9605565e3a44f3c9bb7b
https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg
Orr Zohar
user
orrzohar
65f7b20c26f86cf3378c0231
Serena Yeung-Levy
VideoAgent: Long-form Video Understanding with Large Language Model as
  Agent
Long-form video understanding represents a significant challenge within
computer vision, demanding a model capable of reasoning over long multi-modal
sequences. Motivated by the human cognitive process for long-form video
understanding, we emphasize interactive reasoning and planning over the ability
to process lengthy visual inputs. We introduce a novel agent-based system,
VideoAgent, that employs a large language model as a central agent to
iteratively identify and compile crucial information to answer a question, with
vision-language foundation models serving as tools to translate and retrieve
visual information. Evaluated on the challenging EgoSchema and NExT-QA
benchmarks, VideoAgent achieves 54.1% and 71.3% zero-shot accuracy with only
8.4 and 8.2 frames used on average. These results demonstrate superior
effectiveness and efficiency of our method over the current state-of-the-art
methods, highlighting the potential of agent-based approaches in advancing
long-form video understanding.
VideoAgent, an agent-based system using a large language model, significantly enhances long-form video understanding through interactive reasoning and efficient zero-shot accuracy on EgoSchema and NExT-QA benchmarks.
agent-based system large language model vision-language foundation models iterative reasoning multi-modal sequences zero-shot accuracy