VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning
Videos, with their unique temporal dimension, demand precise grounded
understanding, where answers are directly linked to visual, interpretable
evidence. Despite significant breakthroughs in reasoning capabilities within
Large Language Models, multi-modal reasoning - especially for videos - remains
unexplored. In this work, we introduce VideoMind, a novel video-language agent
designed for temporal-grounded video understanding. VideoMind incorporates two
key innovations: (i) We identify essential capabilities for video temporal
reasoning and develop a role-based agentic workflow, including a planner for
coordinating different roles, a grounder for temporal localization, a verifier
to assess temporal interval accuracy, and an answerer for question-answering.
(ii) To efficiently integrate these diverse roles, we propose a novel
Chain-of-LoRA strategy, enabling seamless role-switching via lightweight LoRA
adaptors while avoiding the overhead of multiple models, thus balancing
efficiency and flexibility. Extensive experiments on 14 public benchmarks
demonstrate that our agent achieves state-of-the-art performance on diverse
video understanding tasks, including 3 on grounded video question-answering, 6
on video temporal grounding, and 5 on general video question-answering,
underscoring its effectiveness in advancing video agent and long-form temporal
reasoning.
67d8eeb17e184aa2954d19f4
Ye Liu
claimed_verified
2025-03-21T15:01:38.181Z
6250eb5c1fd03f78d0ae550f
https://cdn-avatars.huggingface.co/v1/production/uploads/1649470297585-noauth.jpeg
Ye Liu
user
yeliudev
67d8eeb17e184aa2954d19f5
Kevin Qinghong Lin
admin_assigned
2025-03-18T09:28:47.081Z
64440be5af034cdfd69ca3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
Qinghong (Kevin) Lin
user
KevinQHLin
67d8eeb17e184aa2954d19f6
Chang Wen Chen
67d8eeb17e184aa2954d19f7
Mike Zheng Shou
admin_assigned
2025-03-18T09:29:07.309Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning
Videos, with their unique temporal dimension, demand precise grounded
understanding, where answers are directly linked to visual, interpretable
evidence. Despite significant breakthroughs in reasoning capabilities within
Large Language Models, multi-modal reasoning - especially for videos - remains
unexplored. In this work, we introduce VideoMind, a novel video-language agent
designed for temporal-grounded video understanding. VideoMind incorporates two
key innovations: (i) We identify essential capabilities for video temporal
reasoning and develop a role-based agentic workflow, including a planner for
coordinating different roles, a grounder for temporal localization, a verifier
to assess temporal interval accuracy, and an answerer for question-answering.
(ii) To efficiently integrate these diverse roles, we propose a novel
Chain-of-LoRA strategy, enabling seamless role-switching via lightweight LoRA
adaptors while avoiding the overhead of multiple models, thus balancing
efficiency and flexibility. Extensive experiments on 14 public benchmarks
demonstrate that our agent achieves state-of-the-art performance on diverse
video understanding tasks, including 3 on grounded video question-answering, 6
on video temporal grounding, and 5 on general video question-answering,
underscoring its effectiveness in advancing video agent and long-form temporal
reasoning.
VideoMind, a novel video-language agent, achieves state-of-the-art performance in various video understanding tasks using a role-based agentic workflow and a Chain-of-LoRA strategy.
temporal reasoning role-based agentic workflow planner grounder verifier answerer Chain-of-LoRA LoRA adaptors grounded video question-answering video temporal grounding general video question-answering