Deep Video Discovery: Agentic Search with Tool Use for Long-form Video
  Understanding
Long-form video understanding presents significant challenges due to
extensive temporal-spatial complexity and the difficulty of question answering
under such extended contexts. While Large Language Models (LLMs) have
demonstrated considerable advancements in video analysis capabilities and long
context handling, they continue to exhibit limitations when processing
information-dense hour-long videos. To overcome such limitations, we propose
the Deep Video Discovery agent to leverage an agentic search strategy over
segmented video clips. Different from previous video agents manually designing
a rigid workflow, our approach emphasizes the autonomous nature of agents. By
providing a set of search-centric tools on multi-granular video database, our
DVD agent leverages the advanced reasoning capability of LLM to plan on its
current observation state, strategically selects tools, formulates appropriate
parameters for actions, and iteratively refines its internal reasoning in light
of the gathered information. We perform comprehensive evaluation on multiple
long video understanding benchmarks that demonstrates the advantage of the
entire system design. Our DVD agent achieves SOTA performance, significantly
surpassing prior works by a large margin on the challenging LVBench dataset.
Comprehensive ablation studies and in-depth tool analyses are also provided,
yielding insights to further advance intelligent agents tailored for long-form
video understanding tasks. The code will be released later.
68354eec0830dfc6782ba1c4
Xiaoyi Zhang
68354eec0830dfc6782ba1c5
Zhaoyang Jia
68354eec0830dfc6782ba1c6
Zongyu Guo
68354eec0830dfc6782ba1c7
Jiahao Li
68354eec0830dfc6782ba1c8
Bin Li
68354eec0830dfc6782ba1c9
Houqiang Li
68354eec0830dfc6782ba1ca
Yan Lu
Deep Video Discovery: Agentic Search with Tool Use for Long-form Video
  Understanding
Long-form video understanding presents significant challenges due to
extensive temporal-spatial complexity and the difficulty of question answering
under such extended contexts. While Large Language Models (LLMs) have
demonstrated considerable advancements in video analysis capabilities and long
context handling, they continue to exhibit limitations when processing
information-dense hour-long videos. To overcome such limitations, we propose
the Deep Video Discovery agent to leverage an agentic search strategy over
segmented video clips. Different from previous video agents manually designing
a rigid workflow, our approach emphasizes the autonomous nature of agents. By
providing a set of search-centric tools on multi-granular video database, our
DVD agent leverages the advanced reasoning capability of LLM to plan on its
current observation state, strategically selects tools, formulates appropriate
parameters for actions, and iteratively refines its internal reasoning in light
of the gathered information. We perform comprehensive evaluation on multiple
long video understanding benchmarks that demonstrates the advantage of the
entire system design. Our DVD agent achieves SOTA performance, significantly
surpassing prior works by a large margin on the challenging LVBench dataset.
Comprehensive ablation studies and in-depth tool analyses are also provided,
yielding insights to further advance intelligent agents tailored for long-form
video understanding tasks. The code will be released later.
The Deep Video Discovery agent uses an autonomous agentic search strategy with large language models to overcome limitations in long-form video understanding, achieving state-of-the-art results on benchmarks like LVBench.
Deep Video Discovery agent agentic search strategy segmented video clips multi-granular video database advanced reasoning capability LLM autonomous nature observation state search-centric tools internal reasoning long video understanding benchmarks LVBench ablation studies tool analyses