FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame
  Spotlighting
While Large Vision-Language Models (LVLMs) have achieved substantial progress
in video understanding, their application to long video reasoning is hindered
by uniform frame sampling and static textual reasoning, which are inefficient
and struggle to handle visually intensive video tasks. To overcome these
challenges, in this paper, we introduce the concept of thinking with long
videos and propose a novel framework FrameThinker. Within this framework, LVLMs
are able to iteratively interrogate video content. Developing such video
reasoning capabilities in LVLMs presents notable challenges, particularly in
adapting the model to new video actions (e.g. select frame), and designing
reward functions to guide LVLMs to adopt the newly introduced action. To solve
these challenges, we propose a two-phase training strategy, first employing
Supervised Fine-Tuning (SFT) to instill fundamental action capabilities,
followed by Reinforcement Learning (RL) to optimize a strategic decision-making
policy. Notably, in this RL phase, we conduct an in-depth and comprehensive
exploration of the reward design for each action and format reward. Extensive
experiments on reasoning benchmarks like Video-Holmes, LongVideo-Reason, and
long-video understanding benchmarks such as LongVideoBench, MLVU, VideoMME, and
LVBench, demonstrate that FrameThinker achieves a significant average
improvement of +10.4% over baselines while drastically reducing the number of
processed frames. Most notably, our 7B model, FrameThinker establishes a new
state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average
of only 20.6 frames. This not only outperforms the competitive LongVILA-R1
(72.0%) but does so with over 20x fewer frames (vs. 512), demonstrating
unparalleled efficiency and effectiveness.
68dc886d4159d1f2418f9a2e
Zefeng He
68dc886d4159d1f2418f9a2f
Xiaoye Qu
68dc886d4159d1f2418f9a30
Yafu Li
68dc886d4159d1f2418f9a31
Siyuan Huang
admin_assigned
2025-10-14T19:21:33.638Z
66f79b323fe089b75e9e0c04
/avatars/92918bf8913012a3f005f09e03b381c2.svg
Siyuan Huang
user
chamber111
68dc886d4159d1f2418f9a32
Daizong Liu
68dc886d4159d1f2418f9a33
Yu Cheng
FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame
  Spotlighting
While Large Vision-Language Models (LVLMs) have achieved substantial progress
in video understanding, their application to long video reasoning is hindered
by uniform frame sampling and static textual reasoning, which are inefficient
and struggle to handle visually intensive video tasks. To overcome these
challenges, in this paper, we introduce the concept of thinking with long
videos and propose a novel framework FrameThinker. Within this framework, LVLMs
are able to iteratively interrogate video content. Developing such video
reasoning capabilities in LVLMs presents notable challenges, particularly in
adapting the model to new video actions (e.g. select frame), and designing
reward functions to guide LVLMs to adopt the newly introduced action. To solve
these challenges, we propose a two-phase training strategy, first employing
Supervised Fine-Tuning (SFT) to instill fundamental action capabilities,
followed by Reinforcement Learning (RL) to optimize a strategic decision-making
policy. Notably, in this RL phase, we conduct an in-depth and comprehensive
exploration of the reward design for each action and format reward. Extensive
experiments on reasoning benchmarks like Video-Holmes, LongVideo-Reason, and
long-video understanding benchmarks such as LongVideoBench, MLVU, VideoMME, and
LVBench, demonstrate that FrameThinker achieves a significant average
improvement of +10.4% over baselines while drastically reducing the number of
processed frames. Most notably, our 7B model, FrameThinker establishes a new
state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average
of only 20.6 frames. This not only outperforms the competitive LongVILA-R1
(72.0%) but does so with over 20x fewer frames (vs. 512), demonstrating
unparalleled efficiency and effectiveness.
FrameThinker, a novel framework, enhances video reasoning by iteratively interrogating video content through supervised fine-tuning and reinforcement learning, achieving significant improvements and efficiency over existing models.
Large Vision-Language Models LVLMs video understanding long video reasoning frame sampling textual reasoning FrameThinker iterative interrogation video reasoning capabilities Supervised Fine-Tuning SFT Reinforcement Learning RL reward functions Video-Holmes LongVideo-Reason LongVideoBench MLVU VideoMME LVBench LongVILA-R1