Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
Long-form video understanding, characterized by long-range temporal
dependencies and multiple events, remains a challenge. Existing methods often
rely on static reasoning or external visual-language models (VLMs), which face
issues like complexity and sub-optimal performance due to the lack of
end-to-end training. In this paper, we propose Video-MTR, a reinforced
multi-turn reasoning framework designed to enable iterative key video segment
selection and question comprehension. Unlike traditional video reasoning
pipeline, which generate predictions in a single turn, Video-MTR performs
reasoning in multiple turns, selecting video segments progressively based on
the evolving understanding of previously processed segments and the current
question. This iterative process allows for a more refined and contextually
aware analysis of the video. To ensure intermediate reasoning process, we
introduce a novel gated bi-level reward system, combining trajectory-level
rewards based on answer correctness and turn-level rewards emphasizing
frame-query relevance. This system optimizes both video segment selection and
question comprehension, eliminating the need for external VLMs and allowing
end-to-end training. Extensive experiments on benchmarks like VideoMME, MLVU,
and EgoSchema demonstrate that Video-MTR outperforms existing methods in both
accuracy and efficiency, advancing the state-of-the-art in long video
understanding.
68b7f100295f15ff6091152c
Yuan Xie
admin_assigned
2025-09-09T18:36:55.370Z
638ea19e2cc490759fe96ce7
https://cdn-avatars.huggingface.co/v1/production/uploads/1670292335191-638ea19e2cc490759fe96ce7.jpeg
X
user
Phoebe13
68b7f100295f15ff6091152d
Tianshui Chen
claimed_verified
2025-09-05T14:16:34.464Z
66ebf632a867b95ddb5b9414
/avatars/07a00952128d7fc031260d23e1923500.svg
Tianshui Chen
user
TianshuiChen
68b7f100295f15ff6091152e
Zheng Ge
68b7f100295f15ff6091152f
Lionel Ni
Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
Long-form video understanding, characterized by long-range temporal
dependencies and multiple events, remains a challenge. Existing methods often
rely on static reasoning or external visual-language models (VLMs), which face
issues like complexity and sub-optimal performance due to the lack of
end-to-end training. In this paper, we propose Video-MTR, a reinforced
multi-turn reasoning framework designed to enable iterative key video segment
selection and question comprehension. Unlike traditional video reasoning
pipeline, which generate predictions in a single turn, Video-MTR performs
reasoning in multiple turns, selecting video segments progressively based on
the evolving understanding of previously processed segments and the current
question. This iterative process allows for a more refined and contextually
aware analysis of the video. To ensure intermediate reasoning process, we
introduce a novel gated bi-level reward system, combining trajectory-level
rewards based on answer correctness and turn-level rewards emphasizing
frame-query relevance. This system optimizes both video segment selection and
question comprehension, eliminating the need for external VLMs and allowing
end-to-end training. Extensive experiments on benchmarks like VideoMME, MLVU,
and EgoSchema demonstrate that Video-MTR outperforms existing methods in both
accuracy and efficiency, advancing the state-of-the-art in long video
understanding.
Video-MTR, a reinforced multi-turn reasoning framework, improves long-form video understanding by iteratively selecting key segments and comprehending questions, outperforming existing methods in accuracy and efficiency.
multi-turn reasoning key video segment selection question comprehension gated bi-level reward system trajectory-level rewards turn-level rewards frame-query relevance end-to-end training VideoMME MLVU EgoSchema