Mitigating Visual Forgetting via Take-along Visual Conditioning for
  Multi-modal Long CoT Reasoning
Recent advancements in Large Language Models (LLMs) have demonstrated
enhanced reasoning capabilities, evolving from Chain-of-Thought (CoT) prompting
to advanced, product-oriented solutions like OpenAI o1. During our
re-implementation of this model, we noticed that in multimodal tasks requiring
visual input (e.g., geometry problems), Multimodal LLMs (MLLMs) struggle to
maintain focus on the visual information, in other words, MLLMs suffer from a
gradual decline in attention to visual information as reasoning progresses,
causing text-over-relied outputs. To investigate this, we ablate image inputs
during long-chain reasoning. Concretely, we truncate the reasoning process
midway, then re-complete the reasoning process with the input image removed. We
observe only a ~2% accuracy drop on MathVista's test-hard subset, revealing the
model's textual outputs dominate the following reasoning process. Motivated by
this, we propose Take-along Visual Conditioning (TVC), a strategy that shifts
image input to critical reasoning stages and compresses redundant visual tokens
via dynamic pruning. This methodology helps the model retain attention to the
visual components throughout the reasoning. Our approach achieves
state-of-the-art performance on average across five mathematical reasoning
benchmarks (+3.4% vs previous sota), demonstrating the effectiveness of TVC in
enhancing multimodal reasoning systems.
67d8e21dea26d6d743f2adde
Hai-Long Sun
claimed_verified
2025-03-20T10:46:16.064Z
6623975c728f756224d4b768
https://cdn-avatars.huggingface.co/v1/production/uploads/6623975c728f756224d4b768/US9n0k45Y1TF6WqOhgBhZ.jpeg
Allen Sun
user
Allen8
67d8e21dea26d6d743f2addf
Zhun Sun
admin_assigned
2025-03-20T13:28:46.228Z
64b75c9c479b934973de0e98
/avatars/99a02865ddc6af27a77fcf7701f9f666.svg
Zhun Sun
user
zhunsun
67d8e21dea26d6d743f2ade0
Houwen Peng
admin_assigned
2025-03-20T13:28:40.023Z
631952b6f18d0b5d999ca397
/avatars/eb87dfc142a7602f8fb888f7b7b60d38.svg
Peng
user
Houwen
67d8e21dea26d6d743f2ade1
Han-Jia Ye
admin_assigned
2025-03-20T13:28:54.161Z
67ab41b0ef4586b3ea65e7ed
/avatars/4fa69250c6657624276a9af2aea6cd89.svg
Han-Jia Ye
user
han-jia
Mitigating Visual Forgetting via Take-along Visual Conditioning for
  Multi-modal Long CoT Reasoning
Recent advancements in Large Language Models (LLMs) have demonstrated
enhanced reasoning capabilities, evolving from Chain-of-Thought (CoT) prompting
to advanced, product-oriented solutions like OpenAI o1. During our
re-implementation of this model, we noticed that in multimodal tasks requiring
visual input (e.g., geometry problems), Multimodal LLMs (MLLMs) struggle to
maintain focus on the visual information, in other words, MLLMs suffer from a
gradual decline in attention to visual information as reasoning progresses,
causing text-over-relied outputs. To investigate this, we ablate image inputs
during long-chain reasoning. Concretely, we truncate the reasoning process
midway, then re-complete the reasoning process with the input image removed. We
observe only a ~2% accuracy drop on MathVista's test-hard subset, revealing the
model's textual outputs dominate the following reasoning process. Motivated by
this, we propose Take-along Visual Conditioning (TVC), a strategy that shifts
image input to critical reasoning stages and compresses redundant visual tokens
via dynamic pruning. This methodology helps the model retain attention to the
visual components throughout the reasoning. Our approach achieves
state-of-the-art performance on average across five mathematical reasoning
benchmarks (+3.4% vs previous sota), demonstrating the effectiveness of TVC in
enhancing multimodal reasoning systems.
Take-along Visual Conditioning (TVC) strategy enhances multimodal reasoning by focusing visual input at critical stages and compressing redundant visual tokens, achieving better performance in mathematical reasoning benchmarks.
Large Language Models (LLMs) Chain-of-Thought (CoT) OpenAI o1 Multimodal LLMs (MLLMs) attention (MathVista's test-hard subset Take-along Visual Conditioning (TVC) dynamic pruning multimodal reasoning