Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and
  Reasoning in Vision-Language Models
Although current large Vision-Language Models (VLMs) have advanced in
multimodal understanding and reasoning, their fundamental perceptual and
reasoning abilities remain limited. Specifically, even on simple jigsaw tasks,
existing VLMs perform near randomly, revealing deficiencies in core perception
and reasoning capabilities. While high-quality vision-language data can enhance
these capabilities, its scarcity and limited scalability impose significant
constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction
Learning for Enhancing visual perception and reasoning in VLMs. AGILE
formulates jigsaw solving as an interactive process, enabling the model to
progressively engage with the environment. At each step, the model generates
executable code to perform an action based on the current state, while the
environment provides fine-grained visual feedback to guide task completion.
Through this iterative cycle of observation and interaction, the model
incrementally improves its perceptual and reasoning capabilities via
exploration and feedback. Experimental results show that AGILE not only
substantially boosts performance on jigsaw tasks of varying complexity (e.g.,
increasing accuracy from 9.5% to 82.8% under the 2 times 2 setting) but also
demonstrates strong generalization across 9 general vision tasks, achieving an
average improvement of 3.1%. These results indicate notable enhancements in
both perceptual and reasoning abilities. This work opens a new avenue for
advancing reasoning and generalization in multimodal models and provides an
efficient, scalable solution to the scarcity of multimodal reinforcement
learning data. The code and datasets is available at
https://github.com/yuzeng0-0/AGILE .
68df4ea4df49fb0df1e03d57
Yu Zeng
claimed_verified
2025-10-03T12:28:59.473Z
665d652e0f35c005de892108
https://cdn-avatars.huggingface.co/v1/production/uploads/665d652e0f35c005de892108/OGLbgZekX-3XTBkwS8k86.jpeg
Yu Zeng
user
YuZeng260
68df4ea4df49fb0df1e03d58
Wenxuan Huang
claimed_verified
2025-10-11T14:05:43.685Z
67dc162ec8c00778e8689f42
https://cdn-avatars.huggingface.co/v1/production/uploads/67dc162ec8c00778e8689f42/_y_tO6W3ONOkOWbumAFXA.png
Wenxuan Huang
user
Osilly
68df4ea4df49fb0df1e03d59
Shiting Huang
claimed_verified
2025-10-05T12:46:57.125Z
66ae3fbf491b555fef3bac0c
/avatars/47353470d46097ce108d32792dbbf2a2.svg
Shiting Huang
user
chocckaka
68df4ea4df49fb0df1e03d5a
Xikun Bao
68df4ea4df49fb0df1e03d5b
Yukun Qi
68df4ea4df49fb0df1e03d5c
Yiming Zhao
68df4ea4df49fb0df1e03d5d
Qiuchen Wang
68df4ea4df49fb0df1e03d5e
Lin Chen
claimed_verified
2025-10-09T12:49:16.311Z
64b02ec0e5000ae8a572ced5
https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
Lin Chen
user
Lin-Chen
68df4ea4df49fb0df1e03d5f
Zehui Chen
68df4ea4df49fb0df1e03d60
Huaian Chen
68df4ea4df49fb0df1e03d61
Wanli Ouyang
68df4ea4df49fb0df1e03d62
Feng Zhao
Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and
  Reasoning in Vision-Language Models
Although current large Vision-Language Models (VLMs) have advanced in
multimodal understanding and reasoning, their fundamental perceptual and
reasoning abilities remain limited. Specifically, even on simple jigsaw tasks,
existing VLMs perform near randomly, revealing deficiencies in core perception
and reasoning capabilities. While high-quality vision-language data can enhance
these capabilities, its scarcity and limited scalability impose significant
constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction
Learning for Enhancing visual perception and reasoning in VLMs. AGILE
formulates jigsaw solving as an interactive process, enabling the model to
progressively engage with the environment. At each step, the model generates
executable code to perform an action based on the current state, while the
environment provides fine-grained visual feedback to guide task completion.
Through this iterative cycle of observation and interaction, the model
incrementally improves its perceptual and reasoning capabilities via
exploration and feedback. Experimental results show that AGILE not only
substantially boosts performance on jigsaw tasks of varying complexity (e.g.,
increasing accuracy from 9.5% to 82.8% under the 2 times 2 setting) but also
demonstrates strong generalization across 9 general vision tasks, achieving an
average improvement of 3.1%. These results indicate notable enhancements in
both perceptual and reasoning abilities. This work opens a new avenue for
advancing reasoning and generalization in multimodal models and provides an
efficient, scalable solution to the scarcity of multimodal reinforcement
learning data. The code and datasets is available at
https://github.com/yuzeng0-0/AGILE .
AGILE, an interactive jigsaw-solving framework, enhances visual perception and reasoning in Vision-Language Models through iterative action and feedback, improving performance on jigsaw tasks and general vision tasks.
Vision-Language Models VLMs jigsaw tasks perceptual capabilities reasoning capabilities AGILE Agentic jiGsaw Interaction Learning executable code fine-grained visual feedback multimodal understanding multimodal reasoning multimodal reinforcement learning