Visual Jigsaw Post-Training Improves MLLMs
Reinforcement learning based post-training has recently emerged as a powerful
paradigm for enhancing the alignment and reasoning capabilities of multimodal
large language models (MLLMs). While vision-centric post-training is crucial
for enhancing MLLMs' intrinsic understanding of visual signals, current
post-training paradigms are predominantly text-centric, where dense visual
inputs are only leveraged to extract sparse cues for text-based reasoning.
There exist a few approaches in this direction, however, they often still rely
on text as an intermediate mediator or introduce additional visual generative
designs. In this work, we introduce Visual Jigsaw, a generic self-supervised
post-training framework designed to strengthen visual understanding in MLLMs.
Visual Jigsaw is formulated as a general ordering task: visual inputs are
partitioned, shuffled, and the model must reconstruct the visual information by
producing the correct permutation in natural language. This naturally aligns
with reinforcement learning from verifiable rewards (RLVR), requires no
additional visual generative components, and derives its supervisory signal
automatically without any annotations. We instantiate Visual Jigsaw across
three visual modalities, including images, videos, and 3D data. Extensive
experiments demonstrate substantial improvements in fine-grained perception,
temporal reasoning, and 3D spatial understanding. Our findings highlight the
potential of self-supervised vision-centric tasks in post-training MLLMs and
aim to inspire further research on vision-centric pretext designs. Project
Page: https://penghao-wu.github.io/visual_jigsaw/
68db41e1d2bf1f4b15ec72f5
Penghao Wu
68db41e1d2bf1f4b15ec72f6
Yushan Zhang
68db41e1d2bf1f4b15ec72f7
Haiwen Diao
claimed_verified
2025-09-30T06:32:48.312Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
68db41e1d2bf1f4b15ec72f8
Bo Li
claimed_verified
2025-09-30T06:32:57.152Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
68db41e1d2bf1f4b15ec72f9
Lewei Lu
68db41e1d2bf1f4b15ec72fa
Ziwei Liu
Visual Jigsaw Post-Training Improves MLLMs
Reinforcement learning based post-training has recently emerged as a powerful
paradigm for enhancing the alignment and reasoning capabilities of multimodal
large language models (MLLMs). While vision-centric post-training is crucial
for enhancing MLLMs' intrinsic understanding of visual signals, current
post-training paradigms are predominantly text-centric, where dense visual
inputs are only leveraged to extract sparse cues for text-based reasoning.
There exist a few approaches in this direction, however, they often still rely
on text as an intermediate mediator or introduce additional visual generative
designs. In this work, we introduce Visual Jigsaw, a generic self-supervised
post-training framework designed to strengthen visual understanding in MLLMs.
Visual Jigsaw is formulated as a general ordering task: visual inputs are
partitioned, shuffled, and the model must reconstruct the visual information by
producing the correct permutation in natural language. This naturally aligns
with reinforcement learning from verifiable rewards (RLVR), requires no
additional visual generative components, and derives its supervisory signal
automatically without any annotations. We instantiate Visual Jigsaw across
three visual modalities, including images, videos, and 3D data. Extensive
experiments demonstrate substantial improvements in fine-grained perception,
temporal reasoning, and 3D spatial understanding. Our findings highlight the
potential of self-supervised vision-centric tasks in post-training MLLMs and
aim to inspire further research on vision-centric pretext designs. Project
Page: https://penghao-wu.github.io/visual_jigsaw/
Visual Jigsaw, a self-supervised reinforcement learning framework, enhances multimodal large language models' visual understanding through a permutation task without additional annotations or generative components.
reinforcement learning post-training multimodal large language models visual understanding self-supervised permutation task reinforcement learning from verifiable rewards fine-grained perception temporal reasoning 3D spatial understanding