Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO
Improving Multi-modal Large Language Models (MLLMs) in the post-training
stage typically relies on supervised fine-tuning (SFT) or reinforcement
learning (RL). However, these supervised methods require expensive and manually
annotated multi-modal data--an ultimately unsustainable resource. While recent
efforts have explored unsupervised post-training, their methods are complex and
difficult to iterate. In this work, we are the first to investigate the use of
GRPO, a stable and scalable online RL algorithm, for enabling continual
self-improvement without any external supervision. We propose MM-UPT, a simple
yet effective framework for unsupervised post-training of MLLMs. MM-UPT builds
upon GRPO, replacing traditional reward signals with a self-rewarding mechanism
based on majority voting over multiple sampled responses. Our experiments
demonstrate that MM-UPT significantly improves the reasoning ability of
Qwen2.5-VL-7B (e.g., 66.3 %rightarrow72.9 % on MathVista, 62.9
%rightarrow68.7 % on We-Math), using standard dataset without ground truth
labels. MM-UPT also outperforms prior unsupervised baselines and even
approaches the results of supervised GRPO. Furthermore, we show that
incorporating synthetic questions, generated solely by MLLM itself, can boost
performance as well, highlighting a promising approach for scalable
self-improvement. Overall, MM-UPT offers a new paradigm for continual,
autonomous enhancement of MLLMs in the absence of external supervision. Our
code is available at https://github.com/waltonfuture/MM-UPT.
6837c318a4e378954486e45d
Lai Wei
claimed_verified
2025-05-29T07:42:35.145Z
64a16b1aeacb4b50ba1c889d
https://cdn-avatars.huggingface.co/v1/production/uploads/64a16b1aeacb4b50ba1c889d/EhWoBGL6LlFGIVTUsp2F4.jpeg
Lai Wei
user
WaltonFuture
6837c318a4e378954486e45e
Yuting Li
claimed_verified
2025-06-05T12:42:14.354Z
66e3e801ec0343c253da37c3
/avatars/72fc67fa2b25473fd3acba1fbd132f62.svg
Yuting Li
user
Yuting6
6837c318a4e378954486e45f
Chen Wang
6837c318a4e378954486e460
Yue Wang
extracted_confirmed
2025-06-01T07:52:59.306Z
65e095da35ad8b2fe8c80e71
/avatars/225d4be6411dcec2460047ea1a88a5c3.svg
Weiran Huang
user
weiranhuang
6837c318a4e378954486e461
Linghe Kong
6837c318a4e378954486e462
Weiran Huang
extracted_pending
2025-05-29T02:14:48.838Z
65e095da35ad8b2fe8c80e71
/avatars/225d4be6411dcec2460047ea1a88a5c3.svg
Weiran Huang
user
weiranhuang
6837c318a4e378954486e463
Lichao Sun
Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO
Improving Multi-modal Large Language Models (MLLMs) in the post-training
stage typically relies on supervised fine-tuning (SFT) or reinforcement
learning (RL). However, these supervised methods require expensive and manually
annotated multi-modal data--an ultimately unsustainable resource. While recent
efforts have explored unsupervised post-training, their methods are complex and
difficult to iterate. In this work, we are the first to investigate the use of
GRPO, a stable and scalable online RL algorithm, for enabling continual
self-improvement without any external supervision. We propose MM-UPT, a simple
yet effective framework for unsupervised post-training of MLLMs. MM-UPT builds
upon GRPO, replacing traditional reward signals with a self-rewarding mechanism
based on majority voting over multiple sampled responses. Our experiments
demonstrate that MM-UPT significantly improves the reasoning ability of
Qwen2.5-VL-7B (e.g., 66.3 %rightarrow72.9 % on MathVista, 62.9
%rightarrow68.7 % on We-Math), using standard dataset without ground truth
labels. MM-UPT also outperforms prior unsupervised baselines and even
approaches the results of supervised GRPO. Furthermore, we show that
incorporating synthetic questions, generated solely by MLLM itself, can boost
performance as well, highlighting a promising approach for scalable
self-improvement. Overall, MM-UPT offers a new paradigm for continual,
autonomous enhancement of MLLMs in the absence of external supervision. Our
code is available at https://github.com/waltonfuture/MM-UPT.
MM-UPT, a framework employing GRPO and self-rewarding, enhances multi-modal LLMs through unsupervised continual learning, showing performance improvements without manual annotations.
GRPO MM-UPT reinforcement learning unsupervised post-training multi-modal large language models self-rewarding mechanism majority voting synthetic questions MathVista We-Math