Enhancing the Reasoning Ability of Multimodal Large Language Models via
  Mixed Preference Optimization
Existing open-source multimodal large language models (MLLMs) generally
follow a training process involving pre-training and supervised fine-tuning.
However, these models suffer from distribution shifts, which limit their
multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance.
To address this, we introduce a preference optimization (PO) process to enhance
the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data
side, we design an automated preference data construction pipeline to create
MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and
(2) on the model side, we explore integrating PO with MLLMs, developing a
simple yet effective method, termed Mixed Preference Optimization (MPO), which
boosts multimodal CoT performance. Our approach demonstrates improved
performance across multiple benchmarks, particularly in multimodal reasoning
tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on
MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance
comparable to the 10x larger InternVL2-76B. We hope this study could inspire
further advancements in MLLMs. Code, data, and model shall be publicly
released.
673acbc08127cd120caeb52d
Weiyun Wang
admin_assigned
2024-11-22T13:03:42.369Z
619507e7b74b6c591f794340
https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
Weiyun Wang
user
Weiyun1025
673acbc08127cd120caeb52e
Zhe Chen
claimed_verified
2024-12-09T14:48:05.699Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
673acbc08127cd120caeb52f
Wenhai Wang
admin_assigned
2024-11-22T13:03:58.775Z
64d1c560c0c627dfa71bdbe0
/avatars/f42794fe25bffcd870a1bcee69b95298.svg
wenhai.wang
user
wangwhcore
673acbc08127cd120caeb530
Yue Cao
claimed_verified
2025-08-29T11:09:53.080Z
6571382c7644d1128561cebe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bnJ-T3k_w1g1Mr7b7LglK.jpeg
Cao Yue
user
yuecao0119
673acbc08127cd120caeb531
Yangzhou Liu
673acbc08127cd120caeb532
Zhangwei Gao
673acbc08127cd120caeb533
Jinguo Zhu
673acbc08127cd120caeb534
Xizhou Zhu
admin_assigned
2024-11-22T13:04:52.411Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
673acbc08127cd120caeb535
Lewei Lu
admin_assigned
2024-11-22T13:04:57.809Z
65ead3ea908526a39082e641
/avatars/dcf870695fd56b06ca03d82f831e9019.svg
Lewei Lu
user
luotto
673acbc08127cd120caeb536
Yu Qiao
673acbc08127cd120caeb537
Jifeng Dai
admin_assigned
2024-11-22T13:05:03.791Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
Enhancing the Reasoning Ability of Multimodal Large Language Models via
  Mixed Preference Optimization
Existing open-source multimodal large language models (MLLMs) generally
follow a training process involving pre-training and supervised fine-tuning.
However, these models suffer from distribution shifts, which limit their
multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance.
To address this, we introduce a preference optimization (PO) process to enhance
the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data
side, we design an automated preference data construction pipeline to create
MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and
(2) on the model side, we explore integrating PO with MLLMs, developing a
simple yet effective method, termed Mixed Preference Optimization (MPO), which
boosts multimodal CoT performance. Our approach demonstrates improved
performance across multiple benchmarks, particularly in multimodal reasoning
tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on
MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance
comparable to the 10x larger InternVL2-76B. We hope this study could inspire
further advancements in MLLMs. Code, data, and model shall be publicly
released.
A preference optimization process enhances multimodal reasoning, particularly Chain-of-Thought performance, in large language models through automated data construction and mixed preference optimization, demonstrating significant improvements on benchmarks.
multimodal large language models MLLMs distribution shifts Chain-of-Thought preference optimization PO MMPR Mixed Preference Optimization MPO multimodal CoT benchmarks MathVista InternVL2