mDPO: Conditional Preference Optimization for Multimodal Large Language
  Models
Direct preference optimization (DPO) has shown to be an effective method for
large language model (LLM) alignment. Recent works have attempted to apply DPO
to multimodal scenarios but have found it challenging to achieve consistent
improvement. Through a comparative experiment, we identify the unconditional
preference problem in multimodal preference optimization, where the model
overlooks the image condition. To address this problem, we propose mDPO, a
multimodal DPO objective that prevents the over-prioritization of language-only
preferences by also optimizing image preference. Moreover, we introduce a
reward anchor that forces the reward to be positive for chosen responses,
thereby avoiding the decrease in their likelihood -- an intrinsic problem of
relative preference optimization. Experiments on two multimodal LLMs of
different sizes and three widely used benchmarks demonstrate that mDPO
effectively addresses the unconditional preference problem in multimodal
preference optimization and significantly improves model performance,
particularly in reducing hallucination.
6670efee86ff335801c6754d
Fei Wang
claimed_verified
2024-06-18T07:18:01.857Z
652d9db6442fb6963b778295
/avatars/66585a780b520c05f3c3e39f01245f7e.svg
Fei Wang
user
fwnlp
6670efee86ff335801c6754e
Wenxuan Zhou
admin_assigned
2024-06-18T08:30:18.982Z
619414c7979eb76ec3cc5bab
/avatars/c1a7055db6150b1bba92e7b13266b489.svg
Wenxuan Zhou
user
wzhouad
6670efee86ff335801c6754f
James Y. Huang
6670efee86ff335801c67550
Nan Xu
6670efee86ff335801c67551
Sheng Zhang
claimed_verified
2024-06-18T21:27:31.381Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
6670efee86ff335801c67552
Hoifung Poon
admin_assigned
2024-06-18T08:31:32.388Z
664d07456083f276c4feb1a4
/avatars/1bfa6d8f82e9223b47630cefd79d7d0e.svg
Hoifung Poon
user
hoifung
6670efee86ff335801c67553
Muhao Chen
extracted_confirmed
2024-06-18T03:43:23.648Z
666be8ef81f01fbd60e84f01
https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
Muhao Chen
user
Muhao
mDPO: Conditional Preference Optimization for Multimodal Large Language
  Models
Direct preference optimization (DPO) has shown to be an effective method for
large language model (LLM) alignment. Recent works have attempted to apply DPO
to multimodal scenarios but have found it challenging to achieve consistent
improvement. Through a comparative experiment, we identify the unconditional
preference problem in multimodal preference optimization, where the model
overlooks the image condition. To address this problem, we propose mDPO, a
multimodal DPO objective that prevents the over-prioritization of language-only
preferences by also optimizing image preference. Moreover, we introduce a
reward anchor that forces the reward to be positive for chosen responses,
thereby avoiding the decrease in their likelihood -- an intrinsic problem of
relative preference optimization. Experiments on two multimodal LLMs of
different sizes and three widely used benchmarks demonstrate that mDPO
effectively addresses the unconditional preference problem in multimodal
preference optimization and significantly improves model performance,
particularly in reducing hallucination.
mDPO, a multimodal preference optimization method, addresses the unconditional preference problem and enhances model performance in multimodal large language models.
direct preference optimization (DPO) large language model (LLM) multimodal preference optimization unconditional preference problem multimodal DPO image condition reward anchor relative preference optimization hallucination