OViP: Online Vision-Language Preference Learning
Large vision-language models (LVLMs) remain vulnerable to hallucination,
often generating content misaligned with visual inputs. While recent approaches
advance multi-modal Direct Preference Optimization (DPO) to mitigate
hallucination, they typically rely on predefined or randomly edited negative
samples that fail to reflect actual model errors, limiting training efficacy.
In this work, we propose an Online Vision-language Preference Learning (OViP)
framework that dynamically constructs contrastive training data based on the
model's own hallucinated outputs. By identifying semantic differences between
sampled response pairs and synthesizing negative images using a diffusion
model, OViP generates more relevant supervision signals in real time. This
failure-driven training enables adaptive alignment of both textual and visual
preferences. Moreover, we refine existing evaluation protocols to better
capture the trade-off between hallucination suppression and expressiveness.
Experiments on hallucination and general benchmarks demonstrate that OViP
effectively reduces hallucinations while preserving core multi-modal
capabilities.
682fdcc0087ea62f1663df96
Shujun Liu
682fdcc0087ea62f1663df97
Siyuan Wang
682fdcc0087ea62f1663df98
Zejun Li
682fdcc0087ea62f1663df99
Jianxiang Wang
682fdcc0087ea62f1663df9a
Cheng Zeng
682fdcc0087ea62f1663df9b
Zhongyu Wei
OViP: Online Vision-Language Preference Learning
Large vision-language models (LVLMs) remain vulnerable to hallucination,
often generating content misaligned with visual inputs. While recent approaches
advance multi-modal Direct Preference Optimization (DPO) to mitigate
hallucination, they typically rely on predefined or randomly edited negative
samples that fail to reflect actual model errors, limiting training efficacy.
In this work, we propose an Online Vision-language Preference Learning (OViP)
framework that dynamically constructs contrastive training data based on the
model's own hallucinated outputs. By identifying semantic differences between
sampled response pairs and synthesizing negative images using a diffusion
model, OViP generates more relevant supervision signals in real time. This
failure-driven training enables adaptive alignment of both textual and visual
preferences. Moreover, we refine existing evaluation protocols to better
capture the trade-off between hallucination suppression and expressiveness.
Experiments on hallucination and general benchmarks demonstrate that OViP
effectively reduces hallucinations while preserving core multi-modal
capabilities.
OViP dynamically generates contrastive training data using a diffusion model to reduce hallucinations in large vision-language models while maintaining their multi-modal capabilities.
large vision-language models hallucination multi-modal Direct Preference Optimization OViP diffusion model contrastive training semantic differences failure-driven training