Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
  Models with Minimal Contrastive Images
Recent studies have shown that Large Vision-Language Models (VLMs) tend to
neglect image content and over-rely on language-model priors, resulting in
errors in visually grounded tasks and hallucinations. We hypothesize that this
issue arises because existing VLMs are not explicitly trained to generate texts
that are accurately grounded in fine-grained image details. To enhance visual
feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
Optimization), a novel finetuning objective that steers the model toward
capturing important visual details and aligning them with corresponding text
tokens. To further facilitate this detailed alignment, we introduce MVC, a
paired image-text dataset built by automatically filtering and augmenting
visual counterfactual data to challenge the model with hard contrastive cases
involving Minimal Visual Contrasts. Experiments show that our method
consistently improves VLM performance across diverse benchmarks covering
various abilities and domains, achieving up to a 22% reduction in
hallucinations, and significant gains in vision-centric and general tasks.
Notably, these improvements become increasingly pronounced in benchmarks with
higher visual dependency. In short, S-VCO offers a significant enhancement of
VLM's visually-dependent task performance while retaining or even improving the
model's general abilities. We opensource our code at https://s-vco.github.io/
67b7cdac904136d47c3966d8
Shengguang Wu
claimed_verified
2025-02-21T09:59:21.001Z
65222f97ef06bb99753cb829
/avatars/f1a743d74e6d38b916acaec91b4e7e4f.svg
Shengguang Wu
user
danielwusg
67b7cdac904136d47c3966d9
Fan-Yun Sun
claimed_verified
2025-03-19T09:48:05.193Z
66287db3fe7ace12c40026b6
/avatars/ee86cddd6c7f6f456f7643ee0ddb084a.svg
Fan-Yun Sun
user
sunfanyun
67b7cdac904136d47c3966da
Kaiyue Wen
67b7cdac904136d47c3966db
Nick Haber
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
  Models with Minimal Contrastive Images
Recent studies have shown that Large Vision-Language Models (VLMs) tend to
neglect image content and over-rely on language-model priors, resulting in
errors in visually grounded tasks and hallucinations. We hypothesize that this
issue arises because existing VLMs are not explicitly trained to generate texts
that are accurately grounded in fine-grained image details. To enhance visual
feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
Optimization), a novel finetuning objective that steers the model toward
capturing important visual details and aligning them with corresponding text
tokens. To further facilitate this detailed alignment, we introduce MVC, a
paired image-text dataset built by automatically filtering and augmenting
visual counterfactual data to challenge the model with hard contrastive cases
involving Minimal Visual Contrasts. Experiments show that our method
consistently improves VLM performance across diverse benchmarks covering
various abilities and domains, achieving up to a 22% reduction in
hallucinations, and significant gains in vision-centric and general tasks.
Notably, these improvements become increasingly pronounced in benchmarks with
higher visual dependency. In short, S-VCO offers a significant enhancement of
VLM's visually-dependent task performance while retaining or even improving the
model's general abilities. We opensource our code at https://s-vco.github.io/
S-VCO, a novel fine-tuning objective using minimal visual contrasts, enhances VLM performance by improving visual detail alignment and reducing hallucinations in visually grounded tasks.
Large Vision-Language Models (VLMs) Symmetrical Visual Contrastive Optimization (S-VCO) MVC visual contrastive optimization minimal visual contrasts fine-grained image details text tokens visual feedback hallucinations vision-centric tasks general tasks