When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
  Training For Human-Centered Decision Making
Embodied decision-making is fundamental for AI agents operating in real-world
environments. While Visual Language Models (VLMs) have advanced this
capability, they still struggle with complex decisions, particularly in
human-centered situations that require deep reasoning about human needs and
values. In this study, we systematically evaluate open-sourced VLMs on
multimodal human-centered decision-making tasks. We find that LLMs receiving
only textual descriptions unexpectedly outperform their VLM counterparts of
similar scale that process actual images, suggesting that visual alignment may
hinder VLM abilities. To address this challenge, we propose a novel text-only
training approach with synthesized textual data. This method strengthens VLMs'
language components and transfers the learned abilities to multimodal
inference, eliminating the need for expensive image-text paired data.
Furthermore, we show that VLMs can achieve substantial performance gains
through self-improvement, using training data generated by their LLM
counterparts rather than relying on larger teacher models like GPT-4. Our
findings establish a more efficient and scalable approach to enhancing VLMs'
human-centered decision-making capabilities, opening new avenues for optimizing
VLMs through self-improvement mechanisms.
67e35c3bf049c252c672b824
Zhe Hu
67e35c3bf049c252c672b825
Jing Li
67e35c3bf049c252c672b826
Yu Yin
When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only
  Training For Human-Centered Decision Making
Embodied decision-making is fundamental for AI agents operating in real-world
environments. While Visual Language Models (VLMs) have advanced this
capability, they still struggle with complex decisions, particularly in
human-centered situations that require deep reasoning about human needs and
values. In this study, we systematically evaluate open-sourced VLMs on
multimodal human-centered decision-making tasks. We find that LLMs receiving
only textual descriptions unexpectedly outperform their VLM counterparts of
similar scale that process actual images, suggesting that visual alignment may
hinder VLM abilities. To address this challenge, we propose a novel text-only
training approach with synthesized textual data. This method strengthens VLMs'
language components and transfers the learned abilities to multimodal
inference, eliminating the need for expensive image-text paired data.
Furthermore, we show that VLMs can achieve substantial performance gains
through self-improvement, using training data generated by their LLM
counterparts rather than relying on larger teacher models like GPT-4. Our
findings establish a more efficient and scalable approach to enhancing VLMs'
human-centered decision-making capabilities, opening new avenues for optimizing
VLMs through self-improvement mechanisms.
VLMs enhanced with synthesized textual data outperform traditional multimodal VLMs and can improve through self-generated training data, optimizing human-centered decision-making.
Visual Language Models (VLMs) Large Language Models (LLMs) multimodal human-centered decision-making tasks visual alignment text-only training synthesized textual data self-improvement training data generation