Inverse-LLaVA: Eliminating Alignment Pre-training Through Text-to-Vision
  Mapping
Traditional multimodal learning approaches require expensive alignment
pre-training to bridge vision and language modalities, typically projecting
visual features into discrete text token spaces. We challenge both fundamental
assumptions underlying this paradigm by proposing Inverse-LLaVA, a novel
approach that eliminates alignment pre-training entirely while inverting the
conventional mapping direction. Rather than projecting visual features to text
space, our method maps text embeddings into continuous visual representation
space and performs fusion within transformer intermediate layers. Through
selective additive components in attention mechanisms, we enable dynamic
integration of visual and textual representations without requiring massive
image-text alignment datasets. Comprehensive experiments across nine multimodal
benchmarks demonstrate nuanced performance trade-offs: Inverse-LLaVA achieves
notable improvements on reasoning-intensive and cognitive tasks (MM-VET: +0.2%,
VizWiz: +1.8%, ScienceQA: +0.2%, cognitive reasoning: +27.2%), while showing
expected decreases in perception tasks requiring memorized visual-text
associations (celebrity recognition: -49.5%, OCR: -21.3%). These results
provide the first empirical evidence that alignment pre-training is not
necessary for effective multimodal learning, particularly for complex reasoning
tasks. Our work establishes the feasibility of a new paradigm that reduces
computational requirements by 45%, challenges conventional wisdom about
modality fusion, and opens new research directions for efficient multimodal
architectures that preserve modality-specific characteristics. Our project
website with code and additional resources is available at
https://inverse-llava.github.io.
68a3deb1b65388761d07446d
Xuhui Zhan
claimed_verified
2025-08-19T08:04:52.380Z
652c6ee9b118f26df765e4bb
https://cdn-avatars.huggingface.co/v1/production/uploads/652c6ee9b118f26df765e4bb/SwKLzRs5iVm0ORSCH-WLH.jpeg
Xuhui Zhan
user
xuhuizhan5
68a3deb1b65388761d07446e
Tyler Derr
Inverse-LLaVA: Eliminating Alignment Pre-training Through Text-to-Vision
  Mapping
Traditional multimodal learning approaches require expensive alignment
pre-training to bridge vision and language modalities, typically projecting
visual features into discrete text token spaces. We challenge both fundamental
assumptions underlying this paradigm by proposing Inverse-LLaVA, a novel
approach that eliminates alignment pre-training entirely while inverting the
conventional mapping direction. Rather than projecting visual features to text
space, our method maps text embeddings into continuous visual representation
space and performs fusion within transformer intermediate layers. Through
selective additive components in attention mechanisms, we enable dynamic
integration of visual and textual representations without requiring massive
image-text alignment datasets. Comprehensive experiments across nine multimodal
benchmarks demonstrate nuanced performance trade-offs: Inverse-LLaVA achieves
notable improvements on reasoning-intensive and cognitive tasks (MM-VET: +0.2%,
VizWiz: +1.8%, ScienceQA: +0.2%, cognitive reasoning: +27.2%), while showing
expected decreases in perception tasks requiring memorized visual-text
associations (celebrity recognition: -49.5%, OCR: -21.3%). These results
provide the first empirical evidence that alignment pre-training is not
necessary for effective multimodal learning, particularly for complex reasoning
tasks. Our work establishes the feasibility of a new paradigm that reduces
computational requirements by 45%, challenges conventional wisdom about
modality fusion, and opens new research directions for efficient multimodal
architectures that preserve modality-specific characteristics. Our project
website with code and additional resources is available at
https://inverse-llava.github.io.
Inverse-LLaVA eliminates alignment pre-training by mapping text embeddings into continuous visual representation space, improving reasoning tasks while reducing computational requirements.
Inverse-LLaVA multimodal learning alignment pre-training visual features text embeddings continuous visual representation space transformer intermediate layers attention mechanisms MM-VET VizWiz ScienceQA cognitive reasoning celebrity recognition OCR