Learning to See Before Seeing: Demystifying LLM Visual Priors from
  Language Pre-training
Large Language Models (LLMs), despite being trained on text alone,
surprisingly develop rich visual priors. These priors allow latent visual
capabilities to be unlocked for vision tasks with a relatively small amount of
multimodal data, and in some cases, to perform visual tasks without ever having
seen an image. Through systematic analysis, we reveal that visual priors-the
implicit, emergent knowledge about the visual world acquired during language
pre-training-are composed of separable perception and reasoning priors with
unique scaling trends and origins. We show that an LLM's latent visual
reasoning ability is predominantly developed by pre-training on
reasoning-centric data (e.g., code, math, academia) and scales progressively.
This reasoning prior acquired from language pre-training is transferable and
universally applicable to visual reasoning. In contrast, a perception prior
emerges more diffusely from broad corpora, and perception ability is more
sensitive to the vision encoder and visual instruction tuning data. In
parallel, text describing the visual world proves crucial, though its
performance impact saturates rapidly. Leveraging these insights, we propose a
data-centric recipe for pre-training vision-aware LLMs and verify it in 1T
token scale pre-training. Our findings are grounded in over 100 controlled
experiments consuming 500,000 GPU-hours, spanning the full MLLM construction
pipeline-from LLM pre-training to visual alignment and supervised multimodal
fine-tuning-across five model scales, a wide range of data categories and
mixtures, and multiple adaptation setups. Along with our main findings, we
propose and investigate several hypotheses, and introduce the Multi-Level
Existence Bench (MLE-Bench). Together, this work provides a new way of
deliberately cultivating visual priors from language pre-training, paving the
way for the next generation of multimodal LLMs.
68dc87ff4159d1f2418f9a1a
Junlin Han
68dc87ff4159d1f2418f9a1b
Shengbang Tong
68dc87ff4159d1f2418f9a1c
David Fan
68dc87ff4159d1f2418f9a1d
Yufan Ren
68dc87ff4159d1f2418f9a1e
Koustuv Sinha
claimed_verified
2025-10-01T15:29:28.567Z
622b93067b9143726fbedc37
https://cdn-avatars.huggingface.co/v1/production/uploads/1647023505150-622b93067b9143726fbedc37.jpeg
Koustuv Sinha
user
koustuvs
68dc87ff4159d1f2418f9a1f
Philip Torr
68dc87ff4159d1f2418f9a20
Filippos Kokkinos
Learning to See Before Seeing: Demystifying LLM Visual Priors from
  Language Pre-training
Large Language Models (LLMs), despite being trained on text alone,
surprisingly develop rich visual priors. These priors allow latent visual
capabilities to be unlocked for vision tasks with a relatively small amount of
multimodal data, and in some cases, to perform visual tasks without ever having
seen an image. Through systematic analysis, we reveal that visual priors-the
implicit, emergent knowledge about the visual world acquired during language
pre-training-are composed of separable perception and reasoning priors with
unique scaling trends and origins. We show that an LLM's latent visual
reasoning ability is predominantly developed by pre-training on
reasoning-centric data (e.g., code, math, academia) and scales progressively.
This reasoning prior acquired from language pre-training is transferable and
universally applicable to visual reasoning. In contrast, a perception prior
emerges more diffusely from broad corpora, and perception ability is more
sensitive to the vision encoder and visual instruction tuning data. In
parallel, text describing the visual world proves crucial, though its
performance impact saturates rapidly. Leveraging these insights, we propose a
data-centric recipe for pre-training vision-aware LLMs and verify it in 1T
token scale pre-training. Our findings are grounded in over 100 controlled
experiments consuming 500,000 GPU-hours, spanning the full MLLM construction
pipeline-from LLM pre-training to visual alignment and supervised multimodal
fine-tuning-across five model scales, a wide range of data categories and
mixtures, and multiple adaptation setups. Along with our main findings, we
propose and investigate several hypotheses, and introduce the Multi-Level
Existence Bench (MLE-Bench). Together, this work provides a new way of
deliberately cultivating visual priors from language pre-training, paving the
way for the next generation of multimodal LLMs.
LLMs develop visual priors during language pre-training, which can be leveraged for vision tasks with minimal additional data, and these priors are composed of separable perception and reasoning components.
Large Language Models LLMs visual priors latent visual capabilities multimodal data visual tasks implicit knowledge visual world perception priors reasoning priors pre-training reasoning-centric data transferable visual reasoning perception ability vision encoder visual instruction tuning text describing the visual world vision-aware LLMs data-centric recipe pre-training visual alignment supervised multimodal fine-tuning Multi-Level Existence Bench MLE-Bench