From Pixels to Words -- Towards Native Vision-Language Primitives at
  Scale
The edifice of native Vision-Language Models (VLMs) has emerged as a rising
contender to typical modular VLMs, shaped by evolving model architectures and
training paradigms. Yet, two lingering clouds cast shadows over its widespread
exploration and promotion: (-) What fundamental constraints set native VLMs
apart from modular ones, and to what extent can these barriers be overcome? (-)
How to make research in native VLMs more accessible and democratized, thereby
accelerating progress in the field. In this paper, we clarify these challenges
and outline guiding principles for constructing native VLMs. Specifically, one
native VLM primitive should: (i) effectively align pixel and word
representations within a shared semantic space; (ii) seamlessly integrate the
strengths of formerly separate vision and language modules; (iii) inherently
embody various cross-modal properties that support unified vision-language
encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of
native VLMs built from first principles, capable of rivaling top-tier modular
counterparts across diverse real-world scenarios. With only 390M image-text
examples, NEO efficiently develops visual perception from scratch while
mitigating vision-language conflicts inside a dense and monolithic model
crafted from our elaborate primitives. We position NEO as a cornerstone for
scalable and powerful native VLMs, paired with a rich set of reusable
components that foster a cost-effective and extensible ecosystem. Our code and
models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
68f19c5d6e0bef323a68fc30
Haiwen Diao
claimed_verified
2025-10-20T12:13:27.715Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
68f19c5d6e0bef323a68fc31
Mingxuan Li
claimed_verified
2025-10-18T04:49:51.200Z
647167fdc6b7783c87ba08ea
/avatars/0ffb0b8c4f38832e3d8864172d4c1d08.svg
li mingxuan
user
xjtulmx
68f19c5d6e0bef323a68fc32
Silei Wu
68f19c5d6e0bef323a68fc33
Linjun Dai
68f19c5d6e0bef323a68fc34
Xiaohua Wang
68f19c5d6e0bef323a68fc35
Hanming Deng
68f19c5d6e0bef323a68fc36
Lewei Lu
68f19c5d6e0bef323a68fc37
Dahua Lin
68f19c5d6e0bef323a68fc38
Ziwei Liu
From Pixels to Words -- Towards Native Vision-Language Primitives at
  Scale
The edifice of native Vision-Language Models (VLMs) has emerged as a rising
contender to typical modular VLMs, shaped by evolving model architectures and
training paradigms. Yet, two lingering clouds cast shadows over its widespread
exploration and promotion: (-) What fundamental constraints set native VLMs
apart from modular ones, and to what extent can these barriers be overcome? (-)
How to make research in native VLMs more accessible and democratized, thereby
accelerating progress in the field. In this paper, we clarify these challenges
and outline guiding principles for constructing native VLMs. Specifically, one
native VLM primitive should: (i) effectively align pixel and word
representations within a shared semantic space; (ii) seamlessly integrate the
strengths of formerly separate vision and language modules; (iii) inherently
embody various cross-modal properties that support unified vision-language
encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of
native VLMs built from first principles, capable of rivaling top-tier modular
counterparts across diverse real-world scenarios. With only 390M image-text
examples, NEO efficiently develops visual perception from scratch while
mitigating vision-language conflicts inside a dense and monolithic model
crafted from our elaborate primitives. We position NEO as a cornerstone for
scalable and powerful native VLMs, paired with a rich set of reusable
components that foster a cost-effective and extensible ecosystem. Our code and
models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
NEO, a novel family of native Vision-Language Models, addresses fundamental constraints and integrates vision and language within a unified framework, achieving competitive performance with limited data.
Vision-Language Models native VLMs modular VLMs pixel and word representations semantic space cross-modal properties visual perception vision-language conflicts monolithic model reusable components