Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
67e375d3cc93cc8c42da7699
Baifeng Shi
claimed_verified
2025-03-26T20:44:24.349Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
67e375d3cc93cc8c42da769a
Boyi Li
67e375d3cc93cc8c42da769b
Han Cai
claimed_verified
2025-08-06T19:38:04.727Z
650e2b14c945dfc9386a7e28
/avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
Han Cai
user
han-cai
67e375d3cc93cc8c42da769c
Yao Lu
67e375d3cc93cc8c42da769d
Sifei Liu
67e375d3cc93cc8c42da769e
Marco Pavone
67e375d3cc93cc8c42da769f
Jan Kautz
67e375d3cc93cc8c42da76a0
Song Han
67e375d3cc93cc8c42da76a1
Trevor Darrell
67e375d3cc93cc8c42da76a2
Pavlo Molchanov
claimed_verified
2025-04-20T15:04:07.484Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
67e375d3cc93cc8c42da76a3
Hongxu Yin
claimed_verified
2025-03-27T09:04:59.461Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
PS3 enables high-resolution vision pre-training with reduced computational cost, improving multi-modal LLM performance and efficiency.
CLIP-style vision pre-training PS3 contrastive learning local regions detailed captions high-resolution representation learning saliency text prompt multi-modal LLM VILA-HD AnyRes S^2 high-resolution visual perception MLLM NVILA Qwen2-VL 4KPro image QA token pruning