Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
67e375d3cc93cc8c42da7699
Baifeng Shi
claimed_verified
2025-03-26T20:44:24.349Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
67e375d3cc93cc8c42da769a
Boyi Li
67e375d3cc93cc8c42da769b
Han Cai
67e375d3cc93cc8c42da769c
Yao Lu
67e375d3cc93cc8c42da769d
Sifei Liu
67e375d3cc93cc8c42da769e
Marco Pavone
67e375d3cc93cc8c42da769f
Jan Kautz
67e375d3cc93cc8c42da76a0
Song Han
67e375d3cc93cc8c42da76a1
Trevor Darrell
67e375d3cc93cc8c42da76a2
Pavlo Molchanov
67e375d3cc93cc8c42da76a3
Hongxu Yin
claimed_verified
2025-03-27T09:04:59.461Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
Scaling Vision Pre-Training to 4K Resolution
High-resolution perception of visual details is crucial for daily tasks.
Current vision pre-training, however, is still limited to low resolutions
(e.g., 378 x 378 pixels) due to the quadratic cost of processing larger images.
We introduce PS3 that scales CLIP-style vision pre-training to 4K resolution
with a near-constant cost. Instead of contrastive learning on global image
representation, PS3 is pre-trained by selectively processing local regions and
contrasting them with local detailed captions, enabling high-resolution
representation learning with greatly reduced computational overhead. The
pre-trained PS3 is able to both encode the global image at low resolution and
selectively process local high-resolution regions based on their saliency or
relevance to a text prompt. When applying PS3 to multi-modal LLM (MLLM), the
resulting model, named VILA-HD, significantly improves high-resolution visual
perception compared to baselines without high-resolution vision pre-training
such as AnyRes and S^2 while using up to 4.3x fewer tokens. PS3 also unlocks
appealing scaling properties of VILA-HD, including scaling up resolution for
free and scaling up test-time compute for better performance. Compared to state
of the arts, VILA-HD outperforms previous MLLMs such as NVILA and Qwen2-VL
across multiple benchmarks and achieves better efficiency than latest token
pruning approaches. Finally, we find current benchmarks do not require
4K-resolution perception, which motivates us to propose 4KPro, a new benchmark
of image QA at 4K resolution, on which VILA-HD outperforms all previous MLLMs,
including a 14.5% improvement over GPT-4o, and a 3.2% improvement and 2.96x
speedup over Qwen2-VL.
PS3 CLIP-style vision pre-training contrastive learning local regions local detailed captions high-resolution representation learning computational overhead saliency text prompt VILA-HD multi-modal LLM high-resolution visual perception AnyRes S^2 scaling properties test-time compute NVILA Qwen2-VL benchmarks token pruning approaches 4KPer image QA GPT-4o