NVILA: Efficient Frontier Visual Language Models
Visual language models (VLMs) have made significant advances in accuracy in
recent years. However, their efficiency has received much less attention. This
paper introduces NVILA, a family of open VLMs designed to optimize both
efficiency and accuracy. Building on top of VILA, we improve its model
architecture by first scaling up the spatial and temporal resolutions, and then
compressing visual tokens. This "scale-then-compress" approach enables NVILA to
efficiently process high-resolution images and long videos. We also conduct a
systematic investigation to enhance the efficiency of NVILA throughout its
entire lifecycle, from training and fine-tuning to deployment. NVILA matches or
surpasses the accuracy of many leading open and proprietary VLMs across a wide
range of image and video benchmarks. At the same time, it reduces training
costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by
1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and
models available to facilitate reproducibility.
67530600886878c8868445a9
Zhijian Liu
claimed_verified
2025-05-30T06:59:20.813Z
650dac79b959b0e1d41d7378
https://cdn-avatars.huggingface.co/v1/production/uploads/650dac79b959b0e1d41d7378/mzbN0MFk3k8b94FQ40I7L.jpeg
Zhijian Liu
user
zhijianliu
67530600886878c8868445aa
Ligeng Zhu
claimed_verified
2024-12-30T19:37:12.108Z
62b4b5beb25cb80fcf278354
https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
Ligeng Zhu
user
Ligeng-Zhu
67530600886878c8868445ab
Baifeng Shi
claimed_verified
2024-12-09T11:06:59.994Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
67530600886878c8868445ac
Zhuoyang Zhang
67530600886878c8868445ad
Yuming Lou
67530600886878c8868445ae
Shang Yang
claimed_verified
2025-02-21T10:00:24.530Z
641d8bacd526196afc12766d
/avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
Shang Yang
user
Shangy
67530600886878c8868445af
Haocheng Xi
claimed_verified
2025-05-27T07:57:11.027Z
66ce751a8ec9fda2cf5a9e85
/avatars/c17093ca81dad007b3e50bae503955a7.svg
Haocheng Xi
user
xihc-ucb
67530600886878c8868445b0
Shiyi Cao
claimed_verified
2024-12-09T11:06:56.294Z
64ebbae6895a36ab28de811a
https://cdn-avatars.huggingface.co/v1/production/uploads/64ebbae6895a36ab28de811a/gBiaQP4paS4L13eu-yRm7.jpeg
Shiyi Cao
user
eva98
67530600886878c8868445b1
Yuxian Gu
claimed_verified
2024-12-09T11:06:58.017Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
67530600886878c8868445b2
Dacheng Li
67530600886878c8868445b3
Xiuyu Li
67530600886878c8868445b4
Yunhao Fang
67530600886878c8868445b5
Yukang Chen
67530600886878c8868445b6
Cheng-Yu Hsieh
67530600886878c8868445b7
De-An Huang
67530600886878c8868445b8
An-Chieh Cheng
67530600886878c8868445b9
Vishwesh Nath
67530600886878c8868445ba
Jinyi Hu
claimed_verified
2024-12-09T11:06:54.373Z
637a06580a77f602dc4ac922
https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
Jinyi Hu
user
JamesHujy
67530600886878c8868445bb
Sifei Liu
67530600886878c8868445bc
Ranjay Krishna
67530600886878c8868445bd
Daguang Xu
67530600886878c8868445be
Xiaolong Wang
67530600886878c8868445bf
Pavlo Molchanov
claimed_verified
2025-04-20T15:04:52.036Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
67530600886878c8868445c0
Jan Kautz
67530600886878c8868445c1
Hongxu Yin
claimed_verified
2025-03-27T09:05:39.703Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
67530600886878c8868445c2
Song Han
67530600886878c8868445c3
Yao Lu
NVILA: Efficient Frontier Visual Language Models
Visual language models (VLMs) have made significant advances in accuracy in
recent years. However, their efficiency has received much less attention. This
paper introduces NVILA, a family of open VLMs designed to optimize both
efficiency and accuracy. Building on top of VILA, we improve its model
architecture by first scaling up the spatial and temporal resolutions, and then
compressing visual tokens. This "scale-then-compress" approach enables NVILA to
efficiently process high-resolution images and long videos. We also conduct a
systematic investigation to enhance the efficiency of NVILA throughout its
entire lifecycle, from training and fine-tuning to deployment. NVILA matches or
surpasses the accuracy of many leading open and proprietary VLMs across a wide
range of image and video benchmarks. At the same time, it reduces training
costs by 4.5X, fine-tuning memory usage by 3.4X, pre-filling latency by
1.6-2.2X, and decoding latency by 1.2-2.8X. We will soon make our code and
models available to facilitate reproducibility.
NVILA, a family of VLMs, optimizes efficiency and accuracy through a scale-then-compress approach, enhancing performance across various benchmarks while reducing computational costs.
Visual language models NVILA VILA scale-then-compress spatial resolutions temporal resolutions visual tokens system lifecycle training fine-tuning deployment training costs fine-tuning memory usage pre-filling latency decoding latency