LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
Visual encoding constitutes the basis of large multimodal models (LMMs) in
understanding the visual world. Conventional LMMs process images in fixed sizes
and limited resolutions, while recent explorations in this direction are
limited in adaptivity, efficiency, and even correctness. In this work, we first
take GPT-4V and LLaVA-1.5 as representative examples and expose systematic
flaws rooted in their visual encoding strategy. To address the challenges, we
present LLaVA-UHD, a large multimodal model that can efficiently perceive
images in any aspect ratio and high resolution. LLaVA-UHD includes three key
components: (1) An image modularization strategy that divides native-resolution
images into smaller variable-sized slices for efficient and extensible
encoding, (2) a compression module that further condenses image tokens from
visual encoders, and (3) a spatial schema to organize slice tokens for LLMs.
Comprehensive experiments show that LLaVA-UHD outperforms established LMMs
trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our
model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088)
resolution images using only 94% inference computation, and achieves 6.4
accuracy improvement on TextVQA. Moreover, the model can be efficiently trained
in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of
LLaVA-1.5). We make the data and code publicly available at
https://github.com/thunlp/LLaVA-UHD.
65f918b8ee1763d31ade7c29
Ruyi Xu
admin_assigned
2024-03-19T09:35:09.694Z
656db11a271c5c4e334e9c4f
/avatars/274dbf4c7f982397498c70b91bd6712f.svg
Xu Ruyi
user
xrorrim
65f918b8ee1763d31ade7c2a
Yuan Yao
65f918b8ee1763d31ade7c2b
Zonghao Guo
admin_assigned
2024-03-19T09:35:37.443Z
6491af36c1741666238f3bff
/avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
Zonghao Guo
user
guozonghao96
65f918b8ee1763d31ade7c2c
Junbo Cui
65f918b8ee1763d31ade7c2d
Zanlin Ni
admin_assigned
2024-03-19T09:35:52.289Z
63987ffb2ceb55aabe0852f3
/avatars/343b796ff6b8906203904e8c620d7eb5.svg
Zanlin Ni
user
nzl-thu
65f918b8ee1763d31ade7c2e
Chunjiang Ge
admin_assigned
2024-03-19T09:35:59.676Z
645e054ff7a55f0d780a8ff7
/avatars/9614510443bee3bd5d6266efd1c39fc1.svg
Chunjiang Ge
user
HelloJiang
65f918b8ee1763d31ade7c2f
Tat-Seng Chua
admin_assigned
2024-03-19T09:36:06.392Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
65f918b8ee1763d31ade7c30
Zhiyuan Liu
admin_assigned
2024-03-19T09:37:19.924Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
65f918b8ee1763d31ade7c31
Maosong Sun
65f918b8ee1763d31ade7c32
Gao Huang
LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images
Visual encoding constitutes the basis of large multimodal models (LMMs) in
understanding the visual world. Conventional LMMs process images in fixed sizes
and limited resolutions, while recent explorations in this direction are
limited in adaptivity, efficiency, and even correctness. In this work, we first
take GPT-4V and LLaVA-1.5 as representative examples and expose systematic
flaws rooted in their visual encoding strategy. To address the challenges, we
present LLaVA-UHD, a large multimodal model that can efficiently perceive
images in any aspect ratio and high resolution. LLaVA-UHD includes three key
components: (1) An image modularization strategy that divides native-resolution
images into smaller variable-sized slices for efficient and extensible
encoding, (2) a compression module that further condenses image tokens from
visual encoders, and (3) a spatial schema to organize slice tokens for LLMs.
Comprehensive experiments show that LLaVA-UHD outperforms established LMMs
trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our
model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088)
resolution images using only 94% inference computation, and achieves 6.4
accuracy improvement on TextVQA. Moreover, the model can be efficiently trained
in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of
LLaVA-1.5). We make the data and code publicly available at
https://github.com/thunlp/LLaVA-UHD.
LLaVA-UHD, a large multimodal model, efficiently processes images of any aspect ratio and high resolution by modularizing and compressing image tokens, outperforming established models on various benchmarks.
GPT-4V LLaVA-1.5 LLaVA-UHD image modularization compression module spatial schema native-resolution images variable-sized slices image tokens visual encoders TextVQA LLMs academic settings A100 GPUs