LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
  Hierarchical Window Transformer
In multimodal large language models (MLLMs), vision transformers (ViTs) are
widely employed for visual encoding. However, their performance in solving
universal MLLM tasks is not satisfactory. We attribute it to a lack of
information from diverse visual levels, impeding alignment with the various
semantic granularity required for language generation. To address this issue,
we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
transformer that enables capturing diverse visual granularity by constructing
and integrating a high-resolution feature pyramid. As a vision-language
projector, Hiwin transformer comprises two primary modules: (i) an inverse
feature pyramid, constructed by a ViT-derived feature up-sampling process
utilizing high-frequency details from an image pyramid, and (ii) hierarchical
window attention, focusing on a set of key sampling features within cross-scale
windows to condense multi-level feature maps. Extensive experiments demonstrate
that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
benchmarks. Notably, our design brings an average boost of 3.7% across 14
benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
make all the data, model checkpoint, and code publicly available to facilitate
future research.
67640aa2eb7128270fa0404a
Yipeng Zhang
claimed_verified
2024-12-20T08:33:30.587Z
666e91b1623133f1ce35acc5
/avatars/cc78520e6cfb83817c1d0c1ac867ebdd.svg
YipengZhang
user
YipengZhang
67640aa2eb7128270fa0404b
Yifan Liu
67640aa2eb7128270fa0404c
Zonghao Guo
extracted_pending
2024-12-19T11:59:32.252Z
6491af36c1741666238f3bff
/avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
Zonghao Guo
user
guozonghao96
67640aa2eb7128270fa0404d
Yidan Zhang
67640aa2eb7128270fa0404e
Xuesong Yang
67640aa2eb7128270fa0404f
Chi Chen
claimed_verified
2025-01-10T08:21:37.240Z
642086ed290342c5df85662d
/avatars/915a4d7b89455ae97b8544c79286ddf8.svg
Chi Chen
user
carboncoo
67640aa2eb7128270fa04050
Jun Song
admin_assigned
2024-12-20T09:44:46.935Z
645b3bbb4093d5c140deff2d
/avatars/70a7c860f42af143c1f925d9bd2328d1.svg
sj
user
songjun
67640aa2eb7128270fa04051
Bo Zheng
67640aa2eb7128270fa04052
Yuan Yao
admin_assigned
2024-12-20T09:44:53.591Z
664618d83a7681520570b58a
/avatars/c03e25ee9bbf3504cdf2959acf81e0fe.svg
Yuan Yao
user
yuanyao
67640aa2eb7128270fa04053
Zhiyuan Liu
admin_assigned
2024-12-20T09:43:24.984Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67640aa2eb7128270fa04054
Tat-Seng Chua
admin_assigned
2024-12-20T09:43:10.504Z
6570ae84c4993b8fb96f41a8
/avatars/21f7d79d46ac4df0ecff8eca7678b33f.svg
Tat-Seng Chua
user
chuats
67640aa2eb7128270fa04055
Maosong Sun
LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
  Hierarchical Window Transformer
In multimodal large language models (MLLMs), vision transformers (ViTs) are
widely employed for visual encoding. However, their performance in solving
universal MLLM tasks is not satisfactory. We attribute it to a lack of
information from diverse visual levels, impeding alignment with the various
semantic granularity required for language generation. To address this issue,
we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
transformer that enables capturing diverse visual granularity by constructing
and integrating a high-resolution feature pyramid. As a vision-language
projector, Hiwin transformer comprises two primary modules: (i) an inverse
feature pyramid, constructed by a ViT-derived feature up-sampling process
utilizing high-frequency details from an image pyramid, and (ii) hierarchical
window attention, focusing on a set of key sampling features within cross-scale
windows to condense multi-level feature maps. Extensive experiments demonstrate
that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
benchmarks. Notably, our design brings an average boost of 3.7% across 14
benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
make all the data, model checkpoint, and code publicly available to facilitate
future research.
LLaVA-UHD v2, using a Hierarchical Window Transformer, improves performance in multimodal large language models by capturing diverse visual granularity, leading to superior results on benchmarks.
multimodal large language models (MLLMs) vision transformers (ViTs) Hierarchical window transformer inverse feature pyramid hierarchical window attention feature up-sampling feature pyramid cross-scale windows multi-level feature maps