Parameter-Inverted Image Pyramid Networks for Visual Perception and
  Multimodal Understanding
Image pyramids are widely adopted in top-performing methods to obtain
multi-scale features for precise visual perception and understanding. However,
current image pyramids use the same large-scale model to process multiple
resolutions of images, leading to significant computational cost. To address
this challenge, we propose a novel network architecture, called
Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses
pretrained models (ViTs or CNNs) as branches to process multi-scale images,
where images of higher resolutions are processed by smaller network branches to
balance computational cost and performance. To integrate information from
different spatial scales, we further propose a novel cross-branch feature
interaction mechanism. To validate PIIP, we apply it to various perception
models and a representative multimodal large language model called LLaVA, and
conduct extensive experiments on various tasks such as object detection,
segmentation, image classification and multimodal understanding. PIIP achieves
superior performance compared to single-branch and existing multi-resolution
approaches with lower computational cost. When applied to InternViT-6B, a
large-scale vision foundation model, PIIP can improve its performance by 1%-2%
on detection and segmentation with only 40%-60% of the original computation,
finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For
multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and
74.5% on MMBench with only 2.8M training data. Our code is released at
https://github.com/OpenGVLab/PIIP.
6788c7156bc665d65caf8b3c
Zhaokai Wang
claimed_verified
2025-01-16T08:49:11.408Z
665d4b515fdfe8f923e347a7
/avatars/d114b24c02dadfca0a8aee104755a8ec.svg
Zhaokai Wang
user
wzk1015
6788c7156bc665d65caf8b3d
Xizhou Zhu
admin_assigned
2025-01-16T08:51:26.043Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
6788c7156bc665d65caf8b3e
Xue Yang
claimed_verified
2025-04-04T07:10:49.767Z
648e77184cae4f6921dbb382
https://cdn-avatars.huggingface.co/v1/production/uploads/648e77184cae4f6921dbb382/zAAJRvOStC0wZplqVWrk_.jpeg
Xue Yang
user
yangxue
6788c7156bc665d65caf8b3f
Gen Luo
admin_assigned
2025-01-16T08:51:36.685Z
650aac7c23196fb2d86a0b37
/avatars/418035a2e8f514118bc67d16ee41b6b0.svg
Gen Luo
user
favor123
6788c7156bc665d65caf8b40
Hao Li
6788c7156bc665d65caf8b41
Changyao Tian
admin_assigned
2025-01-16T08:51:46.333Z
64b7475efa7eabaae5f7ba94
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
Changyao Tian
user
Changyao
6788c7156bc665d65caf8b42
Wenhan Dou
admin_assigned
2025-01-16T08:51:57.729Z
66efe658de163a536aa84178
/avatars/de547ed48ca37694e08ee926aea03d9f.svg
dou wenhan
user
douwh
6788c7156bc665d65caf8b43
Junqi Ge
6788c7156bc665d65caf8b44
Lewei Lu
admin_assigned
2025-01-16T08:52:10.641Z
65ead3ea908526a39082e641
/avatars/dcf870695fd56b06ca03d82f831e9019.svg
Lewei Lu
user
luotto
6788c7156bc665d65caf8b45
Yu Qiao
6788c7156bc665d65caf8b46
Jifeng Dai
admin_assigned
2025-01-16T08:52:18.008Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
Parameter-Inverted Image Pyramid Networks for Visual Perception and
  Multimodal Understanding
Image pyramids are widely adopted in top-performing methods to obtain
multi-scale features for precise visual perception and understanding. However,
current image pyramids use the same large-scale model to process multiple
resolutions of images, leading to significant computational cost. To address
this challenge, we propose a novel network architecture, called
Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses
pretrained models (ViTs or CNNs) as branches to process multi-scale images,
where images of higher resolutions are processed by smaller network branches to
balance computational cost and performance. To integrate information from
different spatial scales, we further propose a novel cross-branch feature
interaction mechanism. To validate PIIP, we apply it to various perception
models and a representative multimodal large language model called LLaVA, and
conduct extensive experiments on various tasks such as object detection,
segmentation, image classification and multimodal understanding. PIIP achieves
superior performance compared to single-branch and existing multi-resolution
approaches with lower computational cost. When applied to InternViT-6B, a
large-scale vision foundation model, PIIP can improve its performance by 1%-2%
on detection and segmentation with only 40%-60% of the original computation,
finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For
multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and
74.5% on MMBench with only 2.8M training data. Our code is released at
https://github.com/OpenGVLab/PIIP.
Parameter-Inverted Image Pyramid Networks (PIIP) use pretrained models to efficiently process multi-scale images, reducing computational cost while improving performance across various perception and multimodal understanding tasks.
Parameter-Inverted Image Pyramid Networks PIIP ViTs CNNs cross-branch feature interaction InternViT-6B MS COCO ADE20K TextVQA MMBench box AP mIoU