iFormer: Integrating ConvNet and Transformer for Mobile Application
We present a new family of mobile hybrid vision networks, called iFormer,
with a focus on optimizing latency and accuracy on mobile applications. iFormer
effectively integrates the fast local representation capacity of convolution
with the efficient global modeling ability of self-attention. The local
interactions are derived from transforming a standard convolutional network,
i.e., ConvNeXt, to design a more lightweight mobile network. Our newly
introduced mobile modulation attention removes memory-intensive operations in
MHA and employs an efficient modulation mechanism to boost dynamic global
representational capacity. We conduct comprehensive experiments demonstrating
that iFormer outperforms existing lightweight networks across various tasks.
Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k
with a latency of only 1.10 ms on an iPhone 13, surpassing the recently
proposed MobileNetV4 under similar latency constraints. Additionally, our
method shows significant improvements in downstream tasks, including COCO
object detection, instance segmentation, and ADE20k semantic segmentation,
while still maintaining low latency on mobile devices for high-resolution
inputs in these scenarios.
6798706dabdc35456a92212d
Chuanyang Zheng
admin_assigned
2025-01-28T14:16:05.976Z
65019cc870367843160fbb33
/avatars/f5455482fe9efbdeeea1bd3a3c119f02.svg
ZhengChuanyang
user
BillionZheng
iFormer: Integrating ConvNet and Transformer for Mobile Application
We present a new family of mobile hybrid vision networks, called iFormer,
with a focus on optimizing latency and accuracy on mobile applications. iFormer
effectively integrates the fast local representation capacity of convolution
with the efficient global modeling ability of self-attention. The local
interactions are derived from transforming a standard convolutional network,
i.e., ConvNeXt, to design a more lightweight mobile network. Our newly
introduced mobile modulation attention removes memory-intensive operations in
MHA and employs an efficient modulation mechanism to boost dynamic global
representational capacity. We conduct comprehensive experiments demonstrating
that iFormer outperforms existing lightweight networks across various tasks.
Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k
with a latency of only 1.10 ms on an iPhone 13, surpassing the recently
proposed MobileNetV4 under similar latency constraints. Additionally, our
method shows significant improvements in downstream tasks, including COCO
object detection, instance segmentation, and ADE20k semantic segmentation,
while still maintaining low latency on mobile devices for high-resolution
inputs in these scenarios.
iFormer combines convolution and self-attention to create a lightweight, high-performance mobile vision network with low latency across various tasks.
mobile hybrid vision networks iFormer convolution self-attention local interactions ConvNeXt mobile modulation attention memory-intensive operations modulation mechanism ImageNet-1k Top-1 accuracy latency iPhone 13 MobileNetV4 COCO object detection instance segmentation ADE20k semantic segmentation high-resolution inputs