Accelerating Vision Transformers with Adaptive Patch Sizes
Vision Transformers (ViTs) partition input images into uniformly sized
patches regardless of their content, resulting in long input sequence lengths
for high-resolution images. We present Adaptive Patch Transformers (APT), which
addresses this by using multiple different patch sizes within the same image.
APT reduces the total number of input tokens by allocating larger patch sizes
in more homogeneous areas and smaller patches in more complex ones. APT
achieves a drastic speedup in ViT inference and training, increasing throughput
by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and
can be applied to a previously fine-tuned ViT, converging in as little as 1
epoch. It also significantly reduces training and inference time without loss
of performance in high-resolution dense visual tasks, achieving up to 30\%
faster training and inference in visual QA, object detection, and semantic
segmentation.
68f9f0aeb9b2e4ae04673b2d
Rohan Choudhury
68f9f0aeb9b2e4ae04673b2e
JungEun Kim
68f9f0aeb9b2e4ae04673b2f
Jinhyung Park
68f9f0aeb9b2e4ae04673b30
Eunho Yang
68f9f0aeb9b2e4ae04673b31
László A. Jeni
68f9f0aeb9b2e4ae04673b32
Kris M. Kitani
Accelerating Vision Transformers with Adaptive Patch Sizes
Vision Transformers (ViTs) partition input images into uniformly sized
patches regardless of their content, resulting in long input sequence lengths
for high-resolution images. We present Adaptive Patch Transformers (APT), which
addresses this by using multiple different patch sizes within the same image.
APT reduces the total number of input tokens by allocating larger patch sizes
in more homogeneous areas and smaller patches in more complex ones. APT
achieves a drastic speedup in ViT inference and training, increasing throughput
by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and
can be applied to a previously fine-tuned ViT, converging in as little as 1
epoch. It also significantly reduces training and inference time without loss
of performance in high-resolution dense visual tasks, achieving up to 30\%
faster training and inference in visual QA, object detection, and semantic
segmentation.
Adaptive Patch Transformers (APT) improve Vision Transformer (ViT) efficiency by using variable patch sizes, enhancing speed without compromising performance.
Vision Transformers Adaptive Patch Transformers patch sizes input tokens throughput inference training visual QA object detection semantic segmentation