FasterViT: Fast Vision Transformers with Hierarchical Attention
We design a new family of hybrid CNN-ViT neural networks, named FasterViT,
with a focus on high image throughput for computer vision (CV) applications.
FasterViT combines the benefits of fast local representation learning in CNNs
and global modeling properties in ViT. Our newly introduced Hierarchical
Attention (HAT) approach decomposes global self-attention with quadratic
complexity into a multi-level attention with reduced computational costs. We
benefit from efficient window-based self-attention. Each window has access to
dedicated carrier tokens that participate in local and global representation
learning. At a high level, global self-attentions enable the efficient
cross-window communication at lower costs. FasterViT achieves a SOTA
Pareto-front in terms of accuracy \vs image throughput. We have extensively
validated its effectiveness on various CV tasks including classification,
object detection and segmentation. We also show that HAT can be used as a
plug-and-play module for existing networks and enhance them. We further
demonstrate significantly faster and more accurate performance than competitive
counterparts for images with high resolution. Code is available at
https://github.com/NVlabs/FasterViT.
6487e14ceec01aee99ce957d
Ali Hatamizadeh
claimed_verified
2023-06-13T17:39:15.440Z
64414b62603214724ebd2636
https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
Ali
user
ahatamiz
6487e14ceec01aee99ce957e
Greg Heinrich
claimed_verified
2024-10-10T08:03:33.528Z
64c23e06203c836866ed0bdd
https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
Greg Heinrich
user
gheinrich
6487e14ceec01aee99ce957f
Hongxu Yin
claimed_verified
2025-03-27T09:06:44.409Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
6487e14ceec01aee99ce9580
Andrew Tao
6487e14ceec01aee99ce9581
Jose M. Alvarez
6487e14ceec01aee99ce9582
Jan Kautz
6487e14ceec01aee99ce9583
Pavlo Molchanov
claimed_verified
2023-12-20T20:45:49.104Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
FasterViT: Fast Vision Transformers with Hierarchical Attention
We design a new family of hybrid CNN-ViT neural networks, named FasterViT,
with a focus on high image throughput for computer vision (CV) applications.
FasterViT combines the benefits of fast local representation learning in CNNs
and global modeling properties in ViT. Our newly introduced Hierarchical
Attention (HAT) approach decomposes global self-attention with quadratic
complexity into a multi-level attention with reduced computational costs. We
benefit from efficient window-based self-attention. Each window has access to
dedicated carrier tokens that participate in local and global representation
learning. At a high level, global self-attentions enable the efficient
cross-window communication at lower costs. FasterViT achieves a SOTA
Pareto-front in terms of accuracy \vs image throughput. We have extensively
validated its effectiveness on various CV tasks including classification,
object detection and segmentation. We also show that HAT can be used as a
plug-and-play module for existing networks and enhance them. We further
demonstrate significantly faster and more accurate performance than competitive
counterparts for images with high resolution. Code is available at
https://github.com/NVlabs/FasterViT.
FasterViT, a hybrid CNN-ViT model, enhances CV tasks with high image throughput through Hierarchical Attention (HAT) and achieves state-of-the-art performance in accuracy versus throughput.
CNN ViT FasterViT Hierarchical Attention (HAT) global self-attention window-based self-attention carrier tokens cross-window communication PARETO-front image classification object detection segmentation