Mobile V-MoEs: Scaling Down Vision Transformers via Sparse
  Mixture-of-Experts
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse MoEs to
scale-down Vision Transformers (ViTs) to make them more attractive for
resource-constrained vision applications. To this end, we propose a simplified
and mobile-friendly MoE design where entire images rather than individual
patches are routed to the experts. We also propose a stable MoE training
procedure that uses super-class information to guide the router. We empirically
show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
between performance and efficiency than the corresponding dense ViTs. For
example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
64fe72b048411fc789d43764
Erik Daxberger
64fe72b048411fc789d43765
Floris Weers
claimed_verified
2024-03-15T11:02:53.284Z
65f41f1d6724ea19d7bc9052
/avatars/3171a092b40da7518bce00b80f7c6ff0.svg
Floris Weers
user
fweers
64fe72b048411fc789d43766
Bowen Zhang
64fe72b048411fc789d43767
Tom Gunter
64fe72b048411fc789d43768
Ruoming Pang
64fe72b048411fc789d43769
Marcin Eichner
64fe72b048411fc789d4376a
Michael Emmersberger
64fe72b048411fc789d4376b
Yinfei Yang
64fe72b048411fc789d4376c
Alexander Toshev
64fe72b048411fc789d4376d
Xianzhi Du
Mobile V-MoEs: Scaling Down Vision Transformers via Sparse
  Mixture-of-Experts
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse MoEs to
scale-down Vision Transformers (ViTs) to make them more attractive for
resource-constrained vision applications. To this end, we propose a simplified
and mobile-friendly MoE design where entire images rather than individual
patches are routed to the experts. We also propose a stable MoE training
procedure that uses super-class information to guide the router. We empirically
show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
between performance and efficiency than the corresponding dense ViTs. For
example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.
Sparse Mobile Vision MoEs are applied to reduce resource requirements in Vision Transformers, achieving improved performance and efficiency compared to dense models.
Sparse Mixture-of-Experts Visual Transformers MoE ViTs image routing super-class information router Mobile Vision MoEs V-MoEs