An Image is Worth More Than 16x16 Patches: Exploring Transformers on
  Individual Pixels
This work does not introduce a new method. Instead, we present an interesting
finding that questions the necessity of the inductive bias -- locality in
modern computer vision architectures. Concretely, we find that vanilla
Transformers can operate by directly treating each individual pixel as a token
and achieve highly performant results. This is substantially different from the
popular design in Vision Transformer, which maintains the inductive bias from
ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a
token). We mainly showcase the effectiveness of pixels-as-tokens across three
well-studied tasks in computer vision: supervised learning for object
classification, self-supervised learning via masked autoencoding, and image
generation with diffusion models. Although directly operating on individual
pixels is less computationally practical, we believe the community must be
aware of this surprising piece of knowledge when devising the next generation
of neural architectures for computer vision.
666ba4cc3814e415474a9064
Duy-Kien Nguyen
666ba4cc3814e415474a9065
Mahmoud Assran
666ba4cc3814e415474a9066
Unnat Jain
666ba4cc3814e415474a9067
Martin R. Oswald
admin_assigned
2024-06-14T09:40:04.008Z
651ed083e3c8628055890d9b
/avatars/4cb4d3361cfd7f205c82c6e15caa11f6.svg
Martin Oswald
user
martin-r-oswald
666ba4cc3814e415474a9068
Cees G. M. Snoek
666ba4cc3814e415474a9069
Xinlei Chen
admin_assigned
2024-06-14T09:40:17.429Z
63e58e3a006a775275e59e41
/avatars/75262a35b27a2ae1939df9118120d99e.svg
Xinlei Chen
user
endernewton
An Image is Worth More Than 16x16 Patches: Exploring Transformers on
  Individual Pixels
This work does not introduce a new method. Instead, we present an interesting
finding that questions the necessity of the inductive bias -- locality in
modern computer vision architectures. Concretely, we find that vanilla
Transformers can operate by directly treating each individual pixel as a token
and achieve highly performant results. This is substantially different from the
popular design in Vision Transformer, which maintains the inductive bias from
ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a
token). We mainly showcase the effectiveness of pixels-as-tokens across three
well-studied tasks in computer vision: supervised learning for object
classification, self-supervised learning via masked autoencoding, and image
generation with diffusion models. Although directly operating on individual
pixels is less computationally practical, we believe the community must be
aware of this surprising piece of knowledge when devising the next generation
of neural architectures for computer vision.
Vanilla Transformers achieve high performance in computer vision tasks by treating individual pixels as tokens, challenging the conventional use of local neighborhoods.
Transformers tokens Vision Transformer ConvNets local neighborhoods supervised learning object classification self-supervised learning masked autoencoding image generation diffusion models