MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
The input tokens to Vision Transformers carry little semantic meaning as they
are defined as regular equal-sized patches of the input image, regardless of
its content. However, processing uniform background areas of an image should
not necessitate as much compute as dense, cluttered areas. To address this
issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
method introduces a conditional gating mechanism that selects the optimal token
scale for every image region, such that the number of tokens is dynamically
determined per input. The proposed gating module is lightweight, agnostic to
the choice of transformer backbone, and trained within a few epochs (e.g., 20
epochs on ImageNet) with little training overhead. In addition, to enhance the
conditional behavior of the gate during training, we introduce a novel
generalization of the batch-shaping loss. We show that our gating module is
able to learn meaningful semantics despite operating locally at the coarse
patch-level. We validate MSViT on the tasks of classification and segmentation
where it leads to improved accuracy-complexity trade-off.
64a6456a01c586fa61330fd0
Jakob Drachmann Havtorn
claimed_verified
2023-07-07T07:34:53.837Z
64a6dd620517081ff06a6930
/avatars/521c0346cfe877a6c0b193c8a6490eba.svg
Jakob Drachmann Havtorn
user
Jaksen93
64a6456a01c586fa61330fd1
Amelie Royer
claimed_verified
2024-08-06T08:53:49.378Z
66b1ded53ddfb745eddfd072
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/JD85tZKReU8CWS-QDWnTU.png
Amelie Royer
user
ameroyer
64a6456a01c586fa61330fd2
Tijmen Blankevoort
64a6456a01c586fa61330fd3
Babak Ehteshami Bejnordi
claimed_verified
2023-07-07T07:35:00.615Z
645ce510f5760d1530d322cb
/avatars/139d1b5b2e6a83ef39298e4dc17258ff.svg
Babak Ehteshami Bejnordi
user
babakint
MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers
The input tokens to Vision Transformers carry little semantic meaning as they
are defined as regular equal-sized patches of the input image, regardless of
its content. However, processing uniform background areas of an image should
not necessitate as much compute as dense, cluttered areas. To address this
issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our
method introduces a conditional gating mechanism that selects the optimal token
scale for every image region, such that the number of tokens is dynamically
determined per input. The proposed gating module is lightweight, agnostic to
the choice of transformer backbone, and trained within a few epochs (e.g., 20
epochs on ImageNet) with little training overhead. In addition, to enhance the
conditional behavior of the gate during training, we introduce a novel
generalization of the batch-shaping loss. We show that our gating module is
able to learn meaningful semantics despite operating locally at the coarse
patch-level. We validate MSViT on the tasks of classification and segmentation
where it leads to improved accuracy-complexity trade-off.
A dynamic mixed-scale tokenization scheme for Vision Transformers adjusts token scale based on image content, improving accuracy and computational efficiency.
Vision Transformers tokenization scheme MSViT conditional gating mechanism batch-shaping loss classification segmentation accuracy-complexity trade-off