Adaptive Frequency Filters As Efficient Global Token Mixers
Recent vision transformers, large-kernel CNNs and MLPs have attained
remarkable successes in broad vision tasks thanks to their effective
information fusion in the global scope. However, their efficient deployments,
especially on mobile devices, still suffer from noteworthy challenges due to
the heavy computational costs of self-attention mechanisms, large kernels, or
fully connected layers. In this work, we apply conventional convolution theorem
to deep learning for addressing this and reveal that adaptive frequency filters
can serve as efficient global token mixers. With this insight, we propose
Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers
a latent representation to the frequency domain via a Fourier transform and
performs semantic-adaptive frequency filtering via an elementwise
multiplication, which mathematically equals to a token mixing operation in the
original latent space with a dynamic convolution kernel as large as the spatial
resolution of this latent representation. We take AFF token mixers as primary
neural operators to build a lightweight neural network, dubbed AFFNet.
Extensive experiments demonstrate the effectiveness of our proposed AFF token
mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs
compared to other lightweight network designs on broad visual tasks, including
visual recognition and dense prediction tasks.
64c244d8db947599b4ba220b
Zhipeng Huang
admin_assigned
2023-07-27T11:42:16.289Z
635629096b58fa7cc87476ee
/avatars/3d5291741cdcd26cb7cc737f76d1e3ad.svg
Zhipeng Huang
user
woli
64c244d8db947599b4ba220c
Zhizheng Zhang
64c244d8db947599b4ba220d
Cuiling Lan
64c244d8db947599b4ba220e
Zheng-Jun Zha
64c244d8db947599b4ba220f
Yan Lu
64c244d8db947599b4ba2210
Baining Guo
Adaptive Frequency Filters As Efficient Global Token Mixers
Recent vision transformers, large-kernel CNNs and MLPs have attained
remarkable successes in broad vision tasks thanks to their effective
information fusion in the global scope. However, their efficient deployments,
especially on mobile devices, still suffer from noteworthy challenges due to
the heavy computational costs of self-attention mechanisms, large kernels, or
fully connected layers. In this work, we apply conventional convolution theorem
to deep learning for addressing this and reveal that adaptive frequency filters
can serve as efficient global token mixers. With this insight, we propose
Adaptive Frequency Filtering (AFF) token mixer. This neural operator transfers
a latent representation to the frequency domain via a Fourier transform and
performs semantic-adaptive frequency filtering via an elementwise
multiplication, which mathematically equals to a token mixing operation in the
original latent space with a dynamic convolution kernel as large as the spatial
resolution of this latent representation. We take AFF token mixers as primary
neural operators to build a lightweight neural network, dubbed AFFNet.
Extensive experiments demonstrate the effectiveness of our proposed AFF token
mixer and show that AFFNet achieve superior accuracy and efficiency trade-offs
compared to other lightweight network designs on broad visual tasks, including
visual recognition and dense prediction tasks.
A new Adaptive Frequency Filtering (AFF) token mixer based on Fourier transforms improves accuracy and efficiency in lightweight neural networks for visual tasks.
vision transformers large-kernel CNNs MLPs self-attention mechanisms adaptive frequency filters Fourier transform AFF token mixer AFFNet visual recognition dense prediction tasks