Scaling MLPs: A Tale of Inductive Bias
In this work we revisit the most fundamental building block in deep learning,
the multi-layer perceptron (MLP), and study the limits of its performance on
vision tasks. Empirical insights into MLPs are important for multiple reasons.
(1) Given the recent narrative "less inductive bias is better", popularized due
to transformers eclipsing convolutional models, it is natural to explore the
limits of this hypothesis. To that end, MLPs offer an ideal test bed, being
completely free of any inductive bias. (2) MLPs have almost exclusively been
the main protagonist in the deep learning theory literature due to their
mathematical simplicity, serving as a proxy to explain empirical phenomena
observed for more complex architectures. Surprisingly, experimental datapoints
for MLPs are very difficult to find in the literature, especially when coupled
with large pre-training protocols. This discrepancy between practice and theory
is worrying: Do MLPs reflect the empirical advances exhibited by practical
models? Or do theorists need to rethink the role of MLPs as a proxy? We provide
insights into both these aspects. We show that the performance of MLPs
drastically improves with scale (93% on CIFAR10, 79% on CIFAR100, 69% on
TinyImageNet), highlighting that lack of inductive bias can indeed be
compensated. We observe that MLPs mimic the behaviour of their modern
counterparts faithfully, with some components in the learning setting however
surprisingly exhibiting stronger or unexpected behaviours. Due to their
inherent computational efficiency, large pre-training experiments become more
accessible for academic researchers. All of our experiments were run on a
single GPU.
64991d0276d49ee00f7c43f5
Gregor Bachmann
claimed_verified
2023-06-27T07:27:57.277Z
64996389a3f227b05cbd956f
/avatars/a586e7f99efdc7b61e05d62945575096.svg
Gregor Bachmann
user
gregorbachmann
64991d0276d49ee00f7c43f6
Sotiris Anagnostidis
admin_assigned
2023-06-26T07:37:24.102Z
62f8f4ff92e64c61bc6938da
/avatars/d386eb35d2c3d52186b2a8ec957f51bc.svg
Sotiris Anagnostidis
user
sanagnos
64991d0276d49ee00f7c43f7
Thomas Hofmann
Scaling MLPs: A Tale of Inductive Bias
In this work we revisit the most fundamental building block in deep learning,
the multi-layer perceptron (MLP), and study the limits of its performance on
vision tasks. Empirical insights into MLPs are important for multiple reasons.
(1) Given the recent narrative "less inductive bias is better", popularized due
to transformers eclipsing convolutional models, it is natural to explore the
limits of this hypothesis. To that end, MLPs offer an ideal test bed, being
completely free of any inductive bias. (2) MLPs have almost exclusively been
the main protagonist in the deep learning theory literature due to their
mathematical simplicity, serving as a proxy to explain empirical phenomena
observed for more complex architectures. Surprisingly, experimental datapoints
for MLPs are very difficult to find in the literature, especially when coupled
with large pre-training protocols. This discrepancy between practice and theory
is worrying: Do MLPs reflect the empirical advances exhibited by practical
models? Or do theorists need to rethink the role of MLPs as a proxy? We provide
insights into both these aspects. We show that the performance of MLPs
drastically improves with scale (93% on CIFAR10, 79% on CIFAR100, 69% on
TinyImageNet), highlighting that lack of inductive bias can indeed be
compensated. We observe that MLPs mimic the behaviour of their modern
counterparts faithfully, with some components in the learning setting however
surprisingly exhibiting stronger or unexpected behaviours. Due to their
inherent computational efficiency, large pre-training experiments become more
accessible for academic researchers. All of our experiments were run on a
single GPU.
MLPs achieve competitive performance on vision tasks with large-scale pre-training, challenging the narrative that inductive bias is necessary for high accuracy.
multi-layer perceptron (MLP) inductive bias transformers convolutional models pre-training protocols CIFAR10 CIFAR100 TinyImageNet computational efficiency