Just How Flexible are Neural Networks in Practice?
It is widely believed that a neural network can fit a training set containing
at least as many samples as it has parameters, underpinning notions of
overparameterized and underparameterized models. In practice, however, we only
find solutions accessible via our training procedure, including the optimizer
and regularizers, limiting flexibility. Moreover, the exact parameterization of
the function class, built into an architecture, shapes its loss surface and
impacts the minima we find. In this work, we examine the ability of neural
networks to fit data in practice. Our findings indicate that: (1) standard
optimizers find minima where the model can only fit training sets with
significantly fewer samples than it has parameters; (2) convolutional networks
are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
(3) while stochastic training is thought to have a regularizing effect, SGD
actually finds minima that fit more training data than full-batch gradient
descent; (4) the difference in capacity to fit correctly labeled and
incorrectly labeled samples can be predictive of generalization; (5) ReLU
activation functions result in finding minima that fit more data despite being
designed to avoid vanishing and exploding gradients in deep architectures.
66718b191ee1a34c606a95a0
Ravid Shwartz-Ziv
claimed_verified
2024-06-18T13:28:27.898Z
6303f7333926de1f7ec454f3
/avatars/1677a517b8635872a7a905aac727ff59.svg
shwartz ziv
user
ravid
66718b191ee1a34c606a95a1
Micah Goldblum
66718b191ee1a34c606a95a2
Arpit Bansal
66718b191ee1a34c606a95a3
C. Bayan Bruss
66718b191ee1a34c606a95a4
Yann LeCun
admin_assigned
2024-06-18T16:28:45.106Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
66718b191ee1a34c606a95a5
Andrew Gordon Wilson
Just How Flexible are Neural Networks in Practice?
It is widely believed that a neural network can fit a training set containing
at least as many samples as it has parameters, underpinning notions of
overparameterized and underparameterized models. In practice, however, we only
find solutions accessible via our training procedure, including the optimizer
and regularizers, limiting flexibility. Moreover, the exact parameterization of
the function class, built into an architecture, shapes its loss surface and
impacts the minima we find. In this work, we examine the ability of neural
networks to fit data in practice. Our findings indicate that: (1) standard
optimizers find minima where the model can only fit training sets with
significantly fewer samples than it has parameters; (2) convolutional networks
are more parameter-efficient than MLPs and ViTs, even on randomly labeled data;
(3) while stochastic training is thought to have a regularizing effect, SGD
actually finds minima that fit more training data than full-batch gradient
descent; (4) the difference in capacity to fit correctly labeled and
incorrectly labeled samples can be predictive of generalization; (5) ReLU
activation functions result in finding minima that fit more data despite being
designed to avoid vanishing and exploding gradients in deep architectures.
Neural networks trained with standard optimizers often find minima that fit fewer samples than their parameters, with convolutional networks showing greater efficiency and SGD outperforming full-batch gradient descent in fitting capacity.
neural networks overparameterized underparameterized optimizers regularizers loss surface minima convolutional networks MLPs ViTs stochastic training SGD full-batch gradient descent generalization ReLU activation functions