Thermodynamic Natural Gradient Descent
Second-order training methods have better convergence properties than
gradient descent but are rarely used in practice for large-scale training due
to their computational overhead. This can be viewed as a hardware limitation
(imposed by digital computers). Here we show that natural gradient descent
(NGD), a second-order method, can have a similar computational complexity per
iteration to a first-order method, when employing appropriate hardware. We
present a new hybrid digital-analog algorithm for training neural networks that
is equivalent to NGD in a certain parameter regime but avoids prohibitively
costly linear system solves. Our algorithm exploits the thermodynamic
properties of an analog system at equilibrium, and hence requires an analog
thermodynamic computer. The training occurs in a hybrid digital-analog loop,
where the gradient and Fisher information matrix (or any other positive
semi-definite curvature matrix) are calculated at given time intervals while
the analog dynamics take place. We numerically demonstrate the superiority of
this approach over state-of-the-art digital first- and second-order training
methods on classification tasks and language model fine-tuning tasks.
66501d233fa1a30553ee2c03
Kaelan Donatella
extracted_confirmed
2024-05-24T12:17:20.890Z
665084f127935a4a528f53f8
/avatars/fb05a807a1f5f35c143082adc39dae12.svg
Kaelan Donatella
user
KaelanDt
66501d233fa1a30553ee2c04
Samuel Duffield
66501d233fa1a30553ee2c05
Maxwell Aifer
66501d233fa1a30553ee2c06
Denis Melanson
66501d233fa1a30553ee2c07
Gavin Crooks
66501d233fa1a30553ee2c08
Patrick J. Coles
Thermodynamic Natural Gradient Descent
Second-order training methods have better convergence properties than
gradient descent but are rarely used in practice for large-scale training due
to their computational overhead. This can be viewed as a hardware limitation
(imposed by digital computers). Here we show that natural gradient descent
(NGD), a second-order method, can have a similar computational complexity per
iteration to a first-order method, when employing appropriate hardware. We
present a new hybrid digital-analog algorithm for training neural networks that
is equivalent to NGD in a certain parameter regime but avoids prohibitively
costly linear system solves. Our algorithm exploits the thermodynamic
properties of an analog system at equilibrium, and hence requires an analog
thermodynamic computer. The training occurs in a hybrid digital-analog loop,
where the gradient and Fisher information matrix (or any other positive
semi-definite curvature matrix) are calculated at given time intervals while
the analog dynamics take place. We numerically demonstrate the superiority of
this approach over state-of-the-art digital first- and second-order training
methods on classification tasks and language model fine-tuning tasks.
A new hybrid digital-analog algorithm efficiently implements natural gradient descent, achieving superior performance in training neural networks compared to existing digital methods.
natural gradient descent NGD Fisher information matrix curvature matrix thermodynamic properties analog thermodynamic computer second-order training methods gradient descent hybrid digital-analog loop