Reparameterized LLM Training via Orthogonal Equivalence Transformation
While large language models (LLMs) are driving the rapid advancement of
artificial intelligence, effectively and reliably training these large models
remains one of the field's most significant challenges. To address this
challenge, we propose POET, a novel reParameterized training algorithm that
uses Orthogonal Equivalence Transformation to optimize neurons. Specifically,
POET reparameterizes each neuron with two learnable orthogonal matrices and a
fixed random weight matrix. Because of its provable preservation of spectral
properties of weight matrices, POET can stably optimize the objective function
with improved generalization. We further develop efficient approximations that
make POET flexible and scalable for training large-scale neural networks.
Extensive experiments validate the effectiveness and scalability of POET in
training LLMs.
684a649f9b38e1e5a33a6895
Zeju Qiu
684a649f9b38e1e5a33a6896
Simon Buchholz
684a649f9b38e1e5a33a6897
Tim Z. Xiao
684a649f9b38e1e5a33a6898
Maximilian Dax
684a649f9b38e1e5a33a6899
Bernhard Schölkopf
684a649f9b38e1e5a33a689a
Weiyang Liu
Reparameterized LLM Training via Orthogonal Equivalence Transformation
While large language models (LLMs) are driving the rapid advancement of
artificial intelligence, effectively and reliably training these large models
remains one of the field's most significant challenges. To address this
challenge, we propose POET, a novel reParameterized training algorithm that
uses Orthogonal Equivalence Transformation to optimize neurons. Specifically,
POET reparameterizes each neuron with two learnable orthogonal matrices and a
fixed random weight matrix. Because of its provable preservation of spectral
properties of weight matrices, POET can stably optimize the objective function
with improved generalization. We further develop efficient approximations that
make POET flexible and scalable for training large-scale neural networks.
Extensive experiments validate the effectiveness and scalability of POET in
training LLMs.
POET is a reParameterized training algorithm using Orthogonal Equivalence Transformation to optimize neurons in large language models, ensuring stable training and improved generalization.
POET reParameterized training algorithm Orthogonal Equivalence Transformation orthogonal matrices spectral properties weight matrices large language models generalization efficient approximations training scalability