POET is a reParameterized training algorithm using Orthogonal Equivalence Transformation to optimize neurons in large language models, ensuring stable training and improved generalization.
While large language models (LLMs) are driving the rapid advancement of
artificial intelligence, effectively and reliably training these large models
remains one of the field’s most significant challenges. To address this
challenge, we propose POET, a novel reParameterized training algorithm that
uses Orthogonal Equivalence Transformation to optimize neurons. Specifically,
POET reparameterizes each neuron with two learnable orthogonal matrices and a
fixed random weight matrix. Because of its provable preservation of spectral
properties of weight matrices, POET can stably optimize the objective function
with improved generalization. We further develop efficient approximations that
make POET flexible and scalable for training large-scale neural networks.
Extensive experiments validate the effectiveness and scalability of POET in
training LLMs.
