Orthogonal Finetuning Made Scalable
Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation
while preventing catastrophic forgetting, but its high runtime and memory
demands limit practical deployment. We identify the core computational
bottleneck in OFT as its weight-centric implementation, which relies on costly
matrix-matrix multiplications with cubic complexity. To overcome this, we
propose OFTv2, an input-centric reformulation that instead uses matrix-vector
multiplications (i.e., matrix-free computation), reducing the computational
cost to quadratic. We further introduce the Cayley-Neumann parameterization, an
efficient orthogonal parameterization that approximates the matrix inversion in
Cayley transform via a truncated Neumann series. These modifications allow
OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage
without compromising performance. In addition, we extend OFTv2 to support
finetuning quantized foundation models and show that it outperforms the popular
QLoRA in training stability, efficiency, and memory usage.
685c03e9df8a0d6c70bbf924
Zeju Qiu
685c03e9df8a0d6c70bbf925
Weiyang Liu
685c03e9df8a0d6c70bbf926
Adrian Weller
685c03e9df8a0d6c70bbf927
Bernhard Schölkopf
Orthogonal Finetuning Made Scalable
Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation
while preventing catastrophic forgetting, but its high runtime and memory
demands limit practical deployment. We identify the core computational
bottleneck in OFT as its weight-centric implementation, which relies on costly
matrix-matrix multiplications with cubic complexity. To overcome this, we
propose OFTv2, an input-centric reformulation that instead uses matrix-vector
multiplications (i.e., matrix-free computation), reducing the computational
cost to quadratic. We further introduce the Cayley-Neumann parameterization, an
efficient orthogonal parameterization that approximates the matrix inversion in
Cayley transform via a truncated Neumann series. These modifications allow
OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage
without compromising performance. In addition, we extend OFTv2 to support
finetuning quantized foundation models and show that it outperforms the popular
QLoRA in training stability, efficiency, and memory usage.
OFTv2 optimizes orthogonal fine-tuning by shifting from matrix-matrix to matrix-vector multiplications and introducing efficient Cayley-Neumann parameterization, enhancing speed, memory usage, and performance.
Orthogonal finetuning (OFT) weight-centric implementation matrix-matrix multiplications matrix-vector multiplications Cayley-Neumann parameterization matrix-free computation Cayley transform Neumann series GPU memory usage QLoRA