Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Large foundation models are becoming ubiquitous, but training them from
scratch is prohibitively expensive. Thus, efficiently adapting these powerful
models to downstream tasks is increasingly important. In this paper, we study a
principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream
task adaptation. Despite demonstrating good generalizability, OFT still uses a
fairly large number of trainable parameters due to the high dimensionality of
orthogonal matrices. To address this, we start by examining OFT from an
information transmission perspective, and then identify a few key desiderata
that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast
Fourier transform algorithm enables efficient information transmission, we
propose an efficient orthogonal parameterization using butterfly structures. We
apply this parameterization to OFT, creating a novel parameter-efficient
finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a
special case, BOFT introduces a generalized orthogonal finetuning framework.
Finally, we conduct an extensive empirical study of adapting large vision
transformers, large language models, and text-to-image diffusion models to
various downstream tasks in vision and language.
65519c62a43b13ee347a088a
Weiyang Liu
admin_assigned
2023-11-13T08:47:18.767Z
648905d1a15c43c791d4381f
https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
Weiyang Liu
user
wy1iu
65519c62a43b13ee347a088b
Zeju Qiu
65519c62a43b13ee347a088c
Yao Feng
65519c62a43b13ee347a088d
Yuliang Xiu
admin_assigned
2023-11-13T08:48:39.195Z
6183d0b249ef1d984699e4a3
https://cdn-avatars.huggingface.co/v1/production/uploads/6183d0b249ef1d984699e4a3/M6B9b73v8su9H8jfdcbgu.jpeg
Yuliang Xiu
user
Yuliang
65519c62a43b13ee347a088e
Yuxuan Xue
admin_assigned
2023-11-13T08:49:28.761Z
641f12813c49fc33ced76018
/avatars/46f23d38364f86bdd445836f4ad155ec.svg
yuxuanxue
user
yuxuanxue
65519c62a43b13ee347a088f
Longhui Yu
admin_assigned
2023-11-13T08:49:41.033Z
64b753306c169983c982f609
https://cdn-avatars.huggingface.co/v1/production/uploads/64b753306c169983c982f609/oOC_N8cI1Go6avUMvIPVI.jpeg
Longhui Yu
user
Longhui98
65519c62a43b13ee347a0890
Haiwen Feng
claimed_verified
2024-03-22T13:24:43.811Z
644d2ede5c1b4e14d0c065af
/avatars/34945e0ab925d8eabc6580c71e053b11.svg
Haiwen (Haven) Feng
user
havenfeng
65519c62a43b13ee347a0891
Zhen Liu
65519c62a43b13ee347a0892
Juyeon Heo
65519c62a43b13ee347a0893
Songyou Peng
admin_assigned
2023-11-13T08:52:42.204Z
620cb0722d8bc91e1cf2e6e7
https://cdn-avatars.huggingface.co/v1/production/uploads/1644998719617-noauth.jpeg
Songyou Peng
user
syp
65519c62a43b13ee347a0894
Yandong Wen
admin_assigned
2023-11-13T08:52:51.533Z
643c21735fcffe09fb68a46f
/avatars/76aabacd318aa954d4c53094ad456056.svg
Yandong Wen
user
ydwen
65519c62a43b13ee347a0895
Michael J. Black
admin_assigned
2023-11-13T08:52:59.779Z
649fbababa7a05e3626546c8
/avatars/a9aa4b4914649527094fc1b12779e014.svg
Michael J. Black
user
MichaelJBlack
65519c62a43b13ee347a0896
Adrian Weller
65519c62a43b13ee347a0897
Bernhard Schölkopf
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Large foundation models are becoming ubiquitous, but training them from
scratch is prohibitively expensive. Thus, efficiently adapting these powerful
models to downstream tasks is increasingly important. In this paper, we study a
principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream
task adaptation. Despite demonstrating good generalizability, OFT still uses a
fairly large number of trainable parameters due to the high dimensionality of
orthogonal matrices. To address this, we start by examining OFT from an
information transmission perspective, and then identify a few key desiderata
that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast
Fourier transform algorithm enables efficient information transmission, we
propose an efficient orthogonal parameterization using butterfly structures. We
apply this parameterization to OFT, creating a novel parameter-efficient
finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a
special case, BOFT introduces a generalized orthogonal finetuning framework.
Finally, we conduct an extensive empirical study of adapting large vision
transformers, large language models, and text-to-image diffusion models to
various downstream tasks in vision and language.
A novel parameter-efficient finetuning method, Orthogonal Butterfly (BOFT), is proposed to enhance the generalization and reduce the computational cost of adapting large foundation models to downstream tasks.
Orthogonal Finetuning (OFT) parameter-efficiency butterfly structures Orthogonal Butterfly (BOFT) generalized orthogonal finetuning framework vision transformers language models text-to-image diffusion models