Selective Aggregation for Low-Rank Adaptation in Federated Learning
We investigate LoRA in federated learning through the lens of the asymmetry
analysis of the learned A and B matrices. In doing so, we uncover that A
matrices are responsible for learning general knowledge, while B matrices
focus on capturing client-specific knowledge. Based on this finding, we
introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two
low-rank trainable matrices A and B to model the weight update, but only
A matrices are shared with the server for aggregation. Moreover, we delve
into the relationship between the learned A and B matrices in other LoRA
variants, such as rsLoRA and VeRA, revealing a consistent pattern.
Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting
in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm
for integrating LoRA with FL, offering guidance for future work on subsequent
LoRA variants combined with FL. Extensive experimental results on natural
language understanding and generation tasks demonstrate the effectiveness of
the proposed method.
66fdfde460a069010d11c007
Pengxin Guo
claimed_verified
2024-10-03T08:31:28.860Z
668f440894dfc0ed1a7006ed
/avatars/fa0d328300b03bcbbf9b3a7532f28458.svg
Pengxin Guo
user
gpx333
66fdfde460a069010d11c008
Shuang Zeng
admin_assigned
2024-10-03T09:31:15.344Z
66712ff09c609c2484ce4aa0
/avatars/717b96ddef8a4c19ce07ea1fd9e9fd66.svg
Shuang Zeng
user
stevezs
66fdfde460a069010d11c009
Yanran Wang
admin_assigned
2024-10-03T09:31:22.188Z
656f698c80ff527c44e3c33b
/avatars/19ea552ed0bb36260ab0f6e41421f9b3.svg
Yanran Wang
user
yanranw1
66fdfde460a069010d11c00a
Huijie Fan
66fdfde460a069010d11c00b
Feifei Wang
66fdfde460a069010d11c00c
Liangqiong Qu
admin_assigned
2024-10-03T09:32:14.110Z
663058bc2653ec94f4a6235f
/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg
Liangqiong Qu
user
Liangqiong-QU
Selective Aggregation for Low-Rank Adaptation in Federated Learning
We investigate LoRA in federated learning through the lens of the asymmetry
analysis of the learned A and B matrices. In doing so, we uncover that A
matrices are responsible for learning general knowledge, while B matrices
focus on capturing client-specific knowledge. Based on this finding, we
introduce Federated Share-A Low-Rank Adaptation (FedSA-LoRA), which employs two
low-rank trainable matrices A and B to model the weight update, but only
A matrices are shared with the server for aggregation. Moreover, we delve
into the relationship between the learned A and B matrices in other LoRA
variants, such as rsLoRA and VeRA, revealing a consistent pattern.
Consequently, we extend our FedSA-LoRA method to these LoRA variants, resulting
in FedSA-rsLoRA and FedSA-VeRA. In this way, we establish a general paradigm
for integrating LoRA with FL, offering guidance for future work on subsequent
LoRA variants combined with FL. Extensive experimental results on natural
language understanding and generation tasks demonstrate the effectiveness of
the proposed method.
FedSA-LoRA integrates low-rank adaptation with federated learning by asymmetrically sharing only the general knowledge matrices, enhancing efficiency and effectiveness in natural language tasks.
LoRA federated learning asymmetry analysis A matrices B matrices general knowledge client-specific knowledge low-rank trainable matrices weight update FedSA-LoRA rsLoRA VeRA FedSA-rsLoRA FedSA-VeRA natural language understanding natural language generation