FedSVD: Adaptive Orthogonalization for Private Federated Learning with
  LoRA
Low-Rank Adaptation (LoRA), which introduces a product of two trainable
low-rank matrices into frozen pre-trained weights, is widely used for efficient
fine-tuning of language models in federated learning (FL). However, when
combined with differentially private stochastic gradient descent (DP-SGD), LoRA
faces substantial noise amplification: DP-SGD perturbs per-sample gradients,
and the matrix multiplication of the LoRA update (BA) intensifies this
effect. Freezing one matrix (e.g., A) reduces the noise but restricts model
expressiveness, often resulting in suboptimal adaptation. To address this, we
propose FedSVD, a simple yet effective method that introduces a global
reparameterization based on singular value decomposition (SVD). In our
approach, each client optimizes only the B matrix and transmits it to the
server. The server aggregates the B matrices, computes the product BA using
the previous A, and refactorizes the result via SVD. This yields a new
adaptive A composed of the orthonormal right singular vectors of BA, and an
updated B containing the remaining SVD components. This reparameterization
avoids quadratic noise amplification, while allowing A to better capture the
principal directions of the aggregate updates. Moreover, the orthonormal
structure of A bounds the gradient norms of B and preserves more signal
under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD
consistently improves stability and performance across a variety of privacy
settings and benchmarks, outperforming relevant baselines under both private
and non-private regimes.
682bfcec8081928badd176e7
Seanie Lee
claimed_verified
2025-05-20T07:20:22.246Z
64ad5f59b7e4b2c1ce47eb43
/avatars/1f13ebe21a90d8c99920aa2c8cd9ac45.svg
Seanie Lee
user
Seanie-lee
682bfcec8081928badd176e8
Sangwoo Park
claimed_verified
2025-05-21T09:03:34.792Z
638716c14e00d7fc0902fef4
/avatars/5fa8152f8c0e4e600d1a64802c3e0103.svg
Sangwoo Park
user
Sangsang
682bfcec8081928badd176e9
Dong Bok Lee
admin_assigned
2025-05-20T08:31:04.999Z
64f000769e7770db74d44bba
/avatars/d015820380ffb823b1b35df64dcd3457.svg
Dong-Bok Lee
user
dongboklee
682bfcec8081928badd176ea
Dominik Wagner
admin_assigned
2025-05-20T08:31:33.334Z
6311ba6f05cc08a1408d910a
https://cdn-avatars.huggingface.co/v1/production/uploads/1662997515866-6311ba6f05cc08a1408d910a.png
Dominik Wagner
user
dwgnr
682bfcec8081928badd176eb
Haebin Seong
admin_assigned
2025-05-20T08:31:39.425Z
63a9379e2e05ca32e352d93b
/avatars/6cda37befc873a92ed6d5dcba507954a.svg
Haebin Seong
user
hbseong
682bfcec8081928badd176ec
Tobias Bocklet
682bfcec8081928badd176ed
Juho Lee
682bfcec8081928badd176ee
Sung Ju Hwang
FedSVD: Adaptive Orthogonalization for Private Federated Learning with
  LoRA
Low-Rank Adaptation (LoRA), which introduces a product of two trainable
low-rank matrices into frozen pre-trained weights, is widely used for efficient
fine-tuning of language models in federated learning (FL). However, when
combined with differentially private stochastic gradient descent (DP-SGD), LoRA
faces substantial noise amplification: DP-SGD perturbs per-sample gradients,
and the matrix multiplication of the LoRA update (BA) intensifies this
effect. Freezing one matrix (e.g., A) reduces the noise but restricts model
expressiveness, often resulting in suboptimal adaptation. To address this, we
propose FedSVD, a simple yet effective method that introduces a global
reparameterization based on singular value decomposition (SVD). In our
approach, each client optimizes only the B matrix and transmits it to the
server. The server aggregates the B matrices, computes the product BA using
the previous A, and refactorizes the result via SVD. This yields a new
adaptive A composed of the orthonormal right singular vectors of BA, and an
updated B containing the remaining SVD components. This reparameterization
avoids quadratic noise amplification, while allowing A to better capture the
principal directions of the aggregate updates. Moreover, the orthonormal
structure of A bounds the gradient norms of B and preserves more signal
under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD
consistently improves stability and performance across a variety of privacy
settings and benchmarks, outperforming relevant baselines under both private
and non-private regimes.
FedSVD introduces a method for stable and effective fine-tuning of pre-trained language models in federated learning with differential privacy by using singular value decomposition to handle noise and maintain performance.
Low-Rank Adaptation LoRA federated learning FL differentially private stochastic gradient descent DP-SGD singular value decomposition SVD matrix multiplication global reparameterization orthonormal right singular vectors gradient norms