StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
Zhizhong Li, Sina Sajadmanesh, Jingtao Li, Lingjuan Lyu
Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we introduce a geometry-aware extension of LoRA that uses a three-factor decomposition $USV^\top$, separating the adapter's input and output subspaces $V$ and $U$ from the scaling component $S$, in the spirit of singular value decomposition (SVD). Our method constrains $U$ and $V$ to lie on the Stiefel manifold, ensuring their orthonormality throughout training. To optimize on the Stiefel manifold, we employ a flexible geometric optimization design that converts any Euclidean optimizer to a Riemannian optimizer via a modular interface. This enables principled and stable subspace learning while remaining compatible with existing fine-tuning pipelines. Empirical results across a wide range of downstream tasks, including commonsense reasoning, math and code generation, image classification, and image generation, demonstrate the superior performance of our approach against the vanilla LoRA and recent state-of-the-art variants.