RiemannLoRA: A Unified Riemannian Framework for Ambiguity-Free LoRA
  Optimization
Low-Rank Adaptation (LoRA) has become a widely adopted standard for
parameter-efficient fine-tuning of large language models (LLMs), significantly
reducing memory and computational demands. However, challenges remain,
including finding optimal initialization strategies or mitigating
overparametrization in low-rank matrix factorization. In this work, we propose
a novel approach that addresses both of the challenges simultaneously within a
unified framework. Our method treats a set of fixed-rank LoRA matrices as a
smooth manifold. Considering adapters as elements on this manifold removes
overparametrization, while determining the direction of the fastest loss
decrease along the manifold provides initialization. Special care is taken to
obtain numerically stable and computationally efficient implementation of our
method, using best practices from numerical linear algebra and Riemannian
optimization. Experimental results on LLM and diffusion model architectures
demonstrate that RiemannLoRA consistently improves both convergence speed and
final performance over standard LoRA and its state-of-the-art modifications.
687a4aa3ff51f13d3863026b
Vladimir Bogachev
687a4aa3ff51f13d3863026c
Vladimir Aletov
687a4aa3ff51f13d3863026d
Alexander Molozhavenko
687a4aa3ff51f13d3863026e
Denis Bobkov
687a4aa3ff51f13d3863026f
Vera Soboleva
claimed_verified
2025-07-21T08:29:59.864Z
6385d420c12615765caa00b2
/avatars/c125b2d61423908d453231b046fb6a28.svg
Vera Soboleva
user
verasobol
687a4aa3ff51f13d38630270
Aibek Alanov
claimed_verified
2025-07-18T13:58:32.442Z
66680c6451545a8b46c6fd21
/avatars/03707f5ea4e2aa8dc825a9782b00ed85.svg
Aibek Alanov
user
ai-alanov
687a4aa3ff51f13d38630271
Maxim Rakhuba
RiemannLoRA: A Unified Riemannian Framework for Ambiguity-Free LoRA
  Optimization
Low-Rank Adaptation (LoRA) has become a widely adopted standard for
parameter-efficient fine-tuning of large language models (LLMs), significantly
reducing memory and computational demands. However, challenges remain,
including finding optimal initialization strategies or mitigating
overparametrization in low-rank matrix factorization. In this work, we propose
a novel approach that addresses both of the challenges simultaneously within a
unified framework. Our method treats a set of fixed-rank LoRA matrices as a
smooth manifold. Considering adapters as elements on this manifold removes
overparametrization, while determining the direction of the fastest loss
decrease along the manifold provides initialization. Special care is taken to
obtain numerically stable and computationally efficient implementation of our
method, using best practices from numerical linear algebra and Riemannian
optimization. Experimental results on LLM and diffusion model architectures
demonstrate that RiemannLoRA consistently improves both convergence speed and
final performance over standard LoRA and its state-of-the-art modifications.
RiemannLoRA addresses initialization and overparametrization in LoRA by treating LoRA matrices as a smooth manifold, improving convergence speed and performance in LLMs and diffusion models.
Low-Rank Adaptation LoRA parameter-efficient fine-tuning large language models low-rank matrix factorization smooth manifold Riemannian optimization RiemannLoRA convergence speed final performance diffusion models