Train Small, Infer Large: Memory-Efficient LoRA Training for Large
  Language Models
Large Language Models (LLMs) have significantly advanced natural language
processing with exceptional task generalization capabilities. Low-Rank Adaption
(LoRA) offers a cost-effective fine-tuning solution, freezing the original
model parameters and training only lightweight, low-rank adapter matrices.
However, the memory footprint of LoRA is largely dominated by the original
model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA
training scheme founded on the intuition that many neurons in
over-parameterized LLMs have low training utility but are essential for
inference. LoRAM presents a unique twist: it trains on a pruned (small) model
to obtain pruned low-rank matrices, which are then recovered and utilized with
the original (large) model for inference. Additionally, minimal-cost continual
pre-training, performed by the model publishers in advance, aligns the
knowledge discrepancy between pruned and original models. Our extensive
experiments demonstrate the efficacy of LoRAM across various pruning strategies
and downstream tasks. For a model with 70 billion parameters, LoRAM enables
training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA
training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by
structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B
(LLaMA-2-70B), reduces the parameter storage cost that dominates the memory
usage in low-rank matrix training by 15.81times (16.95times), while
achieving dominant performance gains over both the original LLaMA-3.1-70B
(LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B).
67b68f883cd5860d8597eace
Jun Zhang
claimed_verified
2025-02-20T09:36:13.757Z
63fcb42c987f631186e554f2
/avatars/5cf87e9fa21c088c0bd8577d651d01f6.svg
Jun Zhang
user
junzhang98
67b68f883cd5860d8597eacf
Jue Wang
67b68f883cd5860d8597ead0
Huan Li
67b68f883cd5860d8597ead1
Lidan Shou
67b68f883cd5860d8597ead2
Ke Chen
67b68f883cd5860d8597ead3
Yang You
67b68f883cd5860d8597ead4
Guiming Xie
67b68f883cd5860d8597ead5
Xuejian Gong
67b68f883cd5860d8597ead6
Kunlong Zhou
admin_assigned
2025-02-20T17:43:26.626Z
63db54249f2687298a14dc4d
/avatars/7fd31c4056be5c304cd79e17e3a6e560.svg
kunlong zhou
user
kerlomz
Train Small, Infer Large: Memory-Efficient LoRA Training for Large
  Language Models
Large Language Models (LLMs) have significantly advanced natural language
processing with exceptional task generalization capabilities. Low-Rank Adaption
(LoRA) offers a cost-effective fine-tuning solution, freezing the original
model parameters and training only lightweight, low-rank adapter matrices.
However, the memory footprint of LoRA is largely dominated by the original
model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA
training scheme founded on the intuition that many neurons in
over-parameterized LLMs have low training utility but are essential for
inference. LoRAM presents a unique twist: it trains on a pruned (small) model
to obtain pruned low-rank matrices, which are then recovered and utilized with
the original (large) model for inference. Additionally, minimal-cost continual
pre-training, performed by the model publishers in advance, aligns the
knowledge discrepancy between pruned and original models. Our extensive
experiments demonstrate the efficacy of LoRAM across various pruning strategies
and downstream tasks. For a model with 70 billion parameters, LoRAM enables
training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA
training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by
structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B
(LLaMA-2-70B), reduces the parameter storage cost that dominates the memory
usage in low-rank matrix training by 15.81times (16.95times), while
achieving dominant performance gains over both the original LLaMA-3.1-70B
(LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B).
LoRAM is a memory-efficient fine-tuning method for large language models using pruned low-rank matrices, enabling significant performance gains with reduced memory usage.
LLMs Low-Rank Adaption LoRA LoRAM memory footprint over-parameterized models prune pruned low-rank matrices minimal-cost continual pre-training pruning strategies structured pruning quantization parameter storage cost