Full Parameter Fine-tuning for Large Language Models with Limited
  Resources
Large Language Models (LLMs) have revolutionized Natural Language Processing
(NLP) but demand massive GPU resources for training. Lowering the threshold for
LLMs training would encourage greater participation from researchers,
benefiting both academia and society. While existing approaches have focused on
parameter-efficient fine-tuning, which tunes or adds a small number of
parameters, few have addressed the challenge of tuning the full parameters of
LLMs with limited resources. In this work, we propose a new optimizer,
LOw-Memory Optimization (LOMO), which fuses the gradient computation and the
parameter update in one step to reduce memory usage. By integrating LOMO with
existing memory saving techniques, we reduce memory usage to 10.8% compared to
the standard approach (DeepSpeed solution). Consequently, our approach enables
the full parameter fine-tuning of a 65B model on a single machine with 8 RTX
3090, each with 24GB memory.
648ff410274cc5506c9b1d9c
Kai Lv
claimed_verified
2023-06-19T11:44:27.450Z
6485d5b300c9cfe5c2470c81
/avatars/c29aa81d2add795e8448b99274a04b83.svg
Kai
user
KaiLv
648ff410274cc5506c9b1d9d
Yuqing Yang
claimed_verified
2025-05-26T08:47:03.133Z
65c0de12efbb14b39c97f78e
/avatars/18485f79427a35bd9e19f71b67c88dce.svg
Yuqing Yang
user
ayyyq
648ff410274cc5506c9b1d9e
Tengxiao Liu
648ff410274cc5506c9b1d9f
Qinghui Gao
648ff410274cc5506c9b1da0
Qipeng Guo
claimed_verified
2023-06-21T09:17:34.949Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
648ff410274cc5506c9b1da1
Xipeng Qiu
admin_assigned
2023-06-19T13:06:31.125Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
Full Parameter Fine-tuning for Large Language Models with Limited
  Resources
Large Language Models (LLMs) have revolutionized Natural Language Processing
(NLP) but demand massive GPU resources for training. Lowering the threshold for
LLMs training would encourage greater participation from researchers,
benefiting both academia and society. While existing approaches have focused on
parameter-efficient fine-tuning, which tunes or adds a small number of
parameters, few have addressed the challenge of tuning the full parameters of
LLMs with limited resources. In this work, we propose a new optimizer,
LOw-Memory Optimization (LOMO), which fuses the gradient computation and the
parameter update in one step to reduce memory usage. By integrating LOMO with
existing memory saving techniques, we reduce memory usage to 10.8% compared to
the standard approach (DeepSpeed solution). Consequently, our approach enables
the full parameter fine-tuning of a 65B model on a single machine with 8 RTX
3090, each with 24GB memory.
A new optimizer reduces memory usage for full parameter fine-tuning of large language models, enabling 65B model training on a single machine with limited GPUs.
Large Language Models Natural Language Processing parameter-efficient fine-tuning LOw-Memory Optimization gradient computation parameter update memory usage DeepSpeed solution RTX 3090