LoRAShear: Efficient Large Language Model Structured Pruning and
  Knowledge Recovery
Large Language Models (LLMs) have transformed the landscape of artificial
intelligence, while their enormous size presents significant challenges in
terms of computational costs. We introduce LoRAShear, a novel efficient
approach to structurally prune LLMs and recover knowledge. Given general LLMs,
LoRAShear first creates the dependency graphs to discover minimally removal
structures and analyze the knowledge distribution. It then proceeds progressive
structured pruning on LoRA adaptors and enables inherent knowledge transfer to
better preserve the information in the redundant structures. To recover the
lost knowledge during pruning, LoRAShear meticulously studies and proposes a
dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow
down the performance gap to the full models. Numerical results demonstrate that
by only using one GPU within a couple of GPU days, LoRAShear effectively
reduced footprint of LLMs by 20% with only 1.0% performance degradation and
significantly outperforms state-of-the-arts. The source code will be available
at https://github.com/microsoft/lorashear.
654089d31c4b203c012e6da6
Tianyi Chen
claimed_verified
2023-10-31T17:02:01.315Z
64ad94f05a4a60156925ec96
/avatars/643bdb076e703bfcc89cec6fccb756c6.svg
Tianyi Chen
user
tianyic
654089d31c4b203c012e6da7
Tianyu Ding
admin_assigned
2023-10-31T08:24:35.618Z
646b0c6d3e2a7b0659452fc7
https://cdn-avatars.huggingface.co/v1/production/uploads/646b0c6d3e2a7b0659452fc7/aqT9nbrJtOk38tDw_12xW.jpeg
Tianyu Ding
user
tding1
654089d31c4b203c012e6da8
Badal Yadav
654089d31c4b203c012e6da9
Ilya Zharkov
654089d31c4b203c012e6daa
Luming Liang
LoRAShear: Efficient Large Language Model Structured Pruning and
  Knowledge Recovery
Large Language Models (LLMs) have transformed the landscape of artificial
intelligence, while their enormous size presents significant challenges in
terms of computational costs. We introduce LoRAShear, a novel efficient
approach to structurally prune LLMs and recover knowledge. Given general LLMs,
LoRAShear first creates the dependency graphs to discover minimally removal
structures and analyze the knowledge distribution. It then proceeds progressive
structured pruning on LoRA adaptors and enables inherent knowledge transfer to
better preserve the information in the redundant structures. To recover the
lost knowledge during pruning, LoRAShear meticulously studies and proposes a
dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow
down the performance gap to the full models. Numerical results demonstrate that
by only using one GPU within a couple of GPU days, LoRAShear effectively
reduced footprint of LLMs by 20% with only 1.0% performance degradation and
significantly outperforms state-of-the-arts. The source code will be available
at https://github.com/microsoft/lorashear.
LoRAShear is an efficient method for pruning and knowledge recovery in Large Language Models, reducing their size by 20% with minimal performance loss.
Large Language Models LLMs LoRAShear dependency graphs structured pruning LoRA adaptors knowledge transfer dynamic fine-tuning dynamic data adaptors