ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training
  Multiplication-Less Reparameterization
Large language models (LLMs) have shown impressive performance on language
tasks but face challenges when deployed on resource-constrained devices due to
their extensive parameters and reliance on dense multiplications, resulting in
high memory demands and latency bottlenecks. Shift-and-add reparameterization
offers a promising solution by replacing costly multiplications with
hardware-friendly primitives in both the attention and multi-layer perceptron
(MLP) layers of an LLM. However, current reparameterization techniques require
training from scratch or full parameter fine-tuning to restore accuracy, which
is resource-intensive for LLMs. To address this, we propose accelerating
pretrained LLMs through post-training shift-and-add reparameterization,
creating efficient multiplication-free models, dubbed ShiftAddLLM.
Specifically, we quantize each weight matrix into binary matrices paired with
group-wise scaling factors. The associated multiplications are reparameterized
into (1) shifts between activations and scaling factors and (2) queries and
adds according to the binary matrices. To reduce accuracy loss, we present a
multi-objective optimization method to minimize both weight and output
activation reparameterization errors. Additionally, based on varying
sensitivity across layers to reparameterization, we develop an automated bit
allocation strategy to further reduce memory usage and latency. Experiments on
five LLM families and eight tasks consistently validate the effectiveness of
ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points
at comparable or lower latency compared to the most competitive quantized LLMs
at 3 and 2 bits, respectively, and more than 80% memory and energy reductions
over the original LLMs. Codes and models are available at
https://github.com/GATECH-EIC/ShiftAddLLM.
6667cef90c21d689e1936d2a
Haoran You
admin_assigned
2024-06-11T10:37:56.023Z
655fc82eb3c95b72253264e1
https://cdn-avatars.huggingface.co/v1/production/uploads/655fc82eb3c95b72253264e1/EzimumET_Io5lo7e_xW_F.jpeg
Haoran You
user
hryou1998
6667cef90c21d689e1936d2b
Yipin Guo
admin_assigned
2024-06-11T10:38:02.554Z
654329932cfe8660a394815e
/avatars/fc47c383509b4a354a1f93e109ccb025.svg
Yipin Guo
user
yipin0427
6667cef90c21d689e1936d2c
Yichao Fu
claimed_verified
2025-01-01T20:14:42.867Z
62ba66296501b0ff15ba1075
/avatars/de468727cb1240fd4b1f24a19fb237a6.svg
Yichao Fu
user
Viol2000
6667cef90c21d689e1936d2d
Wei Zhou
claimed_verified
2024-08-28T08:01:19.742Z
652dd0f4577d1e51cf57d5fe
/avatars/2ab6cd122e7c137cd6d10f35269341ac.svg
Wei Zhou
user
weizhou03
6667cef90c21d689e1936d2e
Huihong Shi
admin_assigned
2024-06-11T10:38:25.937Z
65ce9be34e5f19019db063b7
/avatars/9e8b7ed2486a7c72e5b65a92017c75e5.svg
Huihong Shi
user
shihuihong
6667cef90c21d689e1936d2f
Xiaofan Zhang
claimed_verified
2024-06-11T16:05:25.779Z
666870bfa2f41408ca503df8
https://cdn-avatars.huggingface.co/v1/production/uploads/666870bfa2f41408ca503df8/-4jeAGmoepwcezE0L-LvN.jpeg
Xiaofan Zhang
user
xiaofanzhang
6667cef90c21d689e1936d30
Souvik Kundu
extracted_pending
2024-06-11T04:13:46.788Z
654026a57b2743b58fa1dc65
/avatars/8d931fd1e582d35b3edc53216b2d0b75.svg
Souvik Kundu
user
souvikku
6667cef90c21d689e1936d31
Amir Yazdanbakhsh
admin_assigned
2024-06-11T10:38:46.873Z
6352dc90a023833e59d6349a
/avatars/76e5e7b51515b58e207c7d2a4b360d30.svg
Amir Yazdanbakhsh
user
ayazdan
6667cef90c21d689e1936d32
Yingyan Lin
claimed_verified
2024-06-11T13:26:39.432Z
659036962a0a886ef0246095
/avatars/4262b790058296a3f13603fbcd3304af.svg
celine lin
user
violin2023
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training
  Multiplication-Less Reparameterization
Large language models (LLMs) have shown impressive performance on language
tasks but face challenges when deployed on resource-constrained devices due to
their extensive parameters and reliance on dense multiplications, resulting in
high memory demands and latency bottlenecks. Shift-and-add reparameterization
offers a promising solution by replacing costly multiplications with
hardware-friendly primitives in both the attention and multi-layer perceptron
(MLP) layers of an LLM. However, current reparameterization techniques require
training from scratch or full parameter fine-tuning to restore accuracy, which
is resource-intensive for LLMs. To address this, we propose accelerating
pretrained LLMs through post-training shift-and-add reparameterization,
creating efficient multiplication-free models, dubbed ShiftAddLLM.
Specifically, we quantize each weight matrix into binary matrices paired with
group-wise scaling factors. The associated multiplications are reparameterized
into (1) shifts between activations and scaling factors and (2) queries and
adds according to the binary matrices. To reduce accuracy loss, we present a
multi-objective optimization method to minimize both weight and output
activation reparameterization errors. Additionally, based on varying
sensitivity across layers to reparameterization, we develop an automated bit
allocation strategy to further reduce memory usage and latency. Experiments on
five LLM families and eight tasks consistently validate the effectiveness of
ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points
at comparable or lower latency compared to the most competitive quantized LLMs
at 3 and 2 bits, respectively, and more than 80% memory and energy reductions
over the original LLMs. Codes and models are available at
https://github.com/GATECH-EIC/ShiftAddLLM.
Shift-and-add reparameterization of large language models reduces memory and latency without significant accuracy loss by converting weight matrices into binary matrices with group-wise scaling factors.
large language models LLMs shift-and-add reparameterization hardware-friendly primitives attention layers multi-layer perceptron MLP layers quantization weight matrices binary matrices group-wise scaling factors multi-objective optimization bit allocation strategy perplexity improvements