Optimize Weight Rounding via Signed Gradient Descent for the
  Quantization of LLMs
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at https://github.com/intel/neural-compressor soon.
64ffd7a696ec8a52185de903
Wenhua Cheng
admin_assigned
2023-09-12T12:17:58.296Z
634772780a9fa1ac49994510
/avatars/2e88e32ac0d45a1c624026e497eb00b3.svg
wenhua cheng
user
wenhuach
64ffd7a696ec8a52185de904
Weiwei Zhang
64ffd7a696ec8a52185de905
Haihao Shen
admin_assigned
2023-09-12T12:19:29.197Z
60ac3318e3de7c7440abb850
https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
Haihao Shen
user
Haihao
64ffd7a696ec8a52185de906
Yiyang Cai
admin_assigned
2023-09-12T12:19:37.996Z
634e0e3b7521b97439520abe
/avatars/d3aac6582dc4862ed640eff2ddb8b3c3.svg
Yiyang Cai
user
frankcai
64ffd7a696ec8a52185de907
Xin He
admin_assigned
2023-09-12T12:19:55.249Z
61e4c97506a27a84a78caaac
/avatars/5849f8d98755070cbee148a344c899ca.svg
He, Xin
user
xinhe
64ffd7a696ec8a52185de908
Kaokao Lv
admin_assigned
2023-09-12T12:20:08.353Z
630881ff6fb2ea4413f16390
/avatars/8cdf266604b24bc9ae599aa2def8debd.svg
lvkaokao
user
lvkaokao
Optimize Weight Rounding via Signed Gradient Descent for the
  Quantization of LLMs
Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at https://github.com/intel/neural-compressor soon.
SignRound, a lightweight block-wise tuning method using signed gradient descent, optimizes weight rounding in weight-only quantization of Large Language Models, outperforming RTN and other recent methods without inference overhead.
weight-only quantization 3-bit 4-bit up and down rounding perturbations SignRound signed gradient descent rounding-to-nearest (RTN)