ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric
  Strategy for Diverse Generative Tasks
This study examines 4-bit quantization methods like GPTQ in large language
models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
extend task scope to more generative categories such as code generation and
abstractive summarization, in which we found that INT4 quantization can
significantly underperform. However, simply shifting to higher precision
formats like FP6 has been particularly challenging, thus overlooked, due to
poor performance caused by the lack of sophisticated integration and system
acceleration strategies on current AI hardware. Our results show that FP6, even
with a coarse-grain quantization scheme, performs robustly across various
algorithms and tasks, demonstrating its superiority in accuracy and
versatility. Notably, with the FP6 quantization, \codestar-15B model performs
comparably to its FP16 counterpart in code generation, and for smaller models
like the 406M it closely matches their baselines in summarization. Neither can
be achieved by INT4. To better accommodate various AI hardware and achieve the
best system performance, we propose a novel 4+2 design for FP6 to achieve
similar latency to the state-of-the-art INT4 fine-grain quantization. With our
design, FP6 can become a promising solution to the current 4-bit quantization
methods used in LLMs.
657bc7f3696ec3dda92fc12e
Xiaoxia Wu
admin_assigned
2023-12-15T13:00:05.048Z
619721da1726b360d54d9ff7
/avatars/f5f8bf0f372bef58848530dd7d7f58de.svg
wu
user
xiaoxia-microsoft
657bc7f3696ec3dda92fc12f
Haojun Xia
657bc7f3696ec3dda92fc130
Stephen Youn
657bc7f3696ec3dda92fc131
Zhen Zheng
claimed_verified
2024-01-26T14:12:07.834Z
65373b2c89dd48faca859d02
https://cdn-avatars.huggingface.co/v1/production/uploads/65373b2c89dd48faca859d02/42HukqnvMykvaoTxnQJjk.jpeg
Zhen Zheng
user
JamesTheZ
657bc7f3696ec3dda92fc132
Shiyang Chen
657bc7f3696ec3dda92fc133
Arash Bakhtiari
claimed_verified
2023-12-15T19:10:40.497Z
6553cda0c81411e2aaf930b1
/avatars/291e4e9cef092d649ab734a3f09a3af1.svg
Arash Bakhtiari
user
arashb
657bc7f3696ec3dda92fc134
Michael Wyatt
admin_assigned
2023-12-15T13:01:35.693Z
62910c52d1be2630c0d38a72
/avatars/40a72a10e3ea17809a09bde77360c194.svg
Michael Wyatt
user
mwyatt
657bc7f3696ec3dda92fc135
Yuxiong He
657bc7f3696ec3dda92fc136
Olatunji Ruwase
admin_assigned
2023-12-15T13:01:52.108Z
6499d685ea2cdac80992e742
/avatars/c32741e7fc57c9a08722fab3877a7b81.svg
Olatunji Ruwase
user
tjruwase
657bc7f3696ec3dda92fc137
Leon Song
extracted_pending
2023-12-15T03:28:52.104Z
651442f5c75a3d4c44de4ac9
/avatars/c60185389bf6379e4360cb54615922ed.svg
Shuaiwen Leon Song
user
leonangel991
657bc7f3696ec3dda92fc138
Zhewei Yao
admin_assigned
2023-12-15T13:01:59.162Z
6352e1953199c2612ec2ff01
/avatars/f75444f1b7cba63ed085774a950a3fc7.svg
Zhewei Yao
user
zheweiyao
ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric
  Strategy for Diverse Generative Tasks
This study examines 4-bit quantization methods like GPTQ in large language
models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
extend task scope to more generative categories such as code generation and
abstractive summarization, in which we found that INT4 quantization can
significantly underperform. However, simply shifting to higher precision
formats like FP6 has been particularly challenging, thus overlooked, due to
poor performance caused by the lack of sophisticated integration and system
acceleration strategies on current AI hardware. Our results show that FP6, even
with a coarse-grain quantization scheme, performs robustly across various
algorithms and tasks, demonstrating its superiority in accuracy and
versatility. Notably, with the FP6 quantization, \codestar-15B model performs
comparably to its FP16 counterpart in code generation, and for smaller models
like the 406M it closely matches their baselines in summarization. Neither can
be achieved by INT4. To better accommodate various AI hardware and achieve the
best system performance, we propose a novel 4+2 design for FP6 to achieve
similar latency to the state-of-the-art INT4 fine-grain quantization. With our
design, FP6 can become a promising solution to the current 4-bit quantization
methods used in LLMs.
FP6 quantization offers robust performance across various tasks in large language models, surpassing INT4 quantization through a novel 4+2 design that maintains latency.
GPTQ quantization methods large language models zero-shot tasks code generation abstractive summarization INT4 FP6 quantization scheme system performance 4+2 design