"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
  Quantization
Despite the popularity of large language model (LLM) quantization for
inference acceleration, significant uncertainty remains regarding the
accuracy-performance trade-offs associated with various quantization formats.
We present a comprehensive empirical study of quantized accuracy, evaluating
popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
examines the difference in text generated by quantized models versus their
uncompressed counterparts. Beyond benchmarks, we also present a couple of
quantization improvements which allowed us to obtain state-of-the-art accuracy
recovery results. Our investigation, encompassing over 500,000 individual
evaluations, yields several key findings: (1) FP8 weight and activation
quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
(W4A16-INT) is competitive with 8-bit integer weight and activation
quantization. To address the question of the "best" format for a given
deployment environment, we conduct inference performance analysis using the
popular open-source vLLM framework on various GPU architectures. We find that
W4A16 offers the best cost-efficiency for synchronous deployments, and for
asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
in asynchronous "continuous batching" deployment of mid- and large-size models
on high-end GPUs. Our results provide a set of practical guidelines for
deploying quantized LLMs across scales and performance requirements.
672a1fde369f174c868341ec
Eldar Kurtic
claimed_verified
2024-11-05T13:41:42.690Z
628e0ce4e53bbd334577fcb0
https://cdn-avatars.huggingface.co/v1/production/uploads/628e0ce4e53bbd334577fcb0/z9V6jwIaHuPirQDKHsMx2.jpeg
Eldar Kurtić
user
ekurtic
672a1fde369f174c868341ed
Alexandre Marques
admin_assigned
2024-11-05T14:46:19.705Z
64ee5432f9ed75901c09a590
/avatars/992163fe824381af39cedec073136967.svg
Alexandre Marques
user
alexmarques
672a1fde369f174c868341ee
Shubhra Pandit
admin_assigned
2024-11-05T14:45:54.783Z
632c828a54e2c512c8f55144
/avatars/a30e4b33b21750833a38ab44a874a4db.svg
Shubhra Pandit
user
shubhrapandit
672a1fde369f174c868341ef
Mark Kurtz
claimed_verified
2024-11-05T20:17:30.459Z
628e965178e24401298387ee
https://cdn-avatars.huggingface.co/v1/production/uploads/628e965178e24401298387ee/IzynaZqiVA6X27MKFiKmF.jpeg
Mark Kurtz
user
markurtz
672a1fde369f174c868341f0
Dan Alistarh
admin_assigned
2024-11-05T14:45:08.138Z
64d100c5d8d0927372e3d4c0
/avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
Dan Alistarh
user
dalistarh
"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
  Quantization
Despite the popularity of large language model (LLM) quantization for
inference acceleration, significant uncertainty remains regarding the
accuracy-performance trade-offs associated with various quantization formats.
We present a comprehensive empirical study of quantized accuracy, evaluating
popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
examines the difference in text generated by quantized models versus their
uncompressed counterparts. Beyond benchmarks, we also present a couple of
quantization improvements which allowed us to obtain state-of-the-art accuracy
recovery results. Our investigation, encompassing over 500,000 individual
evaluations, yields several key findings: (1) FP8 weight and activation
quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
(W4A16-INT) is competitive with 8-bit integer weight and activation
quantization. To address the question of the "best" format for a given
deployment environment, we conduct inference performance analysis using the
popular open-source vLLM framework on various GPU architectures. We find that
W4A16 offers the best cost-efficiency for synchronous deployments, and for
asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
in asynchronous "continuous batching" deployment of mid- and large-size models
on high-end GPUs. Our results provide a set of practical guidelines for
deploying quantized LLMs across scales and performance requirements.
A comprehensive study evaluates quantization formats for LLMs, showing that FP8 W8A8-FP is lossless, W8A8-INT incurs minimal accuracy loss, and W4A16-INT is cost-efficient, providing guidelines for optimal deployment.
LLM quantization FP8 INT8 INT4 W8A8-FP W8A8-INT W4A16-INT vLLM framework GPU architectures synchronous deployment asynchronous deployment continuous batching accuracy-recovery results