Fast Matrix Multiplications for Lookup Table-Quantized LLMs
The deployment of large language models (LLMs) is often constrained by memory
bandwidth, where the primary bottleneck is the cost of transferring model
parameters from the GPU's global memory to its registers. When coupled with
custom kernels that fuse the dequantization and matmul operations, weight-only
quantization can thus enable faster inference by reducing the amount of memory
movement. However, developing high-performance kernels for weight-quantized
LLMs presents substantial challenges, especially when the weights are
compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform,
lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup
table engine for LUT-quantized LLMs, which uses offline restructuring of the
quantized weight matrix to minimize bit manipulations associated with
unpacking, and vectorization and duplication of the lookup table to mitigate
shared memory bandwidth constraints. At batch sizes < 32 and quantization group
size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster
than existing GEMM kernels. As an application of FLUTE, we explore a simple
extension to lookup table-based NormalFloat quantization and apply it to
quantize LLaMA3 to various configurations, obtaining competitive quantization
performance against strong baselines while obtaining an end-to-end throughput
increase of 1.5 to 2 times.
66996c35990749deca4f7c4f
Han Guo
admin_assigned
2024-07-22T10:38:23.048Z
617c0b780fdb1ffed4b10094
/avatars/c9e18b435c35194732ee8e45b474a7b3.svg
Han Guo
user
hanguo
66996c35990749deca4f7c50
William Brandon
66996c35990749deca4f7c51
Radostin Cholakov
claimed_verified
2024-07-19T11:23:22.096Z
629bbadee1af194c64236565
/avatars/2297243fae6456e031e9c3ff6997e8b2.svg
Radostin Cholakov
user
radi-cho
66996c35990749deca4f7c52
Jonathan Ragan-Kelley
admin_assigned
2024-07-22T10:37:29.985Z
6304fd879aef62c40140f3fd
/avatars/98550c142398aeda25cc165f8f5f83d5.svg
Jonathan Ragan-Kelley
user
jrk
66996c35990749deca4f7c53
Eric P. Xing
admin_assigned
2024-07-22T10:37:39.183Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
66996c35990749deca4f7c54
Yoon Kim
Fast Matrix Multiplications for Lookup Table-Quantized LLMs
The deployment of large language models (LLMs) is often constrained by memory
bandwidth, where the primary bottleneck is the cost of transferring model
parameters from the GPU's global memory to its registers. When coupled with
custom kernels that fuse the dequantization and matmul operations, weight-only
quantization can thus enable faster inference by reducing the amount of memory
movement. However, developing high-performance kernels for weight-quantized
LLMs presents substantial challenges, especially when the weights are
compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform,
lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup
table engine for LUT-quantized LLMs, which uses offline restructuring of the
quantized weight matrix to minimize bit manipulations associated with
unpacking, and vectorization and duplication of the lookup table to mitigate
shared memory bandwidth constraints. At batch sizes < 32 and quantization group
size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster
than existing GEMM kernels. As an application of FLUTE, we explore a simple
extension to lookup table-based NormalFloat quantization and apply it to
quantize LLaMA3 to various configurations, obtaining competitive quantization
performance against strong baselines while obtaining an end-to-end throughput
increase of 1.5 to 2 times.
FLUTE, a lookup table engine for quantized language models, accelerates inference by minimizing bit manipulations and optimizing shared memory usage, achieving faster performance compared to existing kernels.
large language models weight-only quantization lookup table quantized weight matrix vectorization GEMM kernels NormalFloat quantization LLaMA3