Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
The growing demand for Large Language Models (LLMs) in applications such as
content generation, intelligent chatbots, and sentiment analysis poses
considerable challenges for LLM service providers. To efficiently use GPU
resources and boost throughput, batching multiple requests has emerged as a
popular paradigm; to further speed up batching, LLM quantization techniques
reduce memory consumption and increase computing capacity. However, prevalent
quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully
leverage the capabilities of modern GPUs, such as 4-bit integer operators,
resulting in sub-optimal performance.
  To maximize LLMs' serving throughput, we introduce Atom, a low-bit
quantization method that achieves high throughput improvements with negligible
accuracy loss. Atom significantly boosts serving throughput by using low-bit
operators and considerably reduces memory consumption via low-bit quantization.
It attains high accuracy by applying a novel mixed-precision and fine-grained
quantization process. We evaluate Atom on 4-bit weight-activation quantization
setups in the serving context. Atom improves end-to-end throughput by up to
7.73times compared to the FP16 and by 2.53times compared to INT8
quantization, while maintaining the same latency target.
65408d341c4b203c012efcde
Yilong Zhao
claimed_verified
2023-11-07T11:20:52.551Z
6549b0a808775ce78e535c6a
/avatars/942066356843d0c424375937f157c975.svg
Yilong Zhao
user
ylzhao
65408d341c4b203c012efcdf
Chien-Yu Lin
admin_assigned
2023-10-31T10:33:33.415Z
64d1950a0fbfb00b919e9481
/avatars/a4a487431b9ae4a8ad916a8b553ec5b2.svg
Chien-Yu Lin
user
cylinbao
65408d341c4b203c012efce0
Kan Zhu
65408d341c4b203c012efce1
Zihao Ye
admin_assigned
2023-10-31T10:33:49.784Z
632b9f58ffe3618eff2ffb93
/avatars/1ffaf2da94fb9f993aa0af4bc3808197.svg
Zihao Ye
user
zhye
65408d341c4b203c012efce2
Lequn Chen
admin_assigned
2023-10-31T10:33:56.655Z
63f6a8f9e94ed998c460302a
/avatars/c2b78e2725a5c8c1cd3ab2e502afff71.svg
Lequn Chen
user
abcdabcd987
65408d341c4b203c012efce3
Size Zheng
admin_assigned
2023-10-31T10:34:03.611Z
636c5cb1d963c567935dd7a7
/avatars/bfe016cd8f644039f21ed36163d36cae.svg
Size Zheng
user
ZCHNO
65408d341c4b203c012efce4
Luis Ceze
65408d341c4b203c012efce5
Arvind Krishnamurthy
65408d341c4b203c012efce6
Tianqi Chen
claimed_verified
2023-10-31T15:04:46.324Z
6407c3cb2e309e654528860f
/avatars/b2177ae76ff8001564d0606752538884.svg
Tianqi Chen
user
tqchen
65408d341c4b203c012efce7
Baris Kasikci
claimed_verified
2023-10-31T17:02:05.536Z
654132fe5a9a913c6c870e79
/avatars/2f6807eddef1929c571977e9af35f952.svg
Baris Kasikci
user
kasikci
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
The growing demand for Large Language Models (LLMs) in applications such as
content generation, intelligent chatbots, and sentiment analysis poses
considerable challenges for LLM service providers. To efficiently use GPU
resources and boost throughput, batching multiple requests has emerged as a
popular paradigm; to further speed up batching, LLM quantization techniques
reduce memory consumption and increase computing capacity. However, prevalent
quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully
leverage the capabilities of modern GPUs, such as 4-bit integer operators,
resulting in sub-optimal performance.
  To maximize LLMs' serving throughput, we introduce Atom, a low-bit
quantization method that achieves high throughput improvements with negligible
accuracy loss. Atom significantly boosts serving throughput by using low-bit
operators and considerably reduces memory consumption via low-bit quantization.
It attains high accuracy by applying a novel mixed-precision and fine-grained
quantization process. We evaluate Atom on 4-bit weight-activation quantization
setups in the serving context. Atom improves end-to-end throughput by up to
7.73times compared to the FP16 and by 2.53times compared to INT8
quantization, while maintaining the same latency target.
Atom is a low-bit quantization method that enhances LLM serving throughput with minimal accuracy loss by utilizing low-bit operators and a mixed-precision process.
Large Language Models LLMs GPU batching quantization memory consumption 4-bit integer operators serving throughput low-bit operators mixed-precision fine-grained quantization FP16 INT8 quantization latency target