TP-Aware Dequantization
In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
65c453ddb25522fdb01dd2bd
Adnan Hoque
extracted_pending
2024-02-08T04:09:01.800Z
65c2782802e7932529fae7b3
/avatars/082e37e2b1bba64f7b41ef52ce32b642.svg
Adnan Hoque
user
adhoq26
65c453ddb25522fdb01dd2be
Mudhakar Srivatsa
admin_assigned
2024-02-08T12:05:16.906Z
650887ccf75ac8c6f958d434
/avatars/6dff3ded603e1975425bb481d0e63380.svg
Mudhakar Srivatsa
user
mudhakar
65c453ddb25522fdb01dd2bf
Chih-Chieh Yang
admin_assigned
2024-02-08T12:05:23.656Z
646b38a8ec9a61e8717aaec5
/avatars/c2d6d4ee1f36889170efdfcb46425014.svg
Chih-Chieh Yang
user
scatjay
65c453ddb25522fdb01dd2c0
Raghu Ganti
admin_assigned
2024-02-08T12:05:30.590Z
61d460b42e1399aebcf80a39
/avatars/7eb33d86f5373a4b3c16b72bfbd4cb0d.svg
Raghu Ganti
user
rganti
TP-Aware Dequantization
In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
Large Language Models (LLMs) Tensor Parallel (TP) data locality global communication quantization kernels Llama-70B Granite-20B MLP layer A100 H100 NVIDIA DGX Systems