TP-Aware Dequantization
In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
65c453ddb25522fdb01dd2bd
Adnan Hoque
extracted_pending
2024-02-08T04:09:01.800Z
65c2782802e7932529fae7b3
/avatars/082e37e2b1bba64f7b41ef52ce32b642.svg
Adnan Hoque
user
adhoq26
65c453ddb25522fdb01dd2be
Mudhakar Srivatsa
admin_assigned
2024-02-08T12:05:16.906Z
650887ccf75ac8c6f958d434
/avatars/6dff3ded603e1975425bb481d0e63380.svg
Mudhakar Srivatsa
user
mudhakar
65c453ddb25522fdb01dd2bf
Chih-Chieh Yang
admin_assigned
2024-02-08T12:05:23.656Z
646b38a8ec9a61e8717aaec5
/avatars/c2d6d4ee1f36889170efdfcb46425014.svg
Chih-Chieh Yang
user
scatjay
65c453ddb25522fdb01dd2c0
Raghu Ganti
admin_assigned
2024-02-08T12:05:30.590Z
61d460b42e1399aebcf80a39
/avatars/7eb33d86f5373a4b3c16b72bfbd4cb0d.svg
Raghu Ganti
user
rganti
TP-Aware Dequantization
In this paper, we present a novel method that reduces model inference latency
during distributed deployment of Large Language Models (LLMs). Our contribution
is an optimized inference deployment scheme that address the current
limitations of state-of-the-art quantization kernels when used in conjunction
with Tensor Parallel (TP). Our method preserves data locality in GPU memory
access patterns and exploits a priori knowledge of TP to reduce global
communication. We demonstrate an up to 1.81x speedup over existing methods for
Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
A new method optimizes inference deployment of Large Language Models by reducing latency and communication in GPU memory using Tensor Parallel.
Large Language Models LLMS quantization kernels Tensor Parallel TP GPU memory a priori knowledge global communication inference latency Llama-70B Granite-20B MLP layer NVIDIA DGX Systems