Efficient LLM inference solution on Intel GPU
Transformer based Large Language Models (LLMs) have been widely used in many
fields, and the efficiency of LLM inference becomes hot topic in real
applications. However, LLMs are usually complicatedly designed in model
structure with massive operations and perform inference in the auto-regressive
mode, making it a challenging task to design a system with high efficiency.
  In this paper, we propose an efficient LLM inference solution with low
latency and high throughput. Firstly, we simplify the LLM decoder layer by
fusing data movement and element-wise operations to reduce the memory access
frequency and lower system latency. We also propose a segment KV cache policy
to keep key/value of the request and response tokens in separate physical
memory for effective device memory management, helping enlarge the runtime
batch size and improve system throughput. A customized
Scaled-Dot-Product-Attention kernel is designed to match our fusion policy
based on the segment KV cache solution. We implement our LLM inference solution
on Intel GPU and publish it publicly. Compared with the standard HuggingFace
implementation, the proposed solution achieves up to 7x lower token latency and
27x higher throughput for some popular LLMs on Intel GPU.
65a09e24e0b49a84d21cca89
Hui Wu
claimed_verified
2024-01-12T10:46:08.306Z
65a1185de96941538121aa81
/avatars/27c5947d20cc91bf5ef61b48d2c50a9c.svg
Hui Wu
user
wuhuikxCSU
65a09e24e0b49a84d21cca8a
Yi Gan
65a09e24e0b49a84d21cca8b
Feng Yuan
claimed_verified
2024-01-15T07:42:04.025Z
65a4823541b6ef119c5a13a4
/avatars/d57e51c4fa0bd3da1d1a96e1a18050f2.svg
Feng Yuan
user
arthuryuan1987
65a09e24e0b49a84d21cca8c
Jing Ma
65a09e24e0b49a84d21cca8d
Wei Zhu
65a09e24e0b49a84d21cca8e
Yutao Xu
admin_assigned
2024-01-12T10:39:14.660Z
642b94595bf2355d02a3c2d7
/avatars/23db8b19705a5e5f0b2bc2e2f4b0d8d0.svg
xuyutao
user
vqaFW
65a09e24e0b49a84d21cca8f
Hong Zhu
65a09e24e0b49a84d21cca90
Yuhua Zhu
admin_assigned
2024-01-12T10:39:29.465Z
6481332d8c6a3b8f11fc40c7
/avatars/16aee3d0e7ca35cf0d036aa71e7738e2.svg
zhuyuhua
user
TakaRika
65a09e24e0b49a84d21cca91
Xiaoli Liu
65a09e24e0b49a84d21cca92
Jinghui Gu
Efficient LLM inference solution on Intel GPU
Transformer based Large Language Models (LLMs) have been widely used in many
fields, and the efficiency of LLM inference becomes hot topic in real
applications. However, LLMs are usually complicatedly designed in model
structure with massive operations and perform inference in the auto-regressive
mode, making it a challenging task to design a system with high efficiency.
  In this paper, we propose an efficient LLM inference solution with low
latency and high throughput. Firstly, we simplify the LLM decoder layer by
fusing data movement and element-wise operations to reduce the memory access
frequency and lower system latency. We also propose a segment KV cache policy
to keep key/value of the request and response tokens in separate physical
memory for effective device memory management, helping enlarge the runtime
batch size and improve system throughput. A customized
Scaled-Dot-Product-Attention kernel is designed to match our fusion policy
based on the segment KV cache solution. We implement our LLM inference solution
on Intel GPU and publish it publicly. Compared with the standard HuggingFace
implementation, the proposed solution achieves up to 7x lower token latency and
27x higher throughput for some popular LLMs on Intel GPU.
The proposed LLM inference solution reduces latency and increases throughput by simplifying decoder layers, using a segment KV cache policy, and optimizing the Scaled-Dot-Product-Attention kernel.
Transformer Large Language Models LLM inference auto-regressive memory access segment KV cache Scaled-Dot-Product-Attention Intel GPU