Prompt Cache: Modular Attention Reuse for Low-Latency Inference
We present Prompt Cache, an approach for accelerating inference for large
language models (LLM) by reusing attention states across different LLM prompts.
Many input prompts have overlapping text segments, such as system messages,
prompt templates, and documents provided for context. Our key insight is that
by precomputing and storing the attention states of these frequently occurring
text segments on the inference server, we can efficiently reuse them when these
segments appear in user prompts. Prompt Cache employs a schema to explicitly
define such reusable text segments, called prompt modules. The schema ensures
positional accuracy during attention state reuse and provides users with an
interface to access cached states in their prompt. Using a prototype
implementation, we evaluate Prompt Cache across several LLMs. We show that
Prompt Cache significantly reduce latency in time-to-first-token, especially
for longer prompts such as document-based question answering and
recommendations. The improvements range from 8x for GPU-based inference to 60x
for CPU-based inference, all while maintaining output accuracy and without the
need for model parameter modifications.
654dbb1bca2f3d20259209be
In Gim
admin_assigned
2023-11-10T10:39:55.008Z
63d31d672727d7888cbc2428
/avatars/1df66789dd7ed8f3c068f9d440860ed9.svg
In Gim
user
ingim
654dbb1bca2f3d20259209bf
Guojun Chen
claimed_verified
2023-11-10T18:39:41.660Z
6500693c051fae19fc4c2c1b
/avatars/10fe24e1eabde8016b7c95a53ac153b7.svg
Guojun Chen
user
Leonana69
654dbb1bca2f3d20259209c0
Seung-seob Lee
admin_assigned
2023-11-10T10:40:25.582Z
64bec1fd9f94ea2554258be0
/avatars/13d8a12f937ab5a95c2c1bfdb136aee9.svg
Seung-seob Lee
user
seungseob7lee
654dbb1bca2f3d20259209c1
Nikhil Sarda
admin_assigned
2023-11-10T10:40:32.989Z
64f9a619a5067f6b65542cbf
/avatars/d0ebeeb7123a8d5f432970e2efb6cd6a.svg
Nikhil Sarda
user
diffoperator
654dbb1bca2f3d20259209c2
Anurag Khandelwal
admin_assigned
2023-11-10T10:40:39.747Z
644d68cf5c1b4e14d0c20be9
/avatars/0dd280cbb75951834a013521734124ec.svg
Anurag Khandelwal 
user
Anurag88
654dbb1bca2f3d20259209c3
Lin Zhong
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
We present Prompt Cache, an approach for accelerating inference for large
language models (LLM) by reusing attention states across different LLM prompts.
Many input prompts have overlapping text segments, such as system messages,
prompt templates, and documents provided for context. Our key insight is that
by precomputing and storing the attention states of these frequently occurring
text segments on the inference server, we can efficiently reuse them when these
segments appear in user prompts. Prompt Cache employs a schema to explicitly
define such reusable text segments, called prompt modules. The schema ensures
positional accuracy during attention state reuse and provides users with an
interface to access cached states in their prompt. Using a prototype
implementation, we evaluate Prompt Cache across several LLMs. We show that
Prompt Cache significantly reduce latency in time-to-first-token, especially
for longer prompts such as document-based question answering and
recommendations. The improvements range from 8x for GPU-based inference to 60x
for CPU-based inference, all while maintaining output accuracy and without the
need for model parameter modifications.
Prompt Cache improves inference speed for large language models by reusing precomputed attention states of common text segments in user prompts.
attention states large language models (LLM) prompt modules positional accuracy time-to-first-token document-based question answering recommendations