Adaptive Semantic Prompt Caching with VectorQ
Semantic prompt caches reduce the latency and cost of large language model
(LLM) inference by reusing cached LLM-generated responses for semantically
similar prompts. Vector similarity metrics assign a numerical score to quantify
the similarity between an embedded prompt and its nearest neighbor in the
cache. Existing systems rely on a static threshold to classify whether the
similarity score is sufficiently high to result in a cache hit. We show that
this one-size-fits-all threshold is insufficient across different prompts. We
propose VectorQ, a framework to learn embedding-specific threshold regions that
adapt to the complexity and uncertainty of an embedding. Through evaluations on
a combination of four diverse datasets, we show that VectorQ consistently
outperforms state-of-the-art systems across all static thresholds, achieving up
to 12x increases in cache hit rate and error rate reductions up to 92%.
67aaa0ebe37429ebdbd113cf
Luis Gaspar Schroeder
67aaa0ebe37429ebdbd113d0
Shu Liu
67aaa0ebe37429ebdbd113d1
Alejandro Cuadron
claimed_verified
2025-02-11T07:55:36.306Z
652a656d1a3250bbfe3bb92d
/avatars/a1c25150d55c493edd9a7f81287fc449.svg
Alejandro Cuadron
user
AlexCuadron
67aaa0ebe37429ebdbd113d2
Mark Zhao
67aaa0ebe37429ebdbd113d3
Stephan Krusche
67aaa0ebe37429ebdbd113d4
Alfons Kemper
67aaa0ebe37429ebdbd113d5
Matei Zaharia
67aaa0ebe37429ebdbd113d6
Joseph E. Gonzalez
Adaptive Semantic Prompt Caching with VectorQ
Semantic prompt caches reduce the latency and cost of large language model
(LLM) inference by reusing cached LLM-generated responses for semantically
similar prompts. Vector similarity metrics assign a numerical score to quantify
the similarity between an embedded prompt and its nearest neighbor in the
cache. Existing systems rely on a static threshold to classify whether the
similarity score is sufficiently high to result in a cache hit. We show that
this one-size-fits-all threshold is insufficient across different prompts. We
propose VectorQ, a framework to learn embedding-specific threshold regions that
adapt to the complexity and uncertainty of an embedding. Through evaluations on
a combination of four diverse datasets, we show that VectorQ consistently
outperforms state-of-the-art systems across all static thresholds, achieving up
to 12x increases in cache hit rate and error rate reductions up to 92%.
VectorQ learns adaptive threshold regions for semantic prompt caching, significantly improving cache hit rates and error reductions in large language model inference.
semantic prompt caches latency cost large language models LLM inference vector similarity metrics cache hit embedding-specific threshold regions complexity uncertainty embedding cache hit rate error rate