MemServe: Context Caching for Disaggregated LLM Serving with Elastic
  Memory Pool
Large language model (LLM) serving has transformed from stateless to stateful
systems, utilizing techniques like context caching and disaggregated inference.
These optimizations extend the lifespan and domain of the KV cache,
necessitating a new architectural approach. We present MemServe, a unified
system that integrates both inter-request and intra-request optimizations.
MemServe introduces MemPool, an elastic memory pool managing distributed memory
and KV caches across serving instances. Using MemPool APIs, MemServe combines
context caching with disaggregated inference for the first time, supported by a
global scheduler that enhances cache reuse through a global prompt tree-based
locality-aware policy. Tests show that MemServe significantly improves job
completion time and time-to-first-time.
667cab9120d68cc58172b03d
Cunchen Hu
667cab9120d68cc58172b03e
Heyang Huang
admin_assigned
2024-06-27T18:55:21.029Z
647fbf98095af0bf116d4de6
/avatars/c449c9a13898590da47faa378b41e20c.svg
Heyang Huang
user
heyangh
667cab9120d68cc58172b03f
Junhao Hu
admin_assigned
2024-06-27T18:55:28.586Z
655336cb0166ff6bd23a979f
/avatars/6b0d8a8064dbcf4db895ca6735dfa0a2.svg
Hu
user
Junhaohu
667cab9120d68cc58172b040
Jiang Xu
667cab9120d68cc58172b041
Xusheng Chen
667cab9120d68cc58172b042
Tao Xie
667cab9120d68cc58172b043
Chenxi Wang
667cab9120d68cc58172b044
Sa Wang
667cab9120d68cc58172b045
Yungang Bao
667cab9120d68cc58172b046
Ninghui Sun
667cab9120d68cc58172b047
Yizhou Shan
claimed_verified
2024-06-27T09:02:32.242Z
6457885a75f8f7d26aa5bc44
/avatars/8ce57c4d60a1f1b5afa2c592207a8335.svg
allthingsdisaggregated
user
lastweek
MemServe: Context Caching for Disaggregated LLM Serving with Elastic
  Memory Pool
Large language model (LLM) serving has transformed from stateless to stateful
systems, utilizing techniques like context caching and disaggregated inference.
These optimizations extend the lifespan and domain of the KV cache,
necessitating a new architectural approach. We present MemServe, a unified
system that integrates both inter-request and intra-request optimizations.
MemServe introduces MemPool, an elastic memory pool managing distributed memory
and KV caches across serving instances. Using MemPool APIs, MemServe combines
context caching with disaggregated inference for the first time, supported by a
global scheduler that enhances cache reuse through a global prompt tree-based
locality-aware policy. Tests show that MemServe significantly improves job
completion time and time-to-first-time.
MemServe integrates inter-request and intra-request optimizations in LLM serving, using MemPool for distributed memory management and a global scheduler for enhanced cache reuse, resulting in improved job completion time and time-to-first-time.
large language model (LLM) serving stateless to stateful systems context caching disaggregated inference KV cache MemPool elastic memory pool distributed memory global scheduler global prompt tree-based locality-aware policy job completion time time-to-first-time