KVCOMM: Online Cross-context KV-cache Communication for Efficient
  LLM-based Multi-agent Systems
Multi-agent large language model (LLM) systems are increasingly adopted for
complex language processing tasks that require communication and coordination
among agents. However, these systems often suffer substantial overhead from
repeated reprocessing of overlapping contexts across agents. In typical
pipelines, once an agent receives a message from its predecessor, the full
context-including prior turns-must be reprocessed from scratch, leading to
inefficient processing. While key-value (KV) caching is an effective solution
for avoiding redundant computation in single-agent settings where prefixes
remain unchanged, it cannot be directly reused in multi-agent scenarios due to
diverging prefixes introduced by agent-specific context extensions. We identify
that the core challenge lies in the offset variance of KV-caches across agents.
To address this, we propose KVCOMM, a training-free framework that enables
efficient prefilling in multi-agent inference by reusing KV-caches and aligning
cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM
estimates and adjusts KV-caches for shared content by referencing a pool of
cached examples-termed anchors-that store observed cache deviations under
varying prefixes. The anchor pool is maintained and updated online, allowing
dynamic adaptation to distinct user requests and context structures. KVCOMM
achieves over 70% reuse rate across diverse multi-agent workloads, including
retrieval-augmented generation, math reasoning, and collaborative coding tasks,
all without quality degradation. Particularly, when each fully-connected agent
receives 1K input tokens with 512 prefix tokens and 512 output tokens under a
five-agent setting, KVCOMM achieves up to 7.8x speedup compared to the standard
prefill pipeline, reducing TTFT from ~430 ms to ~55 ms.
68f0a09c36f8b025381e1aa0
Hancheng Ye
claimed_verified
2025-10-16T10:38:24.967Z
6597f70d260bcd5fb5529613
/avatars/8fd7a210f3cb0de1229b49a3cd1ad8a0.svg
Hancheng Ye
user
HankYe
68f0a09c36f8b025381e1aa1
Zhengqi Gao
68f0a09c36f8b025381e1aa2
Mingyuan Ma
68f0a09c36f8b025381e1aa3
Qinsi Wang
68f0a09c36f8b025381e1aa4
Yuzhe Fu
68f0a09c36f8b025381e1aa5
Ming-Yu Chung
68f0a09c36f8b025381e1aa6
Yueqian Lin
68f0a09c36f8b025381e1aa7
Zhijian Liu
68f0a09c36f8b025381e1aa8
Jianyi Zhang
68f0a09c36f8b025381e1aa9
Danyang Zhuo
68f0a09c36f8b025381e1aaa
Yiran Chen
KVCOMM: Online Cross-context KV-cache Communication for Efficient
  LLM-based Multi-agent Systems
Multi-agent large language model (LLM) systems are increasingly adopted for
complex language processing tasks that require communication and coordination
among agents. However, these systems often suffer substantial overhead from
repeated reprocessing of overlapping contexts across agents. In typical
pipelines, once an agent receives a message from its predecessor, the full
context-including prior turns-must be reprocessed from scratch, leading to
inefficient processing. While key-value (KV) caching is an effective solution
for avoiding redundant computation in single-agent settings where prefixes
remain unchanged, it cannot be directly reused in multi-agent scenarios due to
diverging prefixes introduced by agent-specific context extensions. We identify
that the core challenge lies in the offset variance of KV-caches across agents.
To address this, we propose KVCOMM, a training-free framework that enables
efficient prefilling in multi-agent inference by reusing KV-caches and aligning
cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM
estimates and adjusts KV-caches for shared content by referencing a pool of
cached examples-termed anchors-that store observed cache deviations under
varying prefixes. The anchor pool is maintained and updated online, allowing
dynamic adaptation to distinct user requests and context structures. KVCOMM
achieves over 70% reuse rate across diverse multi-agent workloads, including
retrieval-augmented generation, math reasoning, and collaborative coding tasks,
all without quality degradation. Particularly, when each fully-connected agent
receives 1K input tokens with 512 prefix tokens and 512 output tokens under a
five-agent setting, KVCOMM achieves up to 7.8x speedup compared to the standard
prefill pipeline, reducing TTFT from ~430 ms to ~55 ms.
KVCOMM is a training-free framework that enhances multi-agent LLM systems by efficiently reusing KV-caches and aligning cache offsets, achieving significant speedups without quality loss.
multi-agent large language model key-value caching KVCOMM prefilling cache offsets anchor pool retrieval-augmented generation math reasoning collaborative coding tasks speedup TTFT