The Sum Leaks More Than Its Parts: Compositional Privacy Risks and
  Mitigations in Multi-Agent Collaboration
As large language models (LLMs) become integral to multi-agent systems, new
privacy risks emerge that extend beyond memorization, direct inference, or
single-turn evaluations. In particular, seemingly innocuous responses, when
composed across interactions, can cumulatively enable adversaries to recover
sensitive information, a phenomenon we term compositional privacy leakage. We
present the first systematic study of such compositional privacy leaks and
possible mitigation methods in multi-agent LLM systems. First, we develop a
framework that models how auxiliary knowledge and agent interactions jointly
amplify privacy risks, even when each response is benign in isolation. Next, to
mitigate this, we propose and evaluate two defense strategies: (1)
Theory-of-Mind defense (ToM), where defender agents infer a questioner's intent
by anticipating how their outputs may be exploited by adversaries, and (2)
Collaborative Consensus Defense (CoDef), where responder agents collaborate
with peers who vote based on a shared aggregated state to restrict sensitive
information spread. Crucially, we balance our evaluation across compositions
that expose sensitive information and compositions that yield benign
inferences. Our experiments quantify how these defense strategies differ in
balancing the privacy-utility trade-off. We find that while chain-of-thought
alone offers limited protection to leakage (~39% sensitive blocking rate), our
ToM defense substantially improves sensitive query blocking (up to 97%) but can
reduce benign task success. CoDef achieves the best balance, yielding the
highest Balanced Outcome (79.8%), highlighting the benefit of combining
explicit reasoning with defender collaboration. Together, our results expose a
new class of risks in collaborative LLM deployments and provide actionable
insights for designing safeguards against compositional, context-driven privacy
leakage.
68ccac1b3df9ac65e93dc5a6
Vaidehi Patil
claimed_verified
2025-09-19T06:49:15.268Z
64f64da90efa33bfe0a3d9ba
/avatars/c45fb015433e46a2eeb9518910f75d35.svg
Vaidehi Patil
user
vaidehi99
68ccac1b3df9ac65e93dc5a7
Elias Stengel-Eskin
68ccac1b3df9ac65e93dc5a8
Mohit Bansal
The Sum Leaks More Than Its Parts: Compositional Privacy Risks and
  Mitigations in Multi-Agent Collaboration
As large language models (LLMs) become integral to multi-agent systems, new
privacy risks emerge that extend beyond memorization, direct inference, or
single-turn evaluations. In particular, seemingly innocuous responses, when
composed across interactions, can cumulatively enable adversaries to recover
sensitive information, a phenomenon we term compositional privacy leakage. We
present the first systematic study of such compositional privacy leaks and
possible mitigation methods in multi-agent LLM systems. First, we develop a
framework that models how auxiliary knowledge and agent interactions jointly
amplify privacy risks, even when each response is benign in isolation. Next, to
mitigate this, we propose and evaluate two defense strategies: (1)
Theory-of-Mind defense (ToM), where defender agents infer a questioner's intent
by anticipating how their outputs may be exploited by adversaries, and (2)
Collaborative Consensus Defense (CoDef), where responder agents collaborate
with peers who vote based on a shared aggregated state to restrict sensitive
information spread. Crucially, we balance our evaluation across compositions
that expose sensitive information and compositions that yield benign
inferences. Our experiments quantify how these defense strategies differ in
balancing the privacy-utility trade-off. We find that while chain-of-thought
alone offers limited protection to leakage (~39% sensitive blocking rate), our
ToM defense substantially improves sensitive query blocking (up to 97%) but can
reduce benign task success. CoDef achieves the best balance, yielding the
highest Balanced Outcome (79.8%), highlighting the benefit of combining
explicit reasoning with defender collaboration. Together, our results expose a
new class of risks in collaborative LLM deployments and provide actionable
insights for designing safeguards against compositional, context-driven privacy
leakage.
Research identifies compositional privacy leakage in multi-agent LLM systems and evaluates defense strategies to mitigate it, finding that collaborative consensus defense offers the best balance between privacy and utility.
large language models multi-agent systems compositional privacy leakage Theory-of-Mind defense Collaborative Consensus Defense chain-of-thought privacy-utility trade-off Balanced Outcome