Position: Privacy Is Not Just Memorization!
The discourse on privacy risks in Large Language Models (LLMs) has
disproportionately focused on verbatim memorization of training data, while a
constellation of more immediate and scalable privacy threats remain
underexplored. This position paper argues that the privacy landscape of LLM
systems extends far beyond training data extraction, encompassing risks from
data collection practices, inference-time context leakage, autonomous agent
capabilities, and the democratization of surveillance through deep inference
attacks. We present a comprehensive taxonomy of privacy risks across the LLM
lifecycle -- from data collection through deployment -- and demonstrate through
case studies how current privacy frameworks fail to address these multifaceted
threats. Through a longitudinal analysis of 1,322 AI/ML privacy papers
published at leading conferences over the past decade (2016--2025), we reveal
that while memorization receives outsized attention in technical research, the
most pressing privacy harms lie elsewhere, where current technical approaches
offer little traction and viable paths forward remain unclear. We call for a
fundamental shift in how the research community approaches LLM privacy, moving
beyond the narrow focus of current technical solutions and embracing
interdisciplinary approaches that address the sociotechnical nature of these
emerging threats.
68e56134975ac4c405ef1f3c
Niloofar Mireshghallah
68e56134975ac4c405ef1f3d
Tianshi Li
Position: Privacy Is Not Just Memorization!
The discourse on privacy risks in Large Language Models (LLMs) has
disproportionately focused on verbatim memorization of training data, while a
constellation of more immediate and scalable privacy threats remain
underexplored. This position paper argues that the privacy landscape of LLM
systems extends far beyond training data extraction, encompassing risks from
data collection practices, inference-time context leakage, autonomous agent
capabilities, and the democratization of surveillance through deep inference
attacks. We present a comprehensive taxonomy of privacy risks across the LLM
lifecycle -- from data collection through deployment -- and demonstrate through
case studies how current privacy frameworks fail to address these multifaceted
threats. Through a longitudinal analysis of 1,322 AI/ML privacy papers
published at leading conferences over the past decade (2016--2025), we reveal
that while memorization receives outsized attention in technical research, the
most pressing privacy harms lie elsewhere, where current technical approaches
offer little traction and viable paths forward remain unclear. We call for a
fundamental shift in how the research community approaches LLM privacy, moving
beyond the narrow focus of current technical solutions and embracing
interdisciplinary approaches that address the sociotechnical nature of these
emerging threats.
The paper discusses underexplored privacy risks in Large Language Models (LLMs) beyond verbatim memorization, including data collection, inference-time context leakage, autonomous agent capabilities, and surveillance through deep inference attacks, and calls for a broader interdisciplinary approach to address these threats.
Large Language Models LLMs privacy risks data collection inference-time context leakage autonomous agent capabilities surveillance deep inference attacks privacy frameworks sociotechnical nature