A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI
  Responses
Large Language Models (LLMs) are vulnerable to
jailbreaksx2013methods to elicit harmful or generally impermissible
outputs. Safety measures are developed and assessed on their effectiveness at
defending against jailbreak attacks, indicating a belief that safety is
equivalent to robustness. We assert that current defense mechanisms, such as
output filters and alignment fine-tuning, are, and will remain, fundamentally
insufficient for ensuring model safety. These defenses fail to address risks
arising from dual-intent queries and the ability to composite innocuous outputs
to achieve harmful goals. To address this critical gap, we introduce an
information-theoretic threat model called inferential adversaries who exploit
impermissible information leakage from model outputs to achieve malicious
goals. We distinguish these from commonly studied security adversaries who only
seek to force victim models to generate specific impermissible outputs. We
demonstrate the feasibility of automating inferential adversaries through
question decomposition and response aggregation. To provide safety guarantees,
we define an information censorship criterion for censorship mechanisms,
bounding the leakage of impermissible information. We propose a defense
mechanism which ensures this bound and reveal an intrinsic safety-utility
trade-off. Our work provides the first theoretically grounded understanding of
the requirements for releasing safe LLMs and the utility costs involved.
6686b9c3434e14e5a1ee2182
David Glukhov
admin_assigned
2024-07-10T12:24:05.040Z
653be9f07773b2d575ba3f7c
/avatars/2f605f3bcd8680bbce88ebd316d838ec.svg
David Glukhov
user
WhereIsExit
6686b9c3434e14e5a1ee2183
Ziwen Han
admin_assigned
2024-07-10T12:24:12.680Z
62ed40fe9b538d02b451c211
/avatars/d5486b050fd63ade348158e64a3e5ff7.svg
Han
user
Ziwen
6686b9c3434e14e5a1ee2184
Ilia Shumailov
6686b9c3434e14e5a1ee2185
Vardan Papyan
admin_assigned
2024-07-10T12:24:26.211Z
6581f96f0fb25b9aa270856e
/avatars/5502416aed6352ecf794252735656a4b.svg
Vardan Papyan
user
vardanpapyan
6686b9c3434e14e5a1ee2186
Nicolas Papernot
A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI
  Responses
Large Language Models (LLMs) are vulnerable to
jailbreaksx2013methods to elicit harmful or generally impermissible
outputs. Safety measures are developed and assessed on their effectiveness at
defending against jailbreak attacks, indicating a belief that safety is
equivalent to robustness. We assert that current defense mechanisms, such as
output filters and alignment fine-tuning, are, and will remain, fundamentally
insufficient for ensuring model safety. These defenses fail to address risks
arising from dual-intent queries and the ability to composite innocuous outputs
to achieve harmful goals. To address this critical gap, we introduce an
information-theoretic threat model called inferential adversaries who exploit
impermissible information leakage from model outputs to achieve malicious
goals. We distinguish these from commonly studied security adversaries who only
seek to force victim models to generate specific impermissible outputs. We
demonstrate the feasibility of automating inferential adversaries through
question decomposition and response aggregation. To provide safety guarantees,
we define an information censorship criterion for censorship mechanisms,
bounding the leakage of impermissible information. We propose a defense
mechanism which ensures this bound and reveal an intrinsic safety-utility
trade-off. Our work provides the first theoretically grounded understanding of
the requirements for releasing safe LLMs and the utility costs involved.
LLMs are vulnerable to jailbreaks through inferential adversaries that exploit information leakage, and current defenses are insufficient; a new threat model and defense mechanism are proposed to address this.
LLMs jailbreaks dual-intent queries inferential adversaries question decomposition response aggregation information censorship criterion safety-utility trade-off