The Instruction Hierarchy: Training LLMs to Prioritize Privileged
  Instructions
Today's LLMs are susceptible to prompt injections, jailbreaks, and other
attacks that allow adversaries to overwrite a model's original instructions
with their own malicious prompts. In this work, we argue that one of the
primary vulnerabilities underlying these attacks is that LLMs often consider
system prompts (e.g., text from an application developer) to be the same
priority as text from untrusted users and third parties. To address this, we
propose an instruction hierarchy that explicitly defines how models should
behave when instructions of different priorities conflict. We then propose a
data generation method to demonstrate this hierarchical instruction following
behavior, which teaches LLMs to selectively ignore lower-privileged
instructions. We apply this method to GPT-3.5, showing that it drastically
increases robustness -- even for attack types not seen during training -- while
imposing minimal degradations on standard capabilities.
6627260322bdf1d264974a92
Eric Wallace
admin_assigned
2024-04-23T07:36:33.681Z
63112d2431257261d20d5754
/avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
Eric Wallace
user
EricWallace
6627260322bdf1d264974a93
Kai Xiao
6627260322bdf1d264974a94
Reimar Leike
6627260322bdf1d264974a95
Lilian Weng
admin_assigned
2024-04-23T07:37:04.507Z
5fb2dc322a79d831217aa0cb
/avatars/c4bdc2f907443e34d75116730105510c.svg
Lilian Weng
user
lilianweng
6627260322bdf1d264974a96
Johannes Heidecke
6627260322bdf1d264974a97
Alex Beutel
admin_assigned
2024-04-23T07:37:37.386Z
636dc726ffbe479c979c3046
/avatars/287f422bb0c5a26d8d6c0649ac83ba93.svg
Alex Beutel
user
alexbeutel
The Instruction Hierarchy: Training LLMs to Prioritize Privileged
  Instructions
Today's LLMs are susceptible to prompt injections, jailbreaks, and other
attacks that allow adversaries to overwrite a model's original instructions
with their own malicious prompts. In this work, we argue that one of the
primary vulnerabilities underlying these attacks is that LLMs often consider
system prompts (e.g., text from an application developer) to be the same
priority as text from untrusted users and third parties. To address this, we
propose an instruction hierarchy that explicitly defines how models should
behave when instructions of different priorities conflict. We then propose a
data generation method to demonstrate this hierarchical instruction following
behavior, which teaches LLMs to selectively ignore lower-privileged
instructions. We apply this method to GPT-3.5, showing that it drastically
increases robustness -- even for attack types not seen during training -- while
imposing minimal degradations on standard capabilities.
A hierarchical instruction framework for LLMs increases robustness against malicious prompts by selectively prioritizing system instructions over user inputs.
LLMs prompt injections jailbreaks system prompts instruction hierarchy data generation method hierarchical instruction following GPT-3.5