EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and
  Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
Large language models (LLMs) frequently refuse to respond to pseudo-malicious
instructions: semantically harmless input queries triggering unnecessary LLM
refusals due to conservative safety alignment, significantly impairing user
experience. Collecting such instructions is crucial for evaluating and
mitigating over-refusals, but existing instruction curation methods, like
manual creation or instruction rewriting, either lack scalability or fail to
produce sufficiently diverse and effective refusal-inducing prompts. To address
these limitations, we introduce EVOREFUSE, a prompt optimization approach that
generates diverse pseudo-malicious instructions consistently eliciting
confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm
exploring the instruction space in more diverse directions than existing
methods via mutation strategies and recombination, and iteratively evolves seed
instructions to maximize evidence lower bound on LLM refusal probability. Using
EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582
pseudo-malicious instructions that outperforms the next-best benchmark with
140.41% higher average refusal triggering rate across 9 LLMs, 34.86% greater
lexical diversity, and 40.03% improved LLM response confidence scores; and
EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with
responses for supervised and preference-based alignment training.
LLAMA3.1-8B-INSTRUCT supervisedly fine-tuned on EVOREFUSE-ALIGN achieves up to
14.31% fewer over-refusals than models trained on the second-best alignment
dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals
models trigger over-refusals by overly focusing on sensitive keywords while
ignoring broader context.
6847c9693ec10bdd8ab4df91
Xiaorui Wu
6847c9693ec10bdd8ab4df92
Xiaofeng Mao
6847c9693ec10bdd8ab4df93
Xin Zhang
6847c9693ec10bdd8ab4df94
Fei Li
6847c9693ec10bdd8ab4df95
Chong Teng
6847c9693ec10bdd8ab4df96
Yuxiang Peng
6847c9693ec10bdd8ab4df97
Li Zheng
6847c9693ec10bdd8ab4df98
Donghong Ji
6847c9693ec10bdd8ab4df99
Zhuang Li
EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and
  Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
Large language models (LLMs) frequently refuse to respond to pseudo-malicious
instructions: semantically harmless input queries triggering unnecessary LLM
refusals due to conservative safety alignment, significantly impairing user
experience. Collecting such instructions is crucial for evaluating and
mitigating over-refusals, but existing instruction curation methods, like
manual creation or instruction rewriting, either lack scalability or fail to
produce sufficiently diverse and effective refusal-inducing prompts. To address
these limitations, we introduce EVOREFUSE, a prompt optimization approach that
generates diverse pseudo-malicious instructions consistently eliciting
confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm
exploring the instruction space in more diverse directions than existing
methods via mutation strategies and recombination, and iteratively evolves seed
instructions to maximize evidence lower bound on LLM refusal probability. Using
EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582
pseudo-malicious instructions that outperforms the next-best benchmark with
140.41% higher average refusal triggering rate across 9 LLMs, 34.86% greater
lexical diversity, and 40.03% improved LLM response confidence scores; and
EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with
responses for supervised and preference-based alignment training.
LLAMA3.1-8B-INSTRUCT supervisedly fine-tuned on EVOREFUSE-ALIGN achieves up to
14.31% fewer over-refusals than models trained on the second-best alignment
dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals
models trigger over-refusals by overly focusing on sensitive keywords while
ignoring broader context.
EVOREFUSE, an evolutionary algorithm, generates diverse pseudo-malicious instructions to optimize LLM refusal training, improving user experience without compromising safety.
large language models pseudo-malicious instructions safety alignment instruction optimization evolutionary algorithm mutation strategies recombination evidence lower bound refusal probability lexical diversity LLM response confidence scores over-refusals supervised fine-tuning preference-based alignment training