False Sense of Security: Why Probing-based Malicious Input Detection
  Fails to Generalize
Large Language Models (LLMs) can comply with harmful instructions, raising
serious safety concerns despite their impressive capabilities. Recent work has
leveraged probing-based approaches to study the separability of malicious and
benign inputs in LLMs' internal representations, and researchers have proposed
using such probing methods for safety detection. We systematically re-examine
this paradigm. Motivated by poor out-of-distribution performance, we
hypothesize that probes learn superficial patterns rather than semantic
harmfulness. Through controlled experiments, we confirm this hypothesis and
identify the specific patterns learned: instructional patterns and trigger
words. Our investigation follows a systematic approach, progressing from
demonstrating comparable performance of simple n-gram methods, to controlled
experiments with semantically cleaned datasets, to detailed analysis of pattern
dependencies. These results reveal a false sense of security around current
probing-based approaches and highlight the need to redesign both models and
evaluation protocols, for which we provide further discussions in the hope of
suggesting responsible further research in this direction. We have open-sourced
the project at https://github.com/WangCheng0116/Why-Probe-Fails.
68ba4063736018af705e8e2e
Cheng Wang
68ba4063736018af705e8e2f
Zeming Wei
claimed_verified
2025-09-06T10:57:16.910Z
6329c57e6813868fa49efeaa
/avatars/59f46607d3bd33d98a35679e99a00b32.svg
Zeming Wei
user
ZemingWei
68ba4063736018af705e8e30
Qin Liu
68ba4063736018af705e8e31
Muhao Chen
False Sense of Security: Why Probing-based Malicious Input Detection
  Fails to Generalize
Large Language Models (LLMs) can comply with harmful instructions, raising
serious safety concerns despite their impressive capabilities. Recent work has
leveraged probing-based approaches to study the separability of malicious and
benign inputs in LLMs' internal representations, and researchers have proposed
using such probing methods for safety detection. We systematically re-examine
this paradigm. Motivated by poor out-of-distribution performance, we
hypothesize that probes learn superficial patterns rather than semantic
harmfulness. Through controlled experiments, we confirm this hypothesis and
identify the specific patterns learned: instructional patterns and trigger
words. Our investigation follows a systematic approach, progressing from
demonstrating comparable performance of simple n-gram methods, to controlled
experiments with semantically cleaned datasets, to detailed analysis of pattern
dependencies. These results reveal a false sense of security around current
probing-based approaches and highlight the need to redesign both models and
evaluation protocols, for which we provide further discussions in the hope of
suggesting responsible further research in this direction. We have open-sourced
the project at https://github.com/WangCheng0116/Why-Probe-Fails.
Probing-based approaches for detecting harmful instructions in LLMs are found to rely on superficial patterns rather than semantic understanding, indicating a need for redesigning models and evaluation methods.
Large Language Models LLMs probing-based approaches internal representations safety detection out-of-distribution performance instructional patterns trigger words n-gram methods semantically cleaned datasets pattern dependencies