OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost
  Always!
Large Language Model (LLM) safety is one of the most pressing challenges for
enabling wide-scale deployment. While most studies and global discussions focus
on generic harms, such as models assisting users in harming themselves or
others, enterprises face a more fundamental concern: whether LLM-based agents
are safe for their intended use case. To address this, we introduce operational
safety, defined as an LLM's ability to appropriately accept or refuse user
queries when tasked with a specific purpose. We further propose OffTopicEval,
an evaluation suite and benchmark for measuring operational safety both in
general and within specific agentic use cases. Our evaluations on six model
families comprising 20 open-weight LLMs reveal that while performance varies
across models, all of them remain highly operationally unsafe. Even the
strongest models -- Qwen-3 (235B) with 77.77\% and Mistral (24B) with 79.96\%
-- fall far short of reliable operational safety, while GPT models plateau in
the 62--73\% range, Phi achieves only mid-level scores (48--70\%), and Gemma
and Llama-3 collapse to 39.53\% and 23.84\%, respectively. While operational
safety is a core model alignment issue, to suppress these failures, we propose
prompt-based steering methods: query grounding (Q-ground) and system-prompt
grounding (P-ground), which substantially improve OOD refusal. Q-ground
provides consistent gains of up to 23\%, while P-ground delivers even larger
boosts, raising Llama-3.3 (70B) by 41\% and Qwen-3 (30B) by 27\%. These results
highlight both the urgent need for operational safety interventions and the
promise of prompt-based steering as a first step toward more reliable LLM-based
agents.
68dc986b4159d1f2418f9b7a
Jingdi Lei
claimed_verified
2025-10-01T10:21:37.129Z
65756c5e1488186315c6696d
/avatars/2ec67e0d1921b6635c69669b591a3110.svg
Jingdi Lei
user
huaXiaKyrie
68dc986b4159d1f2418f9b7b
Varun Gumma
68dc986b4159d1f2418f9b7c
Rishabh Bhardwaj
68dc986b4159d1f2418f9b7d
Seok Min Lim
68dc986b4159d1f2418f9b7e
Chuan Li
68dc986b4159d1f2418f9b7f
Amir Zadeh
68dc986b4159d1f2418f9b80
Soujanya Poria
OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost
  Always!
Large Language Model (LLM) safety is one of the most pressing challenges for
enabling wide-scale deployment. While most studies and global discussions focus
on generic harms, such as models assisting users in harming themselves or
others, enterprises face a more fundamental concern: whether LLM-based agents
are safe for their intended use case. To address this, we introduce operational
safety, defined as an LLM's ability to appropriately accept or refuse user
queries when tasked with a specific purpose. We further propose OffTopicEval,
an evaluation suite and benchmark for measuring operational safety both in
general and within specific agentic use cases. Our evaluations on six model
families comprising 20 open-weight LLMs reveal that while performance varies
across models, all of them remain highly operationally unsafe. Even the
strongest models -- Qwen-3 (235B) with 77.77\% and Mistral (24B) with 79.96\%
-- fall far short of reliable operational safety, while GPT models plateau in
the 62--73\% range, Phi achieves only mid-level scores (48--70\%), and Gemma
and Llama-3 collapse to 39.53\% and 23.84\%, respectively. While operational
safety is a core model alignment issue, to suppress these failures, we propose
prompt-based steering methods: query grounding (Q-ground) and system-prompt
grounding (P-ground), which substantially improve OOD refusal. Q-ground
provides consistent gains of up to 23\%, while P-ground delivers even larger
boosts, raising Llama-3.3 (70B) by 41\% and Qwen-3 (30B) by 27\%. These results
highlight both the urgent need for operational safety interventions and the
promise of prompt-based steering as a first step toward more reliable LLM-based
agents.
Operational safety, measured by OffTopicEval, is a critical issue for LLMs, with most models falling short, but prompt-based steering methods show promise in improving out-of-distribution refusal.
Large Language Model (LLM) operational safety OffTopicEval query grounding (Q-ground) system-prompt grounding (P-ground) out-of-distribution (OOD) refusal