Agentic Reinforcement Learning for Search is Unsafe
Agentic reinforcement learning (RL) trains large language models to
autonomously call tools during reasoning, with search as the most common
application. These models excel at multi-step reasoning tasks, but their safety
properties are not well understood. In this study, we show that RL-trained
search models inherit refusal from instruction tuning and often deflect harmful
requests by turning them into safe queries. However, this safety is fragile.
Two simple attacks, one that forces the model to begin response with search
(Search attack), another that encourages models to repeatedly search
(Multi-search attack), trigger cascades of harmful searches and answers. Across
two model families (Qwen, Llama) with both local and web search, these attacks
lower refusal rates by up to 60.0%, answer safety by 82.5%, and search-query
safety by 82.4%. The attacks succeed by triggering models to generate harmful,
request-mirroring search queries before they can generate the inherited refusal
tokens. This exposes a core weakness of current RL training: it rewards
continued generation of effective queries without accounting for their
harmfulness. As a result, RL search models have vulnerabilities that users can
easily exploit, making it urgent to develop safety-aware agentic RL pipelines
optimising for safe search.
68f754d524c4489363111c78
Yushi Yang
68f754d524c4489363111c79
Shreyansh Padarha
68f754d524c4489363111c7a
Andrew Lee
68f754d524c4489363111c7b
Adam Mahdi
Agentic Reinforcement Learning for Search is Unsafe
Agentic reinforcement learning (RL) trains large language models to
autonomously call tools during reasoning, with search as the most common
application. These models excel at multi-step reasoning tasks, but their safety
properties are not well understood. In this study, we show that RL-trained
search models inherit refusal from instruction tuning and often deflect harmful
requests by turning them into safe queries. However, this safety is fragile.
Two simple attacks, one that forces the model to begin response with search
(Search attack), another that encourages models to repeatedly search
(Multi-search attack), trigger cascades of harmful searches and answers. Across
two model families (Qwen, Llama) with both local and web search, these attacks
lower refusal rates by up to 60.0%, answer safety by 82.5%, and search-query
safety by 82.4%. The attacks succeed by triggering models to generate harmful,
request-mirroring search queries before they can generate the inherited refusal
tokens. This exposes a core weakness of current RL training: it rewards
continued generation of effective queries without accounting for their
harmfulness. As a result, RL search models have vulnerabilities that users can
easily exploit, making it urgent to develop safety-aware agentic RL pipelines
optimising for safe search.
Agentic reinforcement learning models trained for search tasks inherit safety mechanisms but are vulnerable to attacks that reduce their refusal and safety rates.
agentic reinforcement learning RL large language models tool calling multi-step reasoning safety properties refusal harmful requests search attack multi-search attack Qwen Llama local search web search refusal tokens safe search