Towards Safety Reasoning in LLMs: AI-agentic Deliberation for
  Policy-embedded CoT Data Creation
Safety reasoning is a recent paradigm where LLMs reason over safety policies
before generating responses, thereby mitigating limitations in existing safety
measures such as over-refusal and jailbreak vulnerabilities. However,
implementing this paradigm is challenging due to the resource-intensive process
of creating high-quality policy-embedded chain-of-thought (CoT) datasets while
ensuring reasoning remains accurate and free from hallucinations or policy
conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation
for Safety Reasoning, a novel data generation recipe that leverages multi-agent
deliberation to iteratively expand reasoning on safety policies. A data refiner
stage in AIDSAFE ensures high-quality outputs by eliminating repetitive,
redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong
foundation for supervised fine-tuning (SFT)-based safety training.
Additionally, to address the need of preference data in alignment stages, such
as DPO training, we introduce a supplemental recipe that uses belief
augmentation to create distinct selected and rejected CoT samples. Our
evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy
adherence and reasoning quality. Consequently, we show that fine-tuning
open-source LLMs on these CoTs can significantly improve safety generalization
and jailbreak robustness while maintaining acceptable utility and over-refusal
accuracy. AIDSAFE-generated CoT datasets can be found here:
https://huggingface.co/datasets/AmazonScience/AIDSAFE
6838d672f527444e97b746db
Tharindu Kumarage
claimed_verified
2025-06-02T07:47:34.691Z
64779935950ad3b14ddb2a3b
/avatars/7e5f6880c3edf1c6cba8f017e2d82239.svg
Tharindu S Kumarage
user
TharinduSK
6838d672f527444e97b746dc
Ninareh Mehrabi
6838d672f527444e97b746dd
Anil Ramakrishna
6838d672f527444e97b746de
Xinyan Zhao
6838d672f527444e97b746df
Richard Zemel
6838d672f527444e97b746e0
Kai-Wei Chang
6838d672f527444e97b746e1
Aram Galstyan
6838d672f527444e97b746e2
Rahul Gupta
6838d672f527444e97b746e3
Charith Peris
claimed_verified
2025-06-02T07:47:37.344Z
626029fdd47e3dbae32d89e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1651094348689-626029fdd47e3dbae32d89e2.jpeg
Charith Peris
user
cperiz
Towards Safety Reasoning in LLMs: AI-agentic Deliberation for
  Policy-embedded CoT Data Creation
Safety reasoning is a recent paradigm where LLMs reason over safety policies
before generating responses, thereby mitigating limitations in existing safety
measures such as over-refusal and jailbreak vulnerabilities. However,
implementing this paradigm is challenging due to the resource-intensive process
of creating high-quality policy-embedded chain-of-thought (CoT) datasets while
ensuring reasoning remains accurate and free from hallucinations or policy
conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation
for Safety Reasoning, a novel data generation recipe that leverages multi-agent
deliberation to iteratively expand reasoning on safety policies. A data refiner
stage in AIDSAFE ensures high-quality outputs by eliminating repetitive,
redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong
foundation for supervised fine-tuning (SFT)-based safety training.
Additionally, to address the need of preference data in alignment stages, such
as DPO training, we introduce a supplemental recipe that uses belief
augmentation to create distinct selected and rejected CoT samples. Our
evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy
adherence and reasoning quality. Consequently, we show that fine-tuning
open-source LLMs on these CoTs can significantly improve safety generalization
and jailbreak robustness while maintaining acceptable utility and over-refusal
accuracy. AIDSAFE-generated CoT datasets can be found here:
https://huggingface.co/datasets/AmazonScience/AIDSAFE
AIDSAFE uses multi-agent deliberation to create high-quality safety policy datasets, enhancing LLM safety without compromising utility.
LLMs safety reasoning policy-embedded chain-of-thought CoT data generation multi-agent deliberation data refiner supervised fine-tuning belief augmentation DPO training policy adherence reasoning quality safety generalization jailbreak robustness utility over-refusal accuracy