AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety
  Alignment of Large Reasoning Models
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in
complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the
multi-step nature of CoT introduces new safety challenges that extend beyond
conventional language model alignment. We identify a failure mode in current
safety CoT tuning methods: the snowball effect, where minor reasoning
deviations progressively amplify throughout the thought process, leading to
either harmful compliance or excessive refusal. This effect stems from models
being trained to imitate perfect reasoning scripts without learning to
self-correct. To address this limitation, we propose AdvChain, an alignment
paradigm that teaches models dynamic self-correction through adversarial CoT
tuning. Our method involves constructing a dataset containing
Temptation-Correction and Hesitation-Correction samples, where models learn to
recover from harmful reasoning drifts and unnecessary cautions. Extensive
experiments show that AdvChain significantly enhances robustness against
jailbreak attacks and CoT hijacking while substantially reducing over-refusal
on benign prompts, achieving a superior safety-utility balance without
compromising reasoning capabilities. Our work establishes a new direction for
building more robust and reliable reasoning models.
68db45a6d2bf1f4b15ec7381
Zihao Zhu
admin_assigned
2025-09-30T12:17:36.269Z
64030d9956038547951c7d55
https://cdn-avatars.huggingface.co/v1/production/uploads/64030d9956038547951c7d55/gRCzhy1yd7cOV6GN-KGHJ.png
Zihao Zhu
user
ZihaoZhu
68db45a6d2bf1f4b15ec7382
Xinyu Wu
68db45a6d2bf1f4b15ec7383
Gehan Hu
admin_assigned
2025-09-30T12:17:50.452Z
68885d808b05c7ce47ba02eb
/avatars/f627e3ea16e94121b87c0a51aa0e31a4.svg
Gehan Hu
user
hugehan
68db45a6d2bf1f4b15ec7384
Siwei Lyu
68db45a6d2bf1f4b15ec7385
Ke Xu
68db45a6d2bf1f4b15ec7386
Baoyuan Wu
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety
  Alignment of Large Reasoning Models
Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in
complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the
multi-step nature of CoT introduces new safety challenges that extend beyond
conventional language model alignment. We identify a failure mode in current
safety CoT tuning methods: the snowball effect, where minor reasoning
deviations progressively amplify throughout the thought process, leading to
either harmful compliance or excessive refusal. This effect stems from models
being trained to imitate perfect reasoning scripts without learning to
self-correct. To address this limitation, we propose AdvChain, an alignment
paradigm that teaches models dynamic self-correction through adversarial CoT
tuning. Our method involves constructing a dataset containing
Temptation-Correction and Hesitation-Correction samples, where models learn to
recover from harmful reasoning drifts and unnecessary cautions. Extensive
experiments show that AdvChain significantly enhances robustness against
jailbreak attacks and CoT hijacking while substantially reducing over-refusal
on benign prompts, achieving a superior safety-utility balance without
compromising reasoning capabilities. Our work establishes a new direction for
building more robust and reliable reasoning models.
AdvChain enhances the safety and reliability of large reasoning models by teaching them dynamic self-correction through adversarial chain-of-thought tuning.
Chain-of-Thought CoT reasoning snowball effect adversarial CoT tuning Temptation-Correction Hesitation-Correction jailbreak attacks CoT hijacking over-refusal reasoning capabilities