Large Reasoning Models Learn Better Alignment from Flawed Thinking
Large reasoning models (LRMs) "think" by generating structured
chain-of-thought (CoT) before producing a final answer, yet they still lack the
ability to reason critically about safety alignment and are easily biased when
a flawed premise is injected into their thought process. We propose RECAP
(Robust Safety Alignment via Counter-Aligned Prefilling), a principled
reinforcement learning (RL) method for post-training that explicitly teaches
models to override flawed reasoning trajectories and reroute to safe and
helpful responses. RECAP trains on a mixture of synthetically generated
counter-aligned CoT prefills and standard prompts, requires no additional
training cost or modifications beyond vanilla reinforcement learning from human
feedback (RLHF), and substantially improves safety and jailbreak robustness,
reduces overrefusal, and preserves core reasoning capability -- all while
maintaining inference token budget. Extensive analysis shows that RECAP-trained
models engage in self-reflection more frequently and remain robust under
adaptive attacks, preserving safety even after repeated attempts to override
their reasoning.
68e15e0973e20ab577841d5e
ShengYun Peng
claimed_verified
2025-10-05T12:45:28.147Z
63756971b18c6b01497ca65f
https://cdn-avatars.huggingface.co/v1/production/uploads/63756971b18c6b01497ca65f/pNiAPL7NuPoHlTPl9huTP.jpeg
Anthony Peng
user
AnthonyPeng
68e15e0973e20ab577841d5f
Eric Smith
claimed_verified
2025-10-07T12:28:43.664Z
64bfc262f71e81cc25f8afdb
/avatars/a61f3512ca3d256abfca82a2102a18a7.svg
Eric Michael Smith
user
ericmsmith
68e15e0973e20ab577841d60
Ivan Evtimov
68e15e0973e20ab577841d61
Song Jiang
68e15e0973e20ab577841d62
Pin-Yu Chen
68e15e0973e20ab577841d63
Hongyuan Zhan
68e15e0973e20ab577841d64
Haozhu Wang
68e15e0973e20ab577841d65
Duen Horng Chau
68e15e0973e20ab577841d66
Mahesh Pasupuleti
claimed_verified
2025-10-07T12:28:39.236Z
65d1bb3cf88da990a6fbd945
/avatars/46143f89be15f9c8e37f022274441ad7.svg
Mahesh Pasupuleti
user
maheshp9
68e15e0973e20ab577841d67
Jianfeng Chi
claimed_verified
2025-10-09T06:52:10.357Z
6530a4eebce21215c3999abd
/avatars/8b22bcbbad5dd4fe5956378f5f343294.svg
Jianfeng Chi
user
jfchi
Large Reasoning Models Learn Better Alignment from Flawed Thinking
Large reasoning models (LRMs) "think" by generating structured
chain-of-thought (CoT) before producing a final answer, yet they still lack the
ability to reason critically about safety alignment and are easily biased when
a flawed premise is injected into their thought process. We propose RECAP
(Robust Safety Alignment via Counter-Aligned Prefilling), a principled
reinforcement learning (RL) method for post-training that explicitly teaches
models to override flawed reasoning trajectories and reroute to safe and
helpful responses. RECAP trains on a mixture of synthetically generated
counter-aligned CoT prefills and standard prompts, requires no additional
training cost or modifications beyond vanilla reinforcement learning from human
feedback (RLHF), and substantially improves safety and jailbreak robustness,
reduces overrefusal, and preserves core reasoning capability -- all while
maintaining inference token budget. Extensive analysis shows that RECAP-trained
models engage in self-reflection more frequently and remain robust under
adaptive attacks, preserving safety even after repeated attempts to override
their reasoning.
RECAP, a reinforcement learning method, enhances the safety and robustness of large reasoning models by teaching them to override flawed reasoning and maintain safety without additional training costs.
chain-of-thought reinforcement learning counter-aligned prefills standard prompts reinforcement learning from human feedback jailbreak robustness overrefusal self-reflection adaptive attacks