Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety
  Assurance
Existing safety assurance research has primarily focused on training-phase
alignment to instill safe behaviors into LLMs. However, recent studies have
exposed these methods' susceptibility to diverse jailbreak attacks.
Concurrently, inference scaling has significantly advanced LLM reasoning
capabilities but remains unexplored in the context of safety assurance.
Addressing this gap, our work pioneers inference scaling for robust and
effective LLM safety against emerging threats. We reveal that conventional
inference scaling techniques, despite their success in reasoning tasks, perform
poorly in safety contexts, even falling short of basic approaches like
Best-of-N Sampling. We attribute this inefficiency to a newly identified
challenge, the exploration--efficiency dilemma, arising from the high
computational overhead associated with frequent process reward model (PRM)
evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference
scaling paradigm tailored explicitly for safety assurance. Central to our
approach is the introduction of a multifurcation reward model (MRM) that
significantly reduces the required number of reward model evaluations. To
operationalize this paradigm, we further propose: (i) a partial supervision
training objective for MRM, (ii) a conservative exploration constraint to
prevent out-of-distribution explorations, and (iii) a Trie-based key--value
caching strategy that facilitates cache sharing across sequences during tree
search. Extensive experiments validate the effectiveness of our method.
Additionally, we publicly release our trained multifurcation reward model
(Saffron-1) and the accompanying token-level safety reward dataset (Safety4M)
to accelerate future research in LLM safety. Our code, model, and data are
publicly available at https://github.com/q-rz/saffron , and our project
homepage is at https://q-rz.github.io/p/saffron .
68479c1e3ec10bdd8ab4de9d
Ruizhong Qiu
admin_assigned
2025-06-10T15:57:26.506Z
65370d95019de94263ad34a7
https://cdn-avatars.huggingface.co/v1/production/uploads/65370d95019de94263ad34a7/fH9SoJfX7yifXup0DvXkm.jpeg
Ruizhong Qiu
user
q-rz
68479c1e3ec10bdd8ab4de9e
Gaotang Li
admin_assigned
2025-06-10T15:57:58.409Z
654d784d71a30c4bca09a319
/avatars/ab9f93122903ccd662267232bab30ad8.svg
Gaotang Li
user
gaotang
68479c1e3ec10bdd8ab4de9f
Tianxin Wei
admin_assigned
2025-06-10T15:57:51.871Z
647bc5022d27d3541df04a91
/avatars/e062092036da6c38f700362c0c5437c0.svg
Tianxin Wei
user
tianxinwei
68479c1e3ec10bdd8ab4dea0
Jingrui He
68479c1e3ec10bdd8ab4dea1
Hanghang Tong
Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety
  Assurance
Existing safety assurance research has primarily focused on training-phase
alignment to instill safe behaviors into LLMs. However, recent studies have
exposed these methods' susceptibility to diverse jailbreak attacks.
Concurrently, inference scaling has significantly advanced LLM reasoning
capabilities but remains unexplored in the context of safety assurance.
Addressing this gap, our work pioneers inference scaling for robust and
effective LLM safety against emerging threats. We reveal that conventional
inference scaling techniques, despite their success in reasoning tasks, perform
poorly in safety contexts, even falling short of basic approaches like
Best-of-N Sampling. We attribute this inefficiency to a newly identified
challenge, the exploration--efficiency dilemma, arising from the high
computational overhead associated with frequent process reward model (PRM)
evaluations. To overcome this dilemma, we propose SAFFRON, a novel inference
scaling paradigm tailored explicitly for safety assurance. Central to our
approach is the introduction of a multifurcation reward model (MRM) that
significantly reduces the required number of reward model evaluations. To
operationalize this paradigm, we further propose: (i) a partial supervision
training objective for MRM, (ii) a conservative exploration constraint to
prevent out-of-distribution explorations, and (iii) a Trie-based key--value
caching strategy that facilitates cache sharing across sequences during tree
search. Extensive experiments validate the effectiveness of our method.
Additionally, we publicly release our trained multifurcation reward model
(Saffron-1) and the accompanying token-level safety reward dataset (Safety4M)
to accelerate future research in LLM safety. Our code, model, and data are
publicly available at https://github.com/q-rz/saffron , and our project
homepage is at https://q-rz.github.io/p/saffron .
SAFFRON, a novel inference scaling paradigm, enhances LLM safety by reducing reward model evaluations through a multifurcation reward model and other optimizations.
LLMs inference scaling safety assurance jailbreak attacks Best-of-N Sampling process reward model exploration--efficiency dilemma multifurcation reward model partial supervision training conservative exploration constraint Trie-based key--value caching Safety4M dataset