Almost Surely Safe Alignment of Large Language Models at Inference-Time
Even highly capable large language models (LLMs) can produce biased or unsafe
responses, and alignment techniques, such as RLHF, aimed at mitigating this
issue, are expensive and prone to overfitting as they retrain the LLM. This
paper introduces a novel inference-time alignment approach that ensures LLMs
generate safe responses almost surely, i.e., with a probability approaching
one. We achieve this by framing the safe generation of inference-time responses
as a constrained Markov decision process within the LLM's latent space.
Crucially, we augment a safety state that tracks the evolution of safety
constraints and enables us to demonstrate formal safety guarantees upon solving
the MDP in the latent space. Building on this foundation, we propose
InferenceGuard, a practical implementation that safely aligns LLMs without
modifying the model weights. Empirically, we demonstrate InferenceGuard
effectively balances safety and task performance, outperforming existing
inference-time alignment methods in generating safe and aligned responses.
67a1e729a6c7d65cad72b3d7
Xiaotong Ji
admin_assigned
2025-02-04T17:40:16.111Z
6682989afc0e69f80acf2845
/avatars/35d48738965fe21fdd79198a17d6c8cc.svg
jixiaotong
user
xiaotong9515
67a1e729a6c7d65cad72b3d8
Shyam Sundhar Ramesh
67a1e729a6c7d65cad72b3d9
Matthieu Zimmer
67a1e729a6c7d65cad72b3da
Ilija Bogunovic
admin_assigned
2025-02-04T17:39:45.148Z
65fd58a41a58717c800d3650
/avatars/9dce361a0417465116c816abdf53e916.svg
Bogunovic
user
ilijabogunovic
67a1e729a6c7d65cad72b3db
Jun Wang
67a1e729a6c7d65cad72b3dc
Haitham Bou Ammar
admin_assigned
2025-02-04T17:39:59.356Z
631c375768f7da9ad2496bf6
https://cdn-avatars.huggingface.co/v1/production/uploads/631c375768f7da9ad2496bf6/1sDOoecA6e1v_hn_VAgUq.jpeg
Haitham Bou Ammar
user
hba123
Almost Surely Safe Alignment of Large Language Models at Inference-Time
Even highly capable large language models (LLMs) can produce biased or unsafe
responses, and alignment techniques, such as RLHF, aimed at mitigating this
issue, are expensive and prone to overfitting as they retrain the LLM. This
paper introduces a novel inference-time alignment approach that ensures LLMs
generate safe responses almost surely, i.e., with a probability approaching
one. We achieve this by framing the safe generation of inference-time responses
as a constrained Markov decision process within the LLM's latent space.
Crucially, we augment a safety state that tracks the evolution of safety
constraints and enables us to demonstrate formal safety guarantees upon solving
the MDP in the latent space. Building on this foundation, we propose
InferenceGuard, a practical implementation that safely aligns LLMs without
modifying the model weights. Empirically, we demonstrate InferenceGuard
effectively balances safety and task performance, outperforming existing
inference-time alignment methods in generating safe and aligned responses.
A novel inference-time alignment approach ensures large language models generate safe responses with high probability using a constrained Markov decision process in the latent space.
large language models LLMs alignment techniques RLHF inference-time alignment constrained Markov decision process latent space safety state safety constraints formal safety guarantees InferenceGuard