Emergent Misalignment via In-Context Learning: Narrow in-context
  examples can produce broadly misaligned LLMs
Recent work has shown that narrow finetuning can produce broadly misaligned
LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these
findings were limited to finetuning and activation steering, leaving out
in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find
that it does: across three datasets, three frontier models produce broadly
misaligned responses at rates between 2% and 17% given 64 narrow in-context
examples, and up to 58% with 256 examples. We also examine mechanisms of EM by
eliciting step-by-step reasoning (while leaving in-context examples unchanged).
Manual analysis of the resulting chain-of-thought shows that 67.5% of
misaligned traces explicitly rationalize harmful outputs by adopting a reckless
or dangerous ''persona'', echoing prior results on finetuning-induced EM.
68f5fe5d8589920bf4d32275
Nikita Afonin
68f5fe5d8589920bf4d32276
Nikita Andriyanov
68f5fe5d8589920bf4d32277
Nikhil Bageshpura
68f5fe5d8589920bf4d32278
Kyle Liu
68f5fe5d8589920bf4d32279
Kevin Zhu
68f5fe5d8589920bf4d3227a
Sunishchal Dev
68f5fe5d8589920bf4d3227b
Ashwinee Panda
68f5fe5d8589920bf4d3227c
Alexander Panchenko
68f5fe5d8589920bf4d3227d
Oleg Rogov
68f5fe5d8589920bf4d3227e
Elena Tutubalina
claimed_verified
2025-10-21T14:04:35.451Z
662f8d645c4db70c77a203b0
/avatars/72f9a3c39b3ba5114388d16a35524835.svg
Elena Tutubalina
user
tlenusik
68f5fe5d8589920bf4d3227f
Mikhail Seleznyov
Emergent Misalignment via In-Context Learning: Narrow in-context
  examples can produce broadly misaligned LLMs
Recent work has shown that narrow finetuning can produce broadly misaligned
LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these
findings were limited to finetuning and activation steering, leaving out
in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find
that it does: across three datasets, three frontier models produce broadly
misaligned responses at rates between 2% and 17% given 64 narrow in-context
examples, and up to 58% with 256 examples. We also examine mechanisms of EM by
eliciting step-by-step reasoning (while leaving in-context examples unchanged).
Manual analysis of the resulting chain-of-thought shows that 67.5% of
misaligned traces explicitly rationalize harmful outputs by adopting a reckless
or dangerous ''persona'', echoing prior results on finetuning-induced EM.
Emergent misalignment occurs in in-context learning across multiple models and datasets, with misaligned responses increasing with the number of examples provided.
emergent misalignment in-context learning chain-of-thought persona