CorrSteer: Steering Improves Task Performance and Safety in LLMs through
  Correlation-based Sparse Autoencoder Feature Selection
Sparse Autoencoders (SAEs) can extract interpretable features from large
language models (LLMs) without supervision. However, their effectiveness in
downstream steering tasks is limited by the requirement for contrastive
datasets or large activation storage. To address these limitations, we propose
CorrSteer, which selects features by correlating sample correctness with SAE
activations from generated tokens at inference time. This approach uses only
inference-time activations to extract more relevant features, thereby avoiding
spurious correlations. It also obtains steering coefficients from average
activations, automating the entire pipeline. Our method shows improved task
performance on QA, bias mitigation, jailbreaking prevention, and reasoning
benchmarks on Gemma 2 2B and LLaMA 3.1 8B, notably achieving a +4.1%
improvement in MMLU performance and a +22.9% improvement in HarmBench with only
4000 samples. Selected features demonstrate semantically meaningful patterns
aligned with each task's requirements, revealing the underlying capabilities
that drive performance. Our work establishes correlationbased selection as an
effective and scalable approach for automated SAE steering across language
model applications.
68a57f3c38160bfd39426a41
Seonglae Cho
claimed_verified
2025-08-20T08:12:44.053Z
64105805928400b416439f10
https://cdn-avatars.huggingface.co/v1/production/uploads/64105805928400b416439f10/i0jFLo47RTDeNl26hiR9y.jpeg
Seonglae Cho
user
seonglae
68a57f3c38160bfd39426a42
Zekun Wu
68a57f3c38160bfd39426a43
Adriano Koshiyama
CorrSteer: Steering Improves Task Performance and Safety in LLMs through
  Correlation-based Sparse Autoencoder Feature Selection
Sparse Autoencoders (SAEs) can extract interpretable features from large
language models (LLMs) without supervision. However, their effectiveness in
downstream steering tasks is limited by the requirement for contrastive
datasets or large activation storage. To address these limitations, we propose
CorrSteer, which selects features by correlating sample correctness with SAE
activations from generated tokens at inference time. This approach uses only
inference-time activations to extract more relevant features, thereby avoiding
spurious correlations. It also obtains steering coefficients from average
activations, automating the entire pipeline. Our method shows improved task
performance on QA, bias mitigation, jailbreaking prevention, and reasoning
benchmarks on Gemma 2 2B and LLaMA 3.1 8B, notably achieving a +4.1%
improvement in MMLU performance and a +22.9% improvement in HarmBench with only
4000 samples. Selected features demonstrate semantically meaningful patterns
aligned with each task's requirements, revealing the underlying capabilities
that drive performance. Our work establishes correlationbased selection as an
effective and scalable approach for automated SAE steering across language
model applications.
CorrSteer improves language model performance by selecting relevant features through correlation with SAE activations at inference time, enhancing tasks like QA, bias mitigation, and reasoning.
Sparse Autoencoders SAEs large language models LLMs contrastive datasets activation storage sample correctness inference-time activations steering coefficients MMLU HarmBench semantically meaningful patterns