Steering When Necessary: Flexible Steering Large Language Models with Backtracking
Jinwei Gan, Zifeng Cheng, Zhiwei Jiang, Cong Wang, Yafeng Yin, Xiang Luo, Yuchen Fu, Qing Gu
Large language models (LLMs) have achieved remarkable performance across many generation tasks.However, effectively aligning them with desired behaviors remains a significant challenge.Activation steering is an effective and cost-efficient approach that directly modifies the activations of LLMs during the inference stage, aligning their responses with the desired behaviors and avoiding the high cost of fine-tuning.Existing methods often rigidly intervene at the first token of all responses, causing the LLM's replies to deviate from their original output and leading to homogenized and less informative responses.To this end, we propose a **F**lexible **A**ctivation **S**teering with **B**acktracking (FASB) framework that flexibly determines whether intervention is needed, when to intervene, and the intervention strength by probing the internal states.The backtracking mechanism enables the correction of deviated tokens and helps steer the model toward the desired behavior.Extensive experiments on the TruthfulQA dataset and six multiple-choice datasets demonstrate that our method outperforms baselines.