Knowing When to Stop: Dynamic Context Cutoff for Large Language Models
Roy Xie, Junlin Wang, Paul Rosu, Chunyuan Deng, Bolun Sun, Zihao Lin, Bhuwan Dhingra
Large language models (LLMs) process entire input contexts indiscriminately, which is inefficient in cases where theinformation required to answer a query is localized within the context.We present dynamic context cutoff, a human-inspired method enabling LLMs to self-terminate processing upon acquiring sufficient task-relevant information. Through analysis of model internals, we discover that specific attention heads inherently encode ``sufficiency signals'' – detectable through lightweight classifiers – that predict when critical information has been processed. This reveals a new efficiency paradigm: models' internal understanding naturally dictates processing needs rather than external compression heuristics. Comprehensive experiments across six QA datasets (up to 40K tokens) with three model families (LLaMA/Qwen/Mistral, 1B-70B) demonstrate $3.4$\% accuracy improvement while achieving $1.33\times$ token reduction on average. Furthermore, our method demonstrates better performance with the same rate of token reduction compared to other context efficiency methods.Additionally, we observe an emergent scaling phenomenon: while smaller models require probing for sufficiency detection, larger models exhibit intrinsic self-assessment capabilities through prompting.