LayerCake: Token-Aware Contrastive Decoding within Large Language Model
  Layers
Large language models (LLMs) excel at natural language understanding and
generation but remain vulnerable to factual errors, limiting their reliability
in knowledge-intensive tasks. While decoding-time strategies provide a
promising efficient solution without training, existing methods typically treat
token-level and layer-level signals in isolation, overlooking the joint
dynamics between them. In this work, we introduce a token-aware,
layer-localized contrastive decoding method that aligns specific token types
with their most influential transformer layers to improve factual generation.
Through empirical attention analysis, we identify two key patterns: punctuation
tokens receive dominant attention in early layers, while conceptual tokens
govern semantic reasoning in intermediate layers. By selectively suppressing
attention to these token types at their respective depths, we achieve the
induction of controlled factual degradation and derive contrastive signals to
guide the final factual decoding. Our method requires no additional training or
model modification, and experiments demonstrate that our method consistently
improves factuality across multiple LLMs and various benchmarks.
6875d1a3257d4f043537058e
Jingze Zhu
6875d1a3257d4f043537058f
Yongliang Wu
6875d1a3257d4f0435370590
Wenbo Zhu
6875d1a3257d4f0435370591
Jiawang Cao
6875d1a3257d4f0435370592
Yanqiang Zheng
6875d1a3257d4f0435370593
Jiawei Chen
6875d1a3257d4f0435370594
Xu Yang
6875d1a3257d4f0435370595
Bernt Schiele
6875d1a3257d4f0435370596
Jonas Fischer
6875d1a3257d4f0435370597
Xinting Hu
LayerCake: Token-Aware Contrastive Decoding within Large Language Model
  Layers
Large language models (LLMs) excel at natural language understanding and
generation but remain vulnerable to factual errors, limiting their reliability
in knowledge-intensive tasks. While decoding-time strategies provide a
promising efficient solution without training, existing methods typically treat
token-level and layer-level signals in isolation, overlooking the joint
dynamics between them. In this work, we introduce a token-aware,
layer-localized contrastive decoding method that aligns specific token types
with their most influential transformer layers to improve factual generation.
Through empirical attention analysis, we identify two key patterns: punctuation
tokens receive dominant attention in early layers, while conceptual tokens
govern semantic reasoning in intermediate layers. By selectively suppressing
attention to these token types at their respective depths, we achieve the
induction of controlled factual degradation and derive contrastive signals to
guide the final factual decoding. Our method requires no additional training or
model modification, and experiments demonstrate that our method consistently
improves factuality across multiple LLMs and various benchmarks.
A token-aware, layer-localized contrastive decoding method improves factual generation in large language models by selectively suppressing attention to specific token types at their respective depths.
large language models natural language understanding natural language generation factual errors decoding-time strategies token-level signals layer-level signals token-aware layer-localized contrastive decoding transformer layers punctuation tokens conceptual tokens semantic reasoning attention analysis factual generation factual decoding