CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge
  into Large Language Models
Large Language Models (LLMs) have achieved remarkable success across various
domains. However, a fundamental question remains: Can LLMs effectively utilize
causal knowledge for prediction and generation? Through empirical studies, we
find that LLMs trained directly on large-scale data often capture spurious
correlations rather than true causal relationships, leading to suboptimal
performance, especially in out-of-distribution (OOD) scenarios. To address this
challenge, we propose Causal Attention Tuning (CAT), a novel approach that
injects fine-grained causal knowledge into the attention mechanism. We propose
an automated pipeline that leverages human priors to automatically generate
token-level causal signals and introduce the Re-Attention mechanism to guide
training, helping the model focus on causal structures while mitigating noise
and biases in attention scores. Experimental results on our proposed Spurious
Token Game (STG) benchmark and multiple downstream tasks demonstrate that our
approach effectively leverages causal knowledge for prediction and remains
robust in OOD scenarios. Implementation details can be found at
https://github.com/Kairong-Han/CAT.
68b85733d43cadaf7a688a0f
Kairong Han
claimed_verified
2025-09-15T15:09:25.856Z
68c6b73ad0bef8d72febf45d
/avatars/7b65febb59f2524a81b08c9f6bd881f5.svg
Kairong Han
user
Kairong-Han
68b85733d43cadaf7a688a10
Wenshuo Zhao
68b85733d43cadaf7a688a11
Ziyu Zhao
68b85733d43cadaf7a688a12
JunJian Ye
68b85733d43cadaf7a688a13
Lujia Pan
68b85733d43cadaf7a688a14
Kun Kuang
CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge
  into Large Language Models
Large Language Models (LLMs) have achieved remarkable success across various
domains. However, a fundamental question remains: Can LLMs effectively utilize
causal knowledge for prediction and generation? Through empirical studies, we
find that LLMs trained directly on large-scale data often capture spurious
correlations rather than true causal relationships, leading to suboptimal
performance, especially in out-of-distribution (OOD) scenarios. To address this
challenge, we propose Causal Attention Tuning (CAT), a novel approach that
injects fine-grained causal knowledge into the attention mechanism. We propose
an automated pipeline that leverages human priors to automatically generate
token-level causal signals and introduce the Re-Attention mechanism to guide
training, helping the model focus on causal structures while mitigating noise
and biases in attention scores. Experimental results on our proposed Spurious
Token Game (STG) benchmark and multiple downstream tasks demonstrate that our
approach effectively leverages causal knowledge for prediction and remains
robust in OOD scenarios. Implementation details can be found at
https://github.com/Kairong-Han/CAT.
Causal Attention Tuning (CAT) enhances Large Language Models (LLMs) by injecting causal knowledge into the attention mechanism, improving prediction accuracy and robustness in out-of-distribution scenarios.
Large Language Models (LLMs) causal knowledge spurious correlations true causal relationships out-of-distribution (OOD) scenarios Causal Attention Tuning (CAT) token-level causal signals Re-Attention mechanism Spurious Token Game (STG)