Linear Correlation in LM's Compositional Generalization and
  Hallucination
The generalization of language models (LMs) is undergoing active debates,
contrasting their potential for general intelligence with their struggles with
basic knowledge composition (e.g., reverse/transition curse). This paper
uncovers the phenomenon of linear correlations in LMs during knowledge
composition. For explanation, there exists a linear transformation between
certain related knowledge that maps the next token prediction logits from one
prompt to another, e.g., "X lives in the city of" rightarrow "X lives in the
country of" for every given X. This mirrors the linearity in human knowledge
composition, such as Paris rightarrow France. Our findings indicate that the
linear transformation is resilient to large-scale fine-tuning, generalizing
updated knowledge when aligned with real-world relationships, but causing
hallucinations when it deviates. Empirical results suggest that linear
correlation can serve as a potential identifier of LM's generalization.
Finally, we show such linear correlations can be learned with a single
feedforward network and pre-trained vocabulary representations, indicating LM
generalization heavily relies on the latter.
67a97eea96d822bc6e13a1bb
Letian Peng
admin_assigned
2025-02-10T16:17:52.581Z
64323dd503d81fa4d26deaf9
https://cdn-avatars.huggingface.co/v1/production/uploads/64323dd503d81fa4d26deaf9/x3ES8VXEZJljxDWvFWaAf.png
Letian Peng
user
KomeijiForce
67a97eea96d822bc6e13a1bc
Chenyang An
admin_assigned
2025-02-10T16:18:00.824Z
6546d644bab28a482e1956c3
/avatars/b35e53afd1acf56534338b7788b49ee1.svg
Chenyang An
user
chenyang-an
67a97eea96d822bc6e13a1bd
Shibo Hao
admin_assigned
2025-02-10T16:18:07.047Z
660ee5df35d092e3fc2a3685
/avatars/a7e0472fb7ea49973f74e3eea13dc964.svg
Shibo Hao
user
Shibo-UCSD
67a97eea96d822bc6e13a1be
Chengyu Dong
admin_assigned
2025-02-10T16:18:12.963Z
668640a1369b09d564b75509
/avatars/ef70bfdaae307a602f0ce0a0753596c7.svg
CHENGYU_DONG
user
sakuraCY
67a97eea96d822bc6e13a1bf
Jingbo Shang
claimed_verified
2025-02-10T09:49:54.200Z
660655119e3555d648f6c6b5
/avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
Jingbo Shang
user
shangjingbo
Linear Correlation in LM's Compositional Generalization and
  Hallucination
The generalization of language models (LMs) is undergoing active debates,
contrasting their potential for general intelligence with their struggles with
basic knowledge composition (e.g., reverse/transition curse). This paper
uncovers the phenomenon of linear correlations in LMs during knowledge
composition. For explanation, there exists a linear transformation between
certain related knowledge that maps the next token prediction logits from one
prompt to another, e.g., "X lives in the city of" rightarrow "X lives in the
country of" for every given X. This mirrors the linearity in human knowledge
composition, such as Paris rightarrow France. Our findings indicate that the
linear transformation is resilient to large-scale fine-tuning, generalizing
updated knowledge when aligned with real-world relationships, but causing
hallucinations when it deviates. Empirical results suggest that linear
correlation can serve as a potential identifier of LM's generalization.
Finally, we show such linear correlations can be learned with a single
feedforward network and pre-trained vocabulary representations, indicating LM
generalization heavily relies on the latter.
Language models exhibit linear correlations in knowledge composition, resilient to fine-tuning and aligned with real-world relationships, which can be identified and learned using feedforward networks and pre-trained vocabulary.
linear correlations language models knowledge composition linear transformation feedforward network pre-trained vocabulary