Do Vision and Language Models Share Concepts? A Vector Space Alignment
  Study
Large-scale pretrained language models (LMs) are said to ``lack the ability
to connect utterances to the world'' (Bender and Koller, 2020), because they do
not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so,
one would expect LM representations to be unrelated to representations induced
by vision models. We present an empirical evaluation across four families of
LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
(ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
towards representations isomorphic to those of vision models, subject to
dispersion, polysemy and frequency. This has important implications for both
multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
2023).
668eef0c0b9a12807b071173
Jiaang Li
claimed_verified
2024-07-11T08:27:02.582Z
6686658221f096c7fb4e0b24
/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
Jiaang Li
user
jaagli
668eef0c0b9a12807b071174
Yova Kementchedjhieva
668eef0c0b9a12807b071175
Constanza Fierro
668eef0c0b9a12807b071176
Anders Søgaard
Do Vision and Language Models Share Concepts? A Vector Space Alignment
  Study
Large-scale pretrained language models (LMs) are said to ``lack the ability
to connect utterances to the world'' (Bender and Koller, 2020), because they do
not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so,
one would expect LM representations to be unrelated to representations induced
by vision models. We present an empirical evaluation across four families of
LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
(ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
towards representations isomorphic to those of vision models, subject to
dispersion, polysemy and frequency. This has important implications for both
multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
2023).
Large-scale pretrained language models partially converge to representations similar to those of vision models, impacting multi-modal processing and the debate on language model understanding.
pretrained language models mental models representations vision models BERT GPT-2 OPT LLaMA-2 ResNet SegFormer MAE isomorphic representations dispersion polysemy frequency