The Semantic Hub Hypothesis: Language Models Share Semantic
  Representations Across Languages and Modalities
Modern language models can process inputs across diverse languages and
modalities. We hypothesize that models acquire this capability through learning
a shared representation space across heterogeneous data types (e.g., different
languages and modalities), which places semantically similar inputs near one
another, even if they are from different modalities/languages. We term this the
semantic hub hypothesis, following the hub-and-spoke model from neuroscience
(Patterson et al., 2007) which posits that semantic knowledge in the human
brain is organized through a transmodal semantic "hub" which integrates
information from various modality-specific "spokes" regions. We first show that
model representations for semantically equivalent inputs in different languages
are similar in the intermediate layers, and that this space can be interpreted
using the model's dominant pretraining language via the logit lens. This
tendency extends to other data types, including arithmetic expressions, code,
and visual/audio inputs. Interventions in the shared representation space in
one data type also predictably affect model outputs in other data types,
suggesting that this shared representations space is not simply a vestigial
byproduct of large-scale training on broad data, but something that is actively
utilized by the model during input processing.
67321c68254c0b2144d7a068
Zhaofeng Wu
claimed_verified
2025-02-14T08:01:54.054Z
6351712b40dffad651f128c7
/avatars/87708c86c1baef548ef556f5d32dca71.svg
Zhaofeng Wu
user
ZhaofengWu
67321c68254c0b2144d7a069
Xinyan Velocity Yu
67321c68254c0b2144d7a06a
Dani Yogatama
67321c68254c0b2144d7a06b
Jiasen Lu
67321c68254c0b2144d7a06c
Yoon Kim
The Semantic Hub Hypothesis: Language Models Share Semantic
  Representations Across Languages and Modalities
Modern language models can process inputs across diverse languages and
modalities. We hypothesize that models acquire this capability through learning
a shared representation space across heterogeneous data types (e.g., different
languages and modalities), which places semantically similar inputs near one
another, even if they are from different modalities/languages. We term this the
semantic hub hypothesis, following the hub-and-spoke model from neuroscience
(Patterson et al., 2007) which posits that semantic knowledge in the human
brain is organized through a transmodal semantic "hub" which integrates
information from various modality-specific "spokes" regions. We first show that
model representations for semantically equivalent inputs in different languages
are similar in the intermediate layers, and that this space can be interpreted
using the model's dominant pretraining language via the logit lens. This
tendency extends to other data types, including arithmetic expressions, code,
and visual/audio inputs. Interventions in the shared representation space in
one data type also predictably affect model outputs in other data types,
suggesting that this shared representations space is not simply a vestigial
byproduct of large-scale training on broad data, but something that is actively
utilized by the model during input processing.
Language models develop a shared representation space that organizes semantically similar inputs from diverse languages and modalities, facilitating cross-modal and cross-lingual understanding and processing.
semantic hub hypothesis hub-and-spoke model semantic representation space pretraining language logit lens arithmetic expressions code visual/audio inputs