Geospatial Mechanistic Interpretability of Large Language Models
Large Language Models (LLMs) have demonstrated unprecedented capabilities
across various natural language processing tasks. Their ability to process and
generate viable text and code has made them ubiquitous in many fields, while
their deployment as knowledge bases and "reasoning" tools remains an area of
ongoing research. In geography, a growing body of literature has been focusing
on evaluating LLMs' geographical knowledge and their ability to perform spatial
reasoning. However, very little is still known about the internal functioning
of these models, especially about how they process geographical information.
  In this chapter, we establish a novel framework for the study of geospatial
mechanistic interpretability - using spatial analysis to reverse engineer how
LLMs handle geographical information. Our aim is to advance our understanding
of the internal representations that these complex models generate while
processing geographical information - what one might call "how LLMs think about
geographic information" if such phrasing was not an undue anthropomorphism.
  We first outline the use of probing in revealing internal structures within
LLMs. We then introduce the field of mechanistic interpretability, discussing
the superposition hypothesis and the role of sparse autoencoders in
disentangling polysemantic internal representations of LLMs into more
interpretable, monosemantic features. In our experiments, we use spatial
autocorrelation to show how features obtained for placenames display spatial
patterns related to their geographic location and can thus be interpreted
geospatially, providing insights into how these models process geographical
information. We conclude by discussing how our framework can help shape the
study and use of foundation models in geography.
681b05ab8e325b4097318969
Stef De Sabbata
claimed_verified
2025-05-07T08:08:24.829Z
661e841972c7030f3fd57eb9
https://cdn-avatars.huggingface.co/v1/production/uploads/661e841972c7030f3fd57eb9/PysszxrOl__sZ-IToZfJU.jpeg
Stef De Sabbata
user
sdesabbata
681b05ab8e325b409731896a
Stefano Mizzaro
681b05ab8e325b409731896b
Kevin Roitero
Geospatial Mechanistic Interpretability of Large Language Models
Large Language Models (LLMs) have demonstrated unprecedented capabilities
across various natural language processing tasks. Their ability to process and
generate viable text and code has made them ubiquitous in many fields, while
their deployment as knowledge bases and "reasoning" tools remains an area of
ongoing research. In geography, a growing body of literature has been focusing
on evaluating LLMs' geographical knowledge and their ability to perform spatial
reasoning. However, very little is still known about the internal functioning
of these models, especially about how they process geographical information.
  In this chapter, we establish a novel framework for the study of geospatial
mechanistic interpretability - using spatial analysis to reverse engineer how
LLMs handle geographical information. Our aim is to advance our understanding
of the internal representations that these complex models generate while
processing geographical information - what one might call "how LLMs think about
geographic information" if such phrasing was not an undue anthropomorphism.
  We first outline the use of probing in revealing internal structures within
LLMs. We then introduce the field of mechanistic interpretability, discussing
the superposition hypothesis and the role of sparse autoencoders in
disentangling polysemantic internal representations of LLMs into more
interpretable, monosemantic features. In our experiments, we use spatial
autocorrelation to show how features obtained for placenames display spatial
patterns related to their geographic location and can thus be interpreted
geospatially, providing insights into how these models process geographical
information. We conclude by discussing how our framework can help shape the
study and use of foundation models in geography.
A framework for understanding how Large Language Models process geographical information using spatial analysis and mechanistic interpretability techniques.
Large Language Models geospatial mechanistic interpretability spatial analysis probing mechanistic interpretability superposition hypothesis sparse autoencoders polysemantic internal representations monosemantic features spatial autocorrelation placenames geographic location