The Geometry of Reasoning: Flowing Logics in Representation Space
We study how large language models (LLMs) ``think'' through their
representation space. We propose a novel geometric framework that models an
LLM's reasoning as flows -- embedding trajectories evolving where logic goes.
We disentangle logical structure from semantics by employing the same natural
deduction propositions with varied semantic carriers, allowing us to test
whether LLMs internalize logic beyond surface form. This perspective connects
reasoning with geometric quantities such as position, velocity, and curvature,
enabling formal analysis in representation and concept spaces. Our theory
establishes: (1) LLM reasoning corresponds to smooth flows in representation
space, and (2) logical statements act as local controllers of these flows'
velocities. Using learned representation proxies, we design controlled
experiments to visualize and quantify reasoning flows, providing empirical
validation of our theoretical framework. Our work serves as both a conceptual
foundation and practical tools for studying reasoning phenomenon, offering a
new lens for interpretability and formal analysis of LLMs' behavior.
68eeda16486b78128f0e3277
Yufa Zhou
claimed_verified
2025-10-16T06:41:59.799Z
658ab894c4b2004663dff3ae
https://cdn-avatars.huggingface.co/v1/production/uploads/658ab894c4b2004663dff3ae/oPRnFuW2Imaa2KkYWNbSf.jpeg
YUFA ZHOU
user
MasterZhou
68eeda16486b78128f0e3278
Yixiao Wang
68eeda16486b78128f0e3279
Xunjian Yin
68eeda16486b78128f0e327a
Shuyan Zhou
68eeda16486b78128f0e327b
Anru R. Zhang
The Geometry of Reasoning: Flowing Logics in Representation Space
We study how large language models (LLMs) ``think'' through their
representation space. We propose a novel geometric framework that models an
LLM's reasoning as flows -- embedding trajectories evolving where logic goes.
We disentangle logical structure from semantics by employing the same natural
deduction propositions with varied semantic carriers, allowing us to test
whether LLMs internalize logic beyond surface form. This perspective connects
reasoning with geometric quantities such as position, velocity, and curvature,
enabling formal analysis in representation and concept spaces. Our theory
establishes: (1) LLM reasoning corresponds to smooth flows in representation
space, and (2) logical statements act as local controllers of these flows'
velocities. Using learned representation proxies, we design controlled
experiments to visualize and quantify reasoning flows, providing empirical
validation of our theoretical framework. Our work serves as both a conceptual
foundation and practical tools for studying reasoning phenomenon, offering a
new lens for interpretability and formal analysis of LLMs' behavior.