Correlation Dimension of Autoregressive Large Language Models
Xin Du, Kumiko Tanaka-Ishii
Large language models (LLMs) have achieved impressive results in naturallanguage generation but continue to exhibit unexplained behaviors—such ashallucinations, repetition, and incoherence—even at low perplexity. This exposesthe limitations of traditional evaluation metrics, which focus on localprediction accuracy while neglecting long-range structural complexity. Weintroduce the correlation dimension, a fractal-geometric measure ofself-similarity, to quantify the implicit textual complexity perceived bylanguage models. Correlation dimension robustly captures the hierarchicalrecurrence structure of text, bridging local and global properties. Experimentsdemonstrate that this measure reveals distinct training phases, reflectscontext-dependent complexity, and reliably detects various forms of degenerationin generated text. The method is computationally efficient, broadly applicableto autoregressive models, and provides a theoretically grounded approach forunderstanding complex generative dynamics of LLMs.