Language Model Behavioral Phases are Consistent Across Scale and Architecture
James Michaelov, Roger Levy, Benjamin Bergen
We show that across architecture (transformer vs. RWKV vs. Mamba), training dataset (OpenWebText vs. The Pile), and scale (14 million parameters to 12 billion parameters), autoregressive language models exhibit highly consistent patterns of change in behavior through pre-training. Based on our analysis of over 1,400 language model checkpoints on over 110,000 tokens of English, we find that up to 98\% of the variance in language model behavior at the word level can be explained by three simple heuristics: the unigram probability (frequency) of a given word, the $n$-gram probability of the word, and the semantic similarity between the word and its context. Furthermore, we see consistent behavioral phases in all language models, with their predicted probabilities for words overfitting to the words' $n$-gram probabilities for increasing $n$ over the course of training, as well as becoming increasingly sensitive to semantic similarity early on in training. Taken together, these results suggest that learning in language models may take a similar path irrespective of model details.