An Attempt to Use Synthetic Pretraining Playgrounds for Language Model Architecture Design
Zeyuan Allen-Zhu
Understanding architectural differences in language models is challenging, particularly at academic-scale pretraining (e.g., 1.3B params, 100B tokens), where results are often dominated by noise. We propose synthetic pretraining tasks to isolate and evaluate key model capabilities. Using this framework, we identify \emph{Canon layers}: lightweight components—named after the musical term—that enhance horizontal information flow across neighboring tokens. Canon layers compute weighted sum of nearby token representations and integrate seamlessly into Transformers, linear attention, state-space models, or any sequence architecture.We present 12 results, including how Canon enhances reasoning depth ($2\times$), reasoning breadth, knowledge manipulation, etc. Canon transforms weak architectures like NoPE to match RoPE and linear attention to rival state-space models (e.g., Mamba2), validated through synthetic tasks and real-world academic-scale pretraining. This synthetic framework isolates core capabilities often obscured at academic scales, offering an \emph{economical, principled path} to guide architecture design as pretraining pipelines improve, aiming to unlock deeper reasoning.