A Unifying Account of In-Context Learning as a Bayesian Loss-Complexity Tradeoff
Daniel Wurgaft, Ekdeep S Lubana, Core Francisco Park, Hidenori Tanaka, Gautam Reddy, Noah Goodman
In-Context Learning (ICL) enables language models to adapt to novel tasks, significantly expanding their generality. Prior work has documented a rich phenomenology of ICL in specific, simplified settings—particularly the emergence of multiple solutions as task diversity and the number of observed samples increase. However, it remains unclear why these phenomena arise and whether they can be captured under a unified framework. Motivated by this, and drawing on common assumptions about simplicity bias and neural scaling, we develop a hierarchical Bayesian account that accurately predicts a Transformer’s generalization behavior in standard ICL settings—without requiring access to its parameters. Our framework views Transformer training as a process of estimating evidence for different solutions, and inference-time behavior as a posterior-weighted average over these solutions’ predictions. This perspective naturally implies a tradeoff between loss and complexity among candidate solutions, offering a unified explanation for prior empirical findings and yielding novel predictions. For instance, we show that sublinear sample efficiency during pretraining can induce a superlinear trend in the timescale for transitioning between solutions as task diversity scales. Together, our results advance a unified theory of ICL grounded in tradeoffs between solution loss and complexity in model space.