L²M: Mutual Information Scaling Law for Long-Context Language Modeling
Zhuo Chen, Oriol Comas, Zhuotao Jin, Di Luo, Marin Soljacic
This paper rigorously establishes a *bipartite* mutual information scaling law in natural language that fundamentally governs its long-range dependencies, providing the key to understanding *long-context language modeling*. We demonstrate that the bipartite mutual information captures multi-token interactions in a manner that is distinct from, and scales independently of, conventional two-point mutual information, offering a more comprehensive characterization of the dependencies crucial for modeling long sequences. Leveraging this scaling law, we formulate the **L**ong-context **L**anguage **M**odeling (L²M) condition, which establishes a minimum requirement on the scaling of a model's latent state size for storing past information necessary for effective long-context modeling. Our theoretical framework and its predictions are validated through experiments on both transformer and state-space models. This work provides a crucial theoretical foundation for advancing the design of large language models toward effective longer context capabilities.