Entropic Forces in Deep and Universal Representation Learning
Liu Ziyin, Yizhou Xu, Isaac Chuang
As more and more emergent phenomena are being discovered in deep learning and large-language models, explaining and understanding their cause has become an urgent need. Here, we propose a rigorous entropic-force theory for understanding the learning dynamics of neural networks trained with stochastic gradient descent (SGD) and its variants. Building on the theory of parameter symmetries and an entropic loss landscape, we show that representation learning is crucially governed by emergent entropic forces arising from stochasticity and discrete-time updates. These forces systematically break continuous parameter symmetries and preserve discrete ones, leading to a series of gradient balance phenomena that resemble the equipartition property of thermal systems. These phenomena, in turn, explain the universal alignment of neural representations between AI models and the seemingly contradictory observations of progressive sharpening and flattening during deep learning optimization. We demonstrate that a combination of entropic forces and symmetry breaking is key to understanding emergent phenomena in deep learning.