Nested Learning: The Illusion of Deep Learning Architectures
Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni
Developing more powerful neural architectures has been in the core of research studies to enhance the capability of foundation models. There has also been an extensive research effort on how to effectively train such architectures using optimization algorithms. While these two components and their roles in modern foundation models are decoupled in the literature, in this paper, we show that how inexplicably these two levels have been separated and introduce Nested Learning (NL) paradigm that represents a model with a set of nested optimization problems. We introduce \emph{Neural Learning Module} that sees the entire process of training a neural architecture and its optimization process as a single integrated system, optimizing a set of objectives at different levels. While having meta-learning as the special instance of nested learning (bi-level), we show how NL goes further by letting (i) the model have more nested layers; and (ii) optimization blocks be placed simultaneously on the same level, addressing different aspects of learning. Taking advantage of the concept of NL, we present a family of nested optimizers based on gradient descent that unifies, explains, and extends existing optimizers, including those with momentum and pre-conditioning techniques. NL provides a mathematically white box deep models, further enhancing the interpretability, and explain how an upper-level optimization block is learning the algorithms in its inner objectives. Building upon this insight, we present Hope, a neural learning module that is dynamically updating its weights using a single  internal objective, adapting its memory based on its past. Our experimental results on NL paradigm, general optimizers, and Hope indicates: (1) the effectiveness of more nested levels; (2) the superior or on-par performance of general optimizers compared to existing well-established methods; and (3) the good performance of Hope neural learning module in tasks such as language modeling, common-sense reasoning, and recall-intensive tasks.