The World Is Bigger: A Computationally-Embedded Perspective on the Big World Hypothesis
Alex Lewandowski, Aditya Ramesh, Edan Meyer, Dale Schuurmans, Marlos C. Machado
Continual learning is often motivated by the idea, known as the big world hypothesis, that the "world is bigger" than the agent. Recent problem formulations capture this idea by explicitly constraining an agent relative to the environment. These constraints lead to solutions in which the agent continually adapts to best use its limited capacity, rather than converging to a fixed solution. However, explicit constraints can be ad hoc, difficult to incorporate, and limiting to the effectiveness of scaling up the agent's capacity. In this paper, we characterize a general problem setting in which an agent of any capacity is implicitly constrained. In particular, we consider the implicit constraint faced by an agent embedded in an environment. We introduce a universal-local environment to embed such an agent using computational universality and transition dynamics that depend on a local neighbourhood of the state-space. The embedded agent is implicitly constrained relative to its environment, represented as a partially observable Markov decision process. We then propose interactivity as a measure of an embedded agent's ability to adapt its future behaviour, conditioned on its past behaviour, using Kolmogorov complexity. We show that maximizing interactivity is a continual reinforcement learning problem, meaning that any agent that stops learning is suboptimal. To support experimentation on continual adaptation, we develop a synthetic benchmark in which an interactivity-maximizing agent constructs its own nonstationary stream of experience from which it must continually learn.