Limit Theorems for Entropy-Regularized and Distributional Reinforcement Learning
Yash Jhaveri, Harley Wiltzer, Patrick Shafto, Marc Bellemare, David Meger
In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize *which* policies will be learned and *what* they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a *particular* optimal policy, via vanishing entropy regularization and a novel *temperature decoupling mechanism*. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects—value functions and *return distributions*. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Inspired by these insights, we provide a convergent algorithm for control in distributional RL, addressing an open problem in distributional RL theory and bringing stability to optimal return distribution estimation.