Improving Regret Approximation for Unsupervised Dynamic Environment Generation
Harry Mead, Bruno Lacerda, Jakob Foerster, Nick Hawes
Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curricula remains a difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods rely on regret approximations that fail to identify challenging levels, which is compounded as the size of the environment grows. We both propose a new regret approximation, Maximised Negative Advantage (MNA), that better identifies these challenging levels and introduce Dynamic Environment Generation for UED (DEGen) to allow for UED to scale to larger environment sizes. We show empirically that MNA outperforms current regret approximations and DEGen substantially outperforms existing methods as the environment size grows.