Ground-Compose-Reinforce: Tasking Reinforcement Learning Agents through Formal Language
Andrew Li, Toryn Klassen, Andrew Wang, Parand A. Alamdari, Sheila McIlraith
Large-scale foundation models have gained broad capabilities through training on large, diverse datasets. But is diverse data necessary for broad capabilities? In this work, we explore a data-efficient approach to building up the capabilities of agents called Ground-Compose-Reinforce. We first ground a predefined set of symbols in the environment by training on a labelled trajectory dataset, which is harnessed by an agent endowed with the compositional semantics of a formal language. This agent can then be tasked through this formal language and can learn to solve any such task through a self-generated learning signal. Controlled experiments on both an image-based gridworld and a MuJoCo robotics domain show that our approach is data-efficient to train from trajectories while supporting diverse, out-of-distribution capabilities by explicitly leveraging compositionality.