Curriculum Design for Trajectory-Constrained Agents with Applications to Compressing LLM Output Tokens
Georgios Tzannetos, Parameswaran Kamalaruban, Adish Singla
Training agents to operate under strict constraints during deployment—such as limited resource budgets or stringent safety requirements—presents significant challenges, especially when these constraints render the task complex. In this work, we propose a curriculum learning strategy that gradually tightens constraints during training, enabling the agent to incrementally master the deployment requirements. Inspired by self-paced learning techniques in unconstrained reinforcement learning (RL), our approach facilitates a smoother transition to challenging environments by initially training on simplified versions of the constraints and progressively introducing the full deployment conditions. We provide a theoretical analysis using an RL agent in a binary-tree Markov Decision Process (MDP) to demonstrate that our curriculum strategy can accelerate training relative to a baseline approach that imposes the trajectory constraints from the outset. Moreover, we empirically validate the effectiveness and generality of our method across agents—both RL and large language model (LLM) agents—and diverse settings, including a binary-tree MDP, a multi-task navigation domain, and a math reasoning task with two benchmarks. These results highlight the potential of curriculum design in enhancing the efficiency and performance of agents operating under complex trajectory constraints during deployment. Additionally, applying our strategy to LLMs, the output tokens can be compressed, achieving a 4.5× inference speedup on consumer hardware, showing its effectiveness for resource-constrained deployment.