PIRO: Toward Stable Reward Learning for Inverse RL via Monotonic Policy Divergence Reduction
Yang Chen, Menglin Zou, Jiaqi Zhang, Yitan Zhang, Junyi Yang, Gaël Gendron, Libo Zhang, Jiamou Liu, Michael Witbrock
Inverse Reinforcement Learning (IRL) aims to recover reward functions that explain expert behavior, offering interpretability and generalization beyond direct imitation. However, most IRL methods suffer from unstable reward learning due to adversarial reward-policy optimization. While recent non-adversarial methods offer improved empirical stability, a principled framework for stable reward learning remains elusive. This work surrounds a key insight: stability can be enforced by construction if each reward update provably reduces the divergence between the learned and expert policies. We formalize this idea through a general reward optimization objective and show that it admits a Majorization-Minimization (MM) procedure with guaranteed monotonic reduction in policy divergence. Building on this, we propose Proximal Inverse Reward Optimization (PIRO), a practical, non-adversarial IRL algorithm that approximates the theoretical guarantee. PIRO explicitly learns reward functions, scales to high-dimensional tasks, and has strong sample efficiency on expert data, while relying on on-policy rollouts for reward updates. Empirically, across OpenAI Gym tasks and a real-world animal behavior modeling task, PIRO substantially improves training stability while matching or exceeding the performance of state-of-the-art IRL methods. This paper demonstrates that reward-driven policy alignment can provide a basis for stable IRL.