Predictive Preference Learning from Human Interventions
Haoyuan Cai, Zhenghao (Mark) Peng, Bolei Zhou
Learning from human involvement aims to incorporate the human subject to monitor and correct agent behavior errors. While learning from corrective demonstrations addresses present mistakes, it fails to avert errors in the agent’s subsequent trajectory. To address this, we introduce Predictive Preference Learning from Human Interventions (PPL), which exploits the implicit preference signals contained in human interventions over predicted future rollouts. The key idea of PPL is to convert each human intervention into contrastive preference labels over the predicted future trajectory with the length $L$, referred to as the preference horizon. By applying preference optimization on these forecasted states, expert corrections are propagated into the regions where the agent is expected to explore, significantly improving learning efficiency and reducing human demonstrations needed. We evaluate our approach with experiments on both autonomous driving and robotic manipulation benchmarks and show its efficiency and generality. Our theoretical analysis further shows that selecting an appropriate preference horizon $L$ balances coverage of risky states with label fidelity, thereby bounding the algorithmic optimality gap.