Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
Yichen Li, Chicheng Zhang
Imitation learning (IL) is a paradigm for training sequential decision-making policies from experts, leveraging offline demonstrations, interactive annotations, or both. Recent advances show that when annotation cost is tallied per trajectory, Behavior Cloning (BC)—which relies solely on offline demonstrations—cannot be improved in general, leaving limited conditions for interactive methods such as DAgger to help. We revisit this conclusion and prove that when the annotation cost is measured per state, a metric commonly used for interactive annotation, algorithms using interactive annotations provably outperform BC.Specifically: (1) we show that a one‑sample‑per‑round DAgger variant provably beats BC and achieves up to an episode‑length improvement in state‑wise annotation cost; (2) considering the problem of hybrid IL where the agent learns from offline demonstrations and interactive annotations, we propose Warm‑start DAgger that performs no worse than using either data source alone, and can sometimes be significantly better,  through an MDP example motivated by compounding error and cold‑start problem in imitation learning practice; (3) experiments on MuJoCo continuous‑control tasks confirm that, with reasonable costs assigned to offline and interactive annotations, interactive and hybrid approaches consistently outperform BC. To the best of our knowledge, our work is the first to formally define the hybrid IL problem and to highlight the benefits of state‑wise interactive annotation.