Predictive Coding Enhances Meta-RL To Achieve Bayes-Optimal Belief Representation Under Partial Observability
Po-Chen Kuo, Han Hou, Will Dabney, Edgar Walker
Learning a compact representation of history is critical for planning and generalization in partially observable environments. While meta-reinforcement learning (RL) agents can attain near Bayes-optimal policies, they often fail to learn the compact Bayes-optimal belief states. This representational inefficiency potentially limits the agent's adaptability and generalization capacity. Inspired by predictive coding in neuroscience---which suggests the brain predicts sensory inputs as a neural implementation of Bayesian inference---and by predictive objectives in deep RL, we investigate whether integrating self-supervised predictive coding modules into meta-RL can facilitate the learning of Bayes-optimal representations. Through state machine simulation, we show that meta-RL with predictive modules consistently produces more interpretable representations that better approximate Bayes-optimal belief states compared to conventional meta-RL across a wide variety of tasks, even when both achieve optimal policies. Notably, in challenging tasks requiring active information seeking, only meta-RL with predictive modules successfully learns optimal representations and policies, whereas conventional meta-RL struggles with inadequate representation learning. Our results strongly suggest the role of predictive learning as a guiding principle for effective representation learning for both biological and artificial agents navigating partial observability.