Agents Robust to Distribution Shifts Learn Causal World Models Even Under Mediation
Matteo Ceriscioli, Karthika Mohan
Recent work [Richens and Everitt, 2024] has shown that agents robust to distribution shifts learn a causal model of their environment. However, these results rely on the assumption of no mediation, i.e., that an agent's actions do not affect their environment, which can be restrictive in many real-world settings. For example, a robot in an industrial plant might interact with tools, move through space, and transform products to complete its task. In this work, we extend the theoretical foundations of robust agency by proving that agents capable of adapting to distribution shifts must learn the underlying causal relationships even in the presence of mediation. We introduce an algorithm for eliciting Causal Influence Diagrams from robust agents using optimal policy oracles, with the flexibility to incorporate prior causal knowledge and demonstrate its effectiveness in mediated single-agent scenarios and multi-agent environments. We identify conditions under which the presence of a single robust agent is sufficient to recover the full causal model and derive optimal policies for other agents in the same environment. Finally, we demonstrate how to apply these results to sequential decision-making tasks modeled as Partially Observable Markov Decision Processes (POMDPs).