A Principle of Pre-Strategy Intervention for Multi-Agent Reinforcement Learning
Anjie Liu, Jianhong Wang, Samuel Kaski, Jun Wang, Mengyue Yang
Guiding Cooperative Multi-Agent Reinforcement Learning (MARL) systems towards desirable outcomes are challenging, particularly when universal guidance on the desirable outcomes over the whole team is impractical. Furthermore, designing mechanisms to coordinate agents currently relies on empirical studies, lacking a unified perspective. To mitigate these issues, we introduce Multi-Agent Influence Diagrams (MAIDs) as a graphical tool to visualize existing coordination mechanisms. Based on MAIDs, we design a new coordination mechanism, referred to as targeted intervention, which is applied to only a single agent. In practice, we introduce a technique for causal inference, called pre-strategy intervention to implement the targeted intervention. Since MAIDs can be regarded as causal diagrams, the causal effect on desirable system outcomes is maximized, implying that the desirable outcomes are achieved. More importantly, the bundled relevance graph analysis from MAIDs is able to predict the solvability of coordination mechanisms via various MARL paradigms. In experiments, we demonstrate effectiveness of our proposed targeted intervention, and verify the result of relevance graph analysis.