On Evaluating Policies for Robust POMDPs
Merlijn Krale, Eline Bovy, Maris Galesloot, Thiago Simão, Nils Jansen
Robust partially observable Markov decision processes (RPOMDPs) model partially observable sequential decision-making problems where an agent must be \emph{robust} against a range of dynamics. RPOMDPs can be viewed as two-player games between an agent, who picks actions, and nature, who adversarially picks dynamics. Evaluating an agent policy requires finding an adversarial nature policy, which is computationally challenging. In this paper, we advance the evaluation of agent policies for RPOMDPs in three ways. First, we discuss suitable benchmarks. We observe that for some RPOMDPs, we can find an optimal agent policy by considering only subsets of nature policies, making them easier to solve. We formalize this concept of \emph{solvability} and construct three benchmarks that are only solvable for expressive sets of nature policies. Second, we describe a provably sound method to evaluate agent policies for RPOMDPs by solving an equivalent MDP. Third, we lift two well-known POMDP upper value bounds to RPOMDPs, which can be used to efficiently approximate the optimality gap of a policy and serve as baselines. Our experimental evaluation shows that (1) our proposed benchmarks cannot be solved by assuming naive nature policies, (2) our method of evaluating policies is accurate, and (3) the approximations provide solid baselines for evaluation.