\section*{Acknowledgements}
This work was partially supported by the UK Engineering and Physical Sciences Research Council (EPSRC), Omidyar Group and The Alan Turing Institute under grant agreements EP/S035761/1 and EP/N510129/1.

\section*{Broader Impact}
By making important early steps into explainability for Reinforcement Learning, this work has strong potential impact across a broad range of areas from autonomous vehicles to medical robotics.
Beneficiaries of this work include researchers and developers who wish to gain additional insight into their agents behaviour, in order to improve performance. Additionally, if accidents arise in a deployed system, our work provides a mechanism to introspect these and attempt to prevent the same thing from happening in the future.

A failure in the proposed system could lead to an incorrect intention being attributed to an agents behaviour. Our proofs show that this cannot happen for table-based Q-learning, but it is potentially possible for DQN. An incorrectly attributed intention could potentially lead to unnecessary or harmful remedial action being taken to correct the agents behaviour. In extreme cases a failure of the system may lead to accountability being incorrectly attributed. Therefore it is vital to keep in mind the limitations of the theoretical guarantees provided with this work.

Our technique does not rely on any dataset biases. However the generated explanations are specific to a single instance of an agent's policy. There are no guarantees about generalization of intention between agents, even those which are similarly trained.

\bibliographystyle{plainnat}

\section{Introduction}

Explaining the behaviour of machine learning algorithms or AI remains a key challenge in machine learning. With the guidelines of the European Union's General Data Protection Regulation (GDPR) \cite{EUdataregulations2018} calling for explainable AI, it has come to the machine learning community's attention that better understanding of black-box models is needed.
Despite substantial research in explaining the behaviour of supervised machine-learning, it
is unclear what should constitute an explanation for Reinforcement
Learning (RL). Current works in explainable reinforcement learning use similar techniques as those used to explain a supervised classifier~\cite{Mott2019}, as such their
explanations highlight what in the current environment drives an agent to
take an action, but not what the agent expects the action to achieve. Frequently, the consequences of the agent's actions are not immediate
and a chain of many decisions all contribute to a single desired outcome.

This paper addresses this problem by asking what chain of events the agent intended
to happen as a result of a particular action choice.
The importance of such explanations based around intended outcome in day-to-day
life is well-known in psychology with Malle~\cite{Malle2001} estimating that
around 70\% of these day-to-day explanations are intent-based. While the notion
of intent makes little sense in the context of supervised classification, it is
directly applicable to agent-based reasoning, and it is perhaps surprising that
we are the first work in explainable RL to directly address this. Recent work~\cite{langley2019explainable} has called for these introspective abilities which they refer to as ``Explainable Agency''.

 We present a simple addition to standard value-based RL
 frameworks which allows us to obtain a projection of predicted future
 trajectories from a current observation and proposed action. Unlike existing approaches such as \cite{Waa2018Contrastive} that predict an agent's
 future states by rerunning repeated forward simulations from the same initial
 state; we instead recover sum of the past events, weighted by the importance
 that the agent put on them when learning a Q-values, that lead to the
 agents current behaviour, and mathematically guarantee that this sum is
 consistent with the agent's Q-values. This allows for local interpretation of the agent's intention based on its behavioural
 policy.

 We mathematically guarantee that this is
 consistent with the agent's Q-values (see figure \ref{fig:teaser}). We hope our method will help RL practitioners trying to understand agent behaviour for model debugging and diagnostics, and potentially lead to more reliable and trustworthy RL systems.

 Mismatches between learnt Q-values and the forward simulations of~\cite{Waa2018Contrastive} exist for
 a variety of reasons: stochasticity of the environment; mismatches between training and test environments\footnote{This
   is particularly common in multi-agent systems, where the behaviour of other
   agents can change.}; systematic biases in the learning
 method~\cite{hasselt2010double}

and the use of
 replay buffers (particularly hindsight experience replay (HER)~\cite{andrychowicz2018hindsight}).

 Our
 representations highlight what is learnt by the agent at train time,
 rather than the behaviour of the agent at test time, allowing them to provide
 insights into some of these failures.

\begin{figure}
\hspace{-7mm}
\begin{tabular}{p{0.25\linewidth}p{0.25\linewidth}|p{0.25\linewidth}p{0.25\linewidth}}
  \includegraphics[width=\linewidth]{supp_figs/blackjack/td/forward_sim_orig.png}&
\includegraphics[width=\linewidth]{supp_figs/blackjack/td/future_hit_orig.png}&
\includegraphics[width=\linewidth]{supp_figs/blackjack/td/forward_sim_underperform.png}&
 \includegraphics[width=\linewidth]{supp_figs/blackjack/td/future_hit_underperform.png}
\end{tabular}

\hspace{-2.75mm}
\begin{tabular} {p{0.5\linewidth} | p{0.5\linewidth}}
\begin{center}
\vspace{-5mm}
\includegraphics[width=0.6\linewidth]{supp_figs/blackjack/td/belief_hit_orig.png}
\end{center}
&
\begin{center}
\vspace{-5mm}
\includegraphics[width=0.6\linewidth]{supp_figs/blackjack/td/belief_hit_underperform.png}
\end{center}
\end{tabular}
\caption{Understanding an agent's expected behaviour.
  These images show predictions of an agent's behaviour in blackjack.
  All plots show an initial
  state where the dealer shows 5, and the player's cards sum to 7 (no high
  aces), and the colour of each cell shows the expected number of times a state will
  be visited, if the agent hits.
  The two leftmost plots show the predicted behaviour, based
  on forward simulation \cite{Waa2018Contrastive}, and `belief' of a well-trained agent, while the two rightmost show the predicted behaviour and `belief' of an underperforming agent
  trained on a small and fixed replay buffer. The bottom two images show the final outcomes the well-trained and underperforming agents believe will occur if they hit. This results in a conservative agent that sticks early owing to an increased belief that they will go bust from a hit.
  \label{fig:teaser}
}
\end{figure}

\end{document}
\section{Related Works}
Many approaches address
interpretability for supervised machine learning. Traditional model-based
machine-learning algorithms, of restricted capacity, such as decision trees
\cite{Quinlan1986decisiontree}, GLM/GAMs \cite{Hastie1986} and RuleFit
\cite{friedman2008} are considered interpretable or intrinsically explainable
due to their simple nature~\cite{rudin2019stop}. However, the dominance of model-free
algorithms and deep learning means that most approaches are not considered intrinsically
explainable due to their high complexity. Unavoidably, a trade-off
between interpretability and performance exists, and instead focus has switched to
local, post-hoc methods of explanation that give insight into complex classifiers.
Two of the largest families of explanations are: \emph{(i)} perturbation-based attribution methods
\cite{Ribeiro2016, LundbergNIPS2017SHAP} that systematically alter input
features, and examine the changes
in the classifier output. This makes it possible to build a local surrogate model that provides
an local importance weight for each input feature. \emph{(ii)}
gradient-based attribution methods \cite{shrikumar2017learning,
  sundararajan2017axiomatic, bach-plos15,SelvarajuDVCPB16,Samek2017}. Here the attribution of input
features is computed in a single forward and backward pass, with the attribution
comprising a derivative of output against input. Fundamentally both approaches
use measures of feature importance to build a
low-complexity model that locally approximates the underlying model. More recently self-explainable neural networks (SENNs) \cite{alvarezmelis2018robust, Teso2019TowardFE, alshedivat2020contextual} are end-to-end models that produce explanations of their own predictions.

Despite recent progress in reinforcement learning, few works have focused on its explainability. \citet{Annasamy2019IDQN} proposed Interpretable Deep Q Network (i-DQN), an architecturally similar model to the vanilla DQN model \cite{mnih2015humanlevel} that introduced a key-value store to concurrently learn latent state representation, and Q-value mapping for intuitive visualisations by means of key clustering, saliency and attention maps. \citet{Mott2019} takes this idea further, representing the key-value store as a recurrent attention model to produce a normalised soft-attention map which tracks the steps an agent takes when solving a task. Both methods focused on identifying components of the world that drives the agent to take a particular action, rather than identifying the agents expected outcomes.
As these works exploit visual attention they are only directly
applicable to image-based RL problems.

Some approaches extract global policy summaries i.e. compressed approximations of
the underlying policy function.
 \citet{Verma18} proposed Programatically Interpretable Reinforcement Learning to syntactically express policy in a high-level programming language. \citet{Topin2019} modeled policy as a Markov chain over abstract states giving high-level explanations.
Our approach gives a local description that says given a current state/action choice, what are the expected future states

\citet{Juozapaitis2019} proposed a decomposition of the scalar reward function into a vector-valued reward function for each sub-reward type.
When used together for qualitative analysis, these provide a compact explanation of why a particular action is preferred over another, in terms of which sub-elements of the reward are expected to change. Our work differs in that we propose a decomposition of the Q-function over state and action space, rather than over reward sub-elements. This makes it possible for us to reason about intended future events in our explanations.  Our approach is complementary to \cite{Juozapaitis2019}; it is possible to decompose the Q-value both across state space and sub-rewards, giving detailed reasons of
 sub-rewards over future states

 Another work closely related to our approach is successor representation (SR)
 learning \cite{dayan1993improving}.
 The SR bears some similarities to our Q-value decomposition. However, this is used specifically to support learning in non-stationary environments with distinct properties. In contrast

 we are
 interested in generating successor representations which are guaranteed to be consistent
 with value functions, in particular Q-learning, Monte Carlo Control, and double Q-learning.

\end{document}
\section{Background}
We  set out the minimal background  of  reinforcement learning
necessary to define our approach to explainable  Reinforcement Learning.
We formalise a reinforcement learning environment as a Markov decision
process (MDP) \cite{SuttonBartoBook}. This MDP consists of a 5-tuple $\langle
\mathcal{S}, \mathcal{A}, p, r, \gamma \rangle$, where $\mathcal{S}$ is the set
of all possible environment states, $\mathcal{A}$ is the set of actions,
$p(s_{t+1}|s_t, a_t)$ is the transitional dynamics of the MDP, $\gamma \in [0,
1]$ is the discount factor and $r(s_{t}, a_{t}, s_{t+1}) \in \mathbb{R}$ is the
reward for executing action $a_t$ in state $s_t$ arriving at state $s_{t+1}$.
Reinforcement learning seeks to find the optimal policy $\pi^*: \mathcal{S}
\rightarrow \mathcal{A}$ that maximise the expectation of cumulative discounted
rewards over the MDP.
\begin{equation}
\pi^*(s) = \argmax_{\pi} \E\left[\left.\sum^\infty_{t=0} \gamma^{t} r(s_t, a_t, s_{t+1}) \right| s_t = s, a_t = a, \pi\right].
\end{equation}
Many RL approaches estimate the state-action value function
$Q^\pi(s_t,a_t)$.\footnote{The reasoning in this paper holds true for estimation of
  state value function $V^\pi (s_t)$ as well, but for conciseness we focus on Q-functions.}
\begin{equation}
Q^{\pi}(s_t, a_t) = \E_{\pi} \left[\left. \sum^{\infty}_{t=0} \gamma^t r_{t+1} \right| s_t = s, a_t = a \right].
\label{eq:qvalue}
\end{equation}
Where we write  $r_{t+1}$ as shorthand for $r(s_t, a_t, s_{t+1})$.
Both value functions satisfy the Bellman equation, and are frequently estimated by Q-learning or Monte Carlo (MC) like methods.
\begin{align}
Q(s_t, a_t) \leftarrow&
	Q(s_t, a_t) + \alpha \left(r + \gamma \max_{a \in \mathcal{A}} Q(s_{t+1}, a) - Q(s_t, a_t)\right) & \text{Q-learning}
\label{eq:Q-update}\\
Q(s_t, a_t) \leftarrow&
	Q(s_t, a_t) + \alpha\left(\sum^{T}_{t'=t} \gamma^{t'-t} r_{t'} - Q(s_t, a_t)\right)  &\text{Monte Carlo}
\label{eq:MC-update}
\end{align}
In Q-value-based deep reinforcement learning \cite{mnih2015humanlevel}, the Q-function is parametrised by network parameters $\theta$. A policy is implicitly learned by computing the gradient of a loss function with experiences $e_t = (s_t, a_t, s_{t+1}, r_t)$ drawn from an experience replay $D_t = \{e_1, ..., e_t\}$. Using $\theta$ and $\theta^-$ for the behavioural network and target network parameters and assuming an $L_2$ loss, the loss function can then be written as
\begin{equation}
L(\theta) = \E_{e \sim D_t }\left[ \left(r + \gamma \max_{a}Q(s_{t+1}, a;\theta) - Q(s_t, a_t;\theta^-) \right) ^2 \right],
\label{eq:dqn-loss}
\end{equation}
The update equation is then simply:
\begin{equation}
\theta \leftarrow \theta + \alpha \nabla_\theta L(\theta).
\label{eq:update}
\end{equation}

\end{document}

\section{Value Function Decomposition}
To understand why an RL agent prefers one action over another,  we want to infer the agent's implicit estimate of $p(s_{t+n}|s_{t},a_{t},\pi)$ for all $n>0$ which reveals the future states used in estimating Q-values.

We do not try to understand the internal computations that led our
agent to map a particular Q-value against a particular action. Instead we wish
to know the expected future states that leads an agent to  decide that
one action in the current state would be preferable to another action.

We prove that post-hoc explanations of the form we are interested in can
not be recovered from an existing agent trained using a traditional Q-learning approach;
we  do this by showing that the inverse problem of recovering expected
future states from a Q-function is fundamentally ill-posed.
 Note that Q-learning is used only as an example. The same reasoning can be
 applied to any value-based reinforcement learning algorithm.

 We then derive a new algorithm that can be run alongside with an existing learning process which lets us obtain such explanations, without altering what is learnt.

\begin{figure}
	\centering
	\includegraphics[width=0.55\textwidth]{./tikz/mdp-figure0.pdf}
	\caption{A simple MDP to illustrate value ambiguity in value iteration methods (see Section \ref{sec:vfd-motivation}).}
	\label{fig:mdp}
\end{figure}

\subsection{Value Ambiguity}
\label{sec:vfd-motivation}
The Q-value function is a convenient tool for producing a loss that directs an agent's behaviour. However, it acts as a bottleneck on the information exposed. It quantifies the ``goodness'' (expectation of discounted rewards) for executing a given action $a$ in state $s$, without explicitly capturing \textit{how} or \textit{why}  such a conclusion is reached.
Furthermore, not only is the required information not directly available, but we also cannot in general, infer it indirectly from the Q-value.
\begin{theorem}
\label{th:value_ambig}
Given any MDP where multiple optimal policies $\pi^*$ exist, it is not possible to produce a post-hoc interpretation due to value ambiguity.
\end{theorem}

The proof follows immediately by contradiction.
Consider the undiscounted, deterministic chain of MDP in Figure \ref{fig:mdp},
where an episode starts at $s_0$. Each state has 2 actions: $a_0$ moves to the
upper node, $a_1$ moves to the lower node. All rewards are $0$ except for
reaching the end of the chain, where $r(s_1, a_0) = r(s_2, a_1) = 2$, $r(s_1,
a_1) = r(s_2, a_0) = 1$. We assume a tabular Q-learning agent trained using the
$\epsilon$-greedy algorithm (with ties broken arbitrarily).

While the value functions are updated and eventually converge to optimality,
there is ambiguity while learning takes place. Clearly, $Q^*(s_0,
a_0) = Q^*(s_0, a_1) = 2$ and there are two optimal trajectories
$\tau^1_{t:T} = (s_0, a_0, 0, s_1, a_0, 2)$ or $\tau^2_{t:T} = (s_1, a_1, 0,
s_2, a_1, 2)$ relating to two separate policies $\pi^*$.

Clearly, in this situation it is not possible to examine a single value estimate
$V^{\pi}(s_0)$ and infer which trajectory the agent expects to
follow. Therefore we can not
 guarantee that any post-hoc interpretation is correct in this
scenario, or in the general case. \qed

Although this is a simple example, it immediately rules out approaches to
generating explanations by training an ML method to predict to future behaviour
of an agent on the basis of the behaviour of previous agents with similar
Q-values.
The problem illustrated by our counterexample is exacerbated if we work with an
extended chain of conceptually similar MDPs, as $s_3$ and $s_6$ could
both contribute to the estimation of the same Q-value. Similarly, the same
situation is observed for $Q^*(s_1, a_0)$ and $Q^*(s_2, a_1)$. More
specifically, it is impossible to answer the contrastive question of ``What different future states are expected to occur if one action is selected over another?''.

\subsection{Explaining Classical Reinforcement Learning}
\label{sec:tabular}
We describe how to augment standard RL learning methods with a simple additional process that allows us to learn a map of discounted expected states $\mathbf{H}$ (which we refer to as the \textit{belief map})
concurrently with learning the Q-value function.

The intuition behind our approach is straightforward. While Q-value based methods \eqref{eq:Q-update} estimate the expected future reward summed over all discounted expected states, our approach preserves additional information, and captures the discounted expected state at the same time. By choosing update rules for $\mathbf{H}$ to match the update equations of \eqref{eq:Q-belief-update} we guarantee consistency between the Q-values and the expected future states.
Subsection~\ref{sec:consistency} proves that these expected states are consistent with the  learnt Q-values.
We present three variations of value function decomposition. For simplicity, we assume the methods are off-policy, but our approach is directly applicable to on-policy methods.

We define $\mathbf{H}(s_t,a_t) \in \mathbb{R}^{S \times A}$ as the expected discounted sum of future state visitations in a deterministic MDP which starts by executing action $a_t$ from state $s$. As a notational convenience we use $1_{s_t, a_t} \in \mathbb{R}^{S \times A}$ to denote a binary indicator function  $1$ at position $s_t,a_t$ and $0$ elsewhere.

\paragraph{Q-Learning}
Every time the Q-value estimator is updated during learning, we update the corresponding entries of $\mathbf{H}$ to maintain consistency.
This is a direct adaption of Equation \ref{eq:Q-update} for application to $\mathbf{H}$, that gives the update rule:
\begin{equation}
\mathbf{H}(s_t, a_t) \leftarrow \mathbf{H}(s_t, a_t) + \alpha \left( 1_{s_t, a_t} + \gamma \mathbf{H}(s_{t+1}, \argmax_{a \in \mathcal{A}} Q(s_{t+1}, a) - \mathbf{H}(s_t, a_t) \right).
\label{eq:Q-belief-update}
\end{equation}

We update all state-action pairs in trajectory $\tau$.

\paragraph{Monte Carlo Control}
Adapting the update to Monte Carlo control methods is straightforward. We modify Equation \ref{eq:Q-update} by replacing $1_{s_t, a_t}$ with a vector sum of discounted state visitations
\begin{equation}
\mathbf{H}(s_t, a_t) \leftarrow \mathbf{H}(s_t, a_t) + \alpha \left( \sum_{t=t'}^T \gamma^{t} 1_{s_t, a_t} - \mathbf{H}(s_t, a_t) \right).
\end{equation}

In tabular settings, the equality holds when both the belief map $\mathbf{H}$ and Q-table $Q$ are zero-initialised. Under these constraints, the recovery of value function is guaranteed. If $\mathbf{H}$ is randomly initialised, the equality does not always hold, but will eventually converge to the true state visitations count, leading to an accurate representation of the agent's behaviour.
\subsection{The Consistency of Belief Maps}
\label{sec:consistency}
We say that a belief map is consistent with a Q-table given reward map $\mathbf{R}$, if the inner product of the belief map of every state-action pair with the reward map  gives the Q-value for the same state-action pair.
More formally,  we assume the current reward is a deterministic function of the current state-action pair,
we define $\mathbf{R} \in \mathbb{R}^{S \times A}$ as a map of rewards for every state-action pair in the MDP, and say that if
\begin{equation}\label{eq:consist}

    \mathrm{vec}(\mathbf{H}(s, a))^{\top} \mathrm{vec}(\mathbf{R}) = Q(s, a) \quad \forall a\in {\cal A}, s\in {\cal S},
\end{equation}
then the belief map $\mathbf{H}$ is consistent with $Q$.
\begin{theorem}
\label{th:belief_map_consistency}
If $\mathbf{H}$ and $Q$ are zero-initialized,
then $\mathbf{H}$ and $Q$ will be consistent for all iterations of the algorithm.
\label{th:vfd}
\end{theorem}
\begin{proof}
Proof follows by induction on $i$, the iteration of the learning algorithm.

\textbf{Base case} ($i=0$): At initialisation ${\bf H}_i(s, a)={\bf 0}$ and $Q_i(s ,a)=0$ for all possible $s$ and $a$ and the statement is trivially true.

\textbf{Inductive step (Q-learning)}: We consider iteration $i$ where we know that the theorem holds for iteration $i-1$. We write $m_t=\argmax\limits_{a \in \mathcal{A}} Q_{i-1}(s_t,a_t)$
\begin{align*}
\mathrm{vec}(\mathbf{H}_i(s_t, a_t))^\top \mathrm{vec}(\mathbf{R}) &= \mathrm{vec}\left( (1-\alpha) \mathbf{H}_{i-1}(s_t, a_t) + \alpha \left( 1_{s_t, a_t} + \gamma \mathbf{H}_{i-1}(s_{t+1}, m_{t+1}) \right)\right) ^\top \mathrm{vec}(\mathbf{R}) \\
& = (1-\alpha) Q_{i-1}(s_t, a_t) + \alpha \mathrm{vec}\left( 1_{s_t, a_t} + \gamma \mathbf{H}_{i-1}(s_{t+1}, m_{t+1}) \right) ^\top \mathrm{vec}(\mathbf{R})\\
& = (1-\alpha) Q_{i-1}(s_t, a_t) + \alpha  r_{t}+ \alpha  Q_{i-1}(s_{t+1}, m_{t+1}) \\
&= Q_{i}(s_t, a_t).
\end{align*}

\textbf{Inductive step (Monte Carlo)}: We consider iteration $i$ where we know that the theorem holds for iteration $i-1$.
We use $\mathbf{H}_i$ to indicate the state of $\mathbf{H}$ at iteration $i$, and $Q_i$ the state of $Q$ at iteration $i$
\begin{align*}
\mathrm{vec}(\mathbf{H}_i(s_t, a_t))^\top \mathrm{vec}(\mathbf{R}) &= \mathrm{vec}\left((1-\alpha) \mathbf{H}_{i-1}(s_t, a_t) + \alpha \sum_{t=t'}r^t \gamma^{t} 1_{s_t, a_t}\right) ^\top \mathrm{vec}(\mathbf{R})\\
& = (1-\alpha) Q_{i-1}(s_t, a_t) + \alpha \mathrm{vec}\left(\sum_{t=t'}^T \gamma^{t} 1_{s_t, a_t}\right) ^\top \mathrm{vec}(\mathbf{R})\\
& = (1-\alpha) Q_{i-1}(s_t, a_t) + \alpha \sum^{T}_{t=t'} \gamma^t r_{t}\\
&= Q_{i}(s_t, a_t).
\end{align*}
as required. See supplementary materials for the result for double Q-learning.
\end{proof}

\subsection{Application to Deep Q-Learning}
Much like Deep Q-learning, function approximation can be used to estimate $\mathbf{H}$. For $\mathbf{H}$ parametrised by $\theta_h$ and given agent policy $\pi_\theta$, the loss function in Equation \eqref{eq:dqn-loss} becomes:
\begin{equation}
L_{h}(\theta_h) = \displaystyle \mathop{\mathbb{E}}_{(s_t, a_t, s_{t+1})}\!\left(1_{s_t, a_t} \!+\! \gamma \mathbf{H}\left(s_t, \argmax_{a \in \mathcal{A}}Q(s_{t+1}, a;\theta) ; \theta^-_h\right) - \mathbf{H}(s_t, a_t;\theta_h)\right)^2 .
\label{eq:L}
\end{equation}
Then the update rule in Equation \ref{eq:update} becomes:
\begin{equation}
\theta_{h} \leftarrow\theta_{h} + \alpha \nabla_{\theta_h} L(\theta_{h}).
\end{equation}
Here $\mathbf{H}(s_t,a_t)$ is an unnormalised density estimation. Where the state and action spaces are discrete, this can be in the form of a vector output, and where at least one is continuous using techniques such as~\cite{de2019block}, alternatively it can be approximated by pre-quantizing the state space using standard clustering techniques.

The mathematical guarantees of consistency from the previous section do not hold for deep Q-learning, with the random initialisation of the networks meaning that it fails in the base case. However, just as an appropriately set up deep Q-learning algorithm will converge to an empiric minimiser of equation \eqref{eq:dqn-loss}, an appropriately setup H-learner will also converge to minimise equation \eqref{eq:L}. We evaluate the deep variants of our approach in the experimental section, and show even without theoretical guarantees, it performs effectively, providing insight into what has been learnt by our agents.

\end{document}
\section{Experiments and Discussions}
\label{sec:experiments}
We evaluate our approach on three standard environments using OpenAI Gym \cite{OpenAIGym} - Blackjack, Cartpole \cite{Barto1990Cartpole} and Taxi \cite{Dietterich2000Taxi}. Each environments poses unique challenges. For each task we briefly describe our assumptions and key experiment parameters. We extract an intuitive local interpretation of the RL agent's intention solely from the belief maps. We verify the correctness of our implementation in each environment using Theorem~\ref{th:belief_map_consistency}, and confirm equation \eqref{eq:consist} holds numerically

We find minimal differences applying our technique to Temporal Difference and Monte Carlo approaches. For conciseness, we report here the findings from TD and the supplementary material contain MC results. RL agents always use an $\epsilon$-greedy algorithm during training.

\paragraph{Blackjack}
\begin{figure}
	\centering
	\begin{subfigure}[b]{0.23\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/bj_10,7,False_stick.png}
		\caption{Belief map for stick.}
		\label{subfig:h-stick}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.23\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/bj_10,7,False_hit.png}
		\caption{Belief map for hit.}
		\label{subfig:h-hit}
	\end{subfigure}
	~
	\begin{minipage}[b][0.25\textheight][s]{0.3\textwidth}
		\begin{subfigure}[b]{\textwidth}
			\centering
			\includegraphics[width=\textwidth]{supp_figs/blackjack/td/accum_bj_10,7,False_stick.png}
			\caption{Outcome belief map for stick.}
			\label{subfig:inter-stick}
		\end{subfigure}
		\vfill
		\begin{subfigure}[b]{\textwidth}
			\centering
			\includegraphics[width=\textwidth]{supp_figs/blackjack/td/accum_bj_10,7,False_hit.png}
			\caption{Outcome belief map for hit.}
			\label{subfig:inter-hit}
		\end{subfigure}
	\end{minipage}
	\caption{Visualisations of belief maps for actions when player sum $=10$, dealer card $=7$ with no usable ace. Unreachable player states are cropped for conciseness, i.e. Player sum $< 4$ or sum $> 21$. }
	\label{fig:bj-belief-maps}

\end{figure}

We model a simplified game of blackjack and assume (1) the shoe consists of infinite cards, (2) the only moves allowed are ``hit''
and ``stick''
and (3) we do not account for blackjacks.

The system dynamics are non-deterministic as the card drawn on each hit is random, and while the reward is stochastic due to the uncertainty in the dealer's hand. To make the reward a deterministic function of the stochastic state  we create four additional states: hit and bust; stuck and won; stuck and drew; stuck and lost.
The agent is trained with $\alpha = 0.1, \gamma = 1$ for $500k$ episodes.
Monte-Carlo learning performed similarly to table-based Q-learning. See the supplementary results for details.

Figure \ref{fig:bj-belief-maps} shows belief maps for each action and the corresponding incurred rewards when the player's hand has a sum of 10, and the dealer shows 7. There is a single point in the state space that shows a high probability in both figure \ref{subfig:h-stick} and \ref{subfig:h-hit}. This point corresponds to the current starting state, which all future trajectories must pass through. Figure \ref{subfig:h-hit} shows  probable future events when the player hits. We observe the intention of the agent from the belief map, that it continues to hit until a high enough sum is obtained. This explains why lower values have lower density, as there are fewer paths to reach these states. The second peak at 20 is expected given that all face cards have a value of 10.

\paragraph{Cartpole}

\begin{figure}
	\begin{tabular}{p{0.5\textwidth}p{0.5\textwidth}}
	\begin{subfigure}[b]{0.5\textwidth}
		\centering
		\includegraphics[width=\textwidth]{figures/cp_q_belief_s79_a0.png}
		\caption{Belief map for Q-Learning}
		\label{subfig:cartpole-qh}
	\end{subfigure}&
	\begin{subfigure}[b]{0.5\textwidth}
		\centering
		\includegraphics[width=\textwidth]{figures/cp_q_trajectory.png}
		\caption{Forward Simulation~\cite{Waa2018Contrastive} for Q-learning}
		\label{subfig:cartpole-q-trajectory}
	\end{subfigure}\\
	\begin{subfigure}[b]{0.5\textwidth}
		\centering
		\includegraphics[width=\textwidth]{figures/cp_dqn_belief_s79_a1.png}
		\caption{Belief map for deep Q-Learning}
		\label{subfig:cartpole-dqnh}
	\end{subfigure}&
	\begin{subfigure}[b]{0.5\textwidth}
		\centering
		\includegraphics[width=\textwidth]{figures/cp_dqn_trajectory.png}
		\caption{Forward Simulation~\cite{Waa2018Contrastive} for deep Q-learning}
		\label{subfig:cartpole-dqn-trajectory}
	\end{subfigure}
	\end{tabular}
	\caption{Belief maps and trajectory visualisations for cartpole simulation. Only states where the cart is in the middle are shown as it rarely drifts away to extreme positions on the left or right. A contrastive versions showing how expected state varies with action choice can be seen in the supplementary materials.}
	\label{fig:cartpole-h}

\end{figure}

\cite{Barto1990Cartpole} is a classic continuous control problem with a 4-tuple continuous state space $\langle x, \dot{x}, \theta, \dot{\theta} \rangle$ : cart position, cart velocity, pole angle and pole velocity, and a discrete action space consisting of 2 actions: left and right. For a like-with-like comparison across standard learning approaches, we
discretized the state-space
into bins of $(3, 3, 6, 3)$.

We investigate agent intention in cartpole learnt from vanilla Q-learning and DQN. In both methods, we train the agent until it is reasonably stable. The vanilla Q-learning agent is trained with learning rate $\alpha=0.1$ and $\gamma = 1$. The DQN agent is trained using Adam \cite{kingma2014adam} with gradient clipping at $[-1, 1]$ and $\alpha=0.0001, \gamma = 1$, while the belief map is  generated by a deep network which we refer to as a \textit{Deep Belief Network} (DBN) trained using the same hyperparameters.

Figure \ref{fig:cartpole-h} shows the belief map versus the actual trajectory of the episode produced by forward simulation of the agent. The mismatch between forward simulation and expected states is particularly interesting. We believe this is because cartpole is chaotic but deterministic. Rerunning the same forward simulation results in exactly the same behaviour each time, however, as the agent sees a quantised approximation of the initial state, it has much greater uncertainty regarding future states.

\paragraph{Taxi}
\begin{figure*}
	\centering
		\hsize=\textwidth
		\begin{subfigure}[b]{\hsize}
			\centering
			\includegraphics[width=\hsize]{figures/merged_9_qtaxi.png}
		\end{subfigure}
		~
		\begin{subfigure}[b]{\hsize}
			\centering
			\includegraphics[width=0.66\hsize]{figures/merged_14_qtaxi.png}

		\end{subfigure}
	\caption{Varying intention of the Q-learning agent in the Taxi environment \cite{Dietterich2000Taxi}. Each image represents the belief map at one point along a trajectory, with time advancing from left to right.
	Colour intensity denotes the confidence that the agent will visit a state.
	Yellow indicates the passenger is not present, whereas green means they are. `R', `G', `Y', `B', each represents a possible location of the passenger and the destination. Blue text indicates the location of the passenger if they are not in the car, and red text denotes destination.}
	\label{fig:taxi-h}
\end{figure*}

\begin{figure}
	\centering
		\hsize=\textwidth
		\begin{subfigure}[b]{\hsize}
			\centering
			\includegraphics[width=\hsize]{figures/merged_9_dqntaxi.png}
		\end{subfigure}\vfill
		~
		\begin{subfigure}[b]{0.6\hsize}
			\includegraphics[width=\hsize]{figures/merged_14_dqntaxi.png}
		\end{subfigure}
	\caption{How the belief maps of the DQN agent vary over time in the Taxi \cite{Dietterich2000Taxi}.}
	\label{fig:taxi-dqnh}
\end{figure}

\cite{Dietterich2000Taxi} is a challenging environment where an agent must first navigate from a randomised start point, around obstacles, to collect a passenger from a random location, then pick up that passenger and navigate to a randomised destination.
We investigate agent intention learnt by Q-learning and DQN. The Q-learning agent is trained with $\alpha = 0.4, \gamma = 1$ using a $4$-tuple state representation of car x and y location, passenger location and destination. DQN used $\alpha=0.0001, \gamma = 1$ for $100k$ episodes.\looseness=-1

Figure \ref{fig:taxi-h} and Figure \ref{fig:taxi-dqnh} shows visualisations of the RL agents' belief maps at several steps along a trajectory (animations are available in the supplementary material). For both methods we can intuitively reason about the intentions of the agents. These intentions become clearer over time, as the multiple potential routes (including the outward and return journey) which are overlaid, collapse to the single shortest path towards the goal.

\begin{figure}
	\centering
	\hsize=0.95\textwidth
	\begin{subfigure}[b]{\hsize}
		\centering
		\includegraphics[width=0.6\hsize]{figures/contrast_009.png}
		\caption{$a^0$: move east, $a^1$: move south. $Q(s, a^0) = 7.7147$, $Q(s, a^1) = 7.0777$.}
		\label{subfig:contrastive-value-ambiguity}
	\end{subfigure}
	~
	\begin{subfigure}[b]{\hsize}
		\centering
		\includegraphics[width=0.6\hsize]{figures/contrast_011.png}
		\caption{$a^0$: move south, $a^1$: move east. $Q(s, a^0) = 11.87$, $Q(s, a^1) = 7.6599$.}
		\label{subfig:contrastive-bounce-back}
	\end{subfigure}
	~
	\begin{subfigure}[b]{\hsize}
		\centering
		\includegraphics[width=0.6\hsize]{figures/contrast_001.png}
		\caption{$a^0$: move north, $a^1$: move east. $Q(s, a^0) = -2.3744$, $Q(s, a^1) = -5.204$.}
		\label{subfig:contrastive-failure}
	\end{subfigure}
	\caption{Contrastive explanations of taxi tabular Q-learning agent during the episode in Figure \ref{fig:taxi-h}. Here we denote $a^0$ as the best action, $a^1$ as the second best action.
	Current state is highlighted by the strong hue of the car. From left to right we have: (1) $\mathbf{H}(s, a^0)$, (2), $\mathbf{H}(s, a^1)$, (3) $\min(0, \mathbf{H}(s, a^1) - \mathbf{H}(s, a^0))$, (4) $\min(0, \mathbf{H}(s, a^0) - \mathbf{H}(s, a^1))$, (5) the intersection of the two expected outcomes, given by $\min(\mathbf{H}(s, a^0),\mathbf{H}(s, a^1))$.}
	\label{fig:taxi-q-contrastive}
\end{figure}

\paragraph{Contrastive Explanations}
Figure \ref{fig:taxi-q-contrastive} demonstrates the capability of our method to extract contrastive explanations. We ask "What change in future states does the agent believe will happen when one action is selected over another?" To generate these explanations, we scale the belief map to $[0, 1]$. Figure \ref{subfig:contrastive-value-ambiguity}  shows a real example of the value ambiguity problem of Theorem 1. By contrasting the belief maps associated with the best and second best actions we see two distinct but equally good policies were discovered.
Such explanations is not possible with vanilla Q-learning
We also identify a failure case in Figure \ref{subfig:contrastive-failure} where taking the suboptimal action would cause the car to be stuck in the same state. Even for DQN, although the mathematical guarantee is lost, we extract an approximation of the agent's intention; see supplementary material for contrastive explanations for the DQN agent.

\end{document}
\section{Conclusions and Future Work}
We have proposed a novel approach for explaining what outcomes are implicitly expected by reinforcement learning agents.
We proposed a meaningful definition of intention for RL agents and proved no post-hoc method could generate such explanations. We proposed modifications of standard learning methods that generate such explanations for existing RL approaches, and proved consistency of our approaches with tabular methods. We further showed how it can be extended to deep RL techniques and demonstrated its effectiveness on multiple reinforcement learning problems. Finally we provided code to allow future researchers to introspect their own RL environments and agents.\looseness=-1

A noticeable limitation of our method is that it is best suited for problems where tabular reinforcement learning works well (i.e. low-dimensional and easily visualisable). Just as deep learning substantially increased the applicability of reinforcement learning it is interesting to ask how it could increase the applicability of our approach. One potential answer lies in the use of concept activation vectors \cite{kim2017interpretability}, which allows for shared concepts between humans and machine learning algorithms.  For example, if a concept was trained to recognise a pinned piece in chess our approach would be able to explain a move by saying that it will give rise to a pinned knight later. Importantly, given a set of concepts $F$ defined over the state space ${\cal S}$, it would be possible to train the belief map over the low-dimensional space $F({\cal S})$, rather than ${\cal S}$, substantially improving the scalability of our approach. As such, while our approach is directly applicable to problems with easily mappable state-spaces, it also lays the mathematical foundations for explainable agents in more complex systems.

\end{document}
\section{Double Q-learning}
\label{sec:double-q-learning}
Double Q-learning\cite{hasselt2010double} is a modification of Q-learning which  maintains two Q-tables making it is less prone to over-estimation. After each episode, one of the Q-tables ($Q^A$ or $Q^B$) is
randomly updated, using one of the following two equations.
\begin{align*}
Q^{A}_{i}(s_{t},a_{t})&\leftarrow(1-\alpha)Q^{A}(s_{t},a_{t})+\alpha\left(r_{t}+\gamma Q^{B}_{i-1}\left(s_{t+1},\arg\max_{a}Q^{A}_{i-1}(s_{t+1},a)\right)\right)\\
  Q^{B}_{i}(s_{t},a_{t})&\leftarrow(1-\alpha)Q^{B}(s_{t},a_{t})+\alpha\left(r_{t}+\gamma Q^{A}_{i-1}\left(s_{t+1},\arg\max_{a}Q^{B}_{i-1}(s_{t+1},a)\right)\right)
\end{align*}
\paragraph{Belief Map Update Step} Similarly, we maintain two belief maps (${\bf H}_A$
and ${\bf H}_B$)  and
update them in sync with the Q-tables, that is each time $Q_A$ is updated, we
also update ${\bf H}_A$ and the same for $Q_B$ and ${\bf H}_B$. The updates are given by the
following two equations
\begin{align*}
 {\bf H}^{A}_{i}(s_{t},a_{t})&\leftarrow(1-\alpha){\bf H}^{A}(s_{t},a_{t})+\alpha\left({1_{s_{t},a_{t}}}+\gamma {\bf H}^{B}\left(s_{t+1},\arg\max_{a}Q^{A}(s_{t+1},a)\right)\right)\\
 {\bf H}^{B}_{i}(s_{t},a_{t})&\leftarrow(1-\alpha){\bf H}^{B}(s_{t},a_{t})+\alpha\left({1_{s_{t},a_{t}}}+\gamma {\bf H}^{A}\left(s_{t+1},\arg\max_{a}Q^{B}(s_{t+1},a)\right)\right)
\end{align*}
\paragraph{Proof of Consistency}
\begin{proof}
The proof is almost identical to that of Q-learning in the main body of the paper. In the base case, all $Q$
and ${\bf H}$ are zero initialised, and therefore consistent. For the inductive step,
after each episode, one of $A$ or $B$ is updated, and we assume that both $Q^A_{i-1}$
and $Q^B_{i-1}$ are consistent with ${\bf H}^A_{i-1}$ and ${\bf H}^B_{i-1}$ . Without loss of generality, we
assume   $A$ is selected: We consider time $i$ and we write $m_t=\argmax\limits_{a \in \mathcal{A}}
Q^A_{i-1}(s_t,a)$ then:
\begin{align*}
\mathrm{vec}(\mathbf{H}^A_i(s_t, a_t))^\top \mathrm{vec}(\mathbf{R}) &= \mathrm{vec}\left( (1-\alpha) \mathbf{H}^A_{i-1}(s_t, a_t) + \alpha \left( 1_{s_t, a_t} + \gamma \mathbf{H}^B_{i-1}(s_{t+1}, m_{t+1}) \right)\right) ^\top \mathrm{vec}(\mathbf{R})\\
& = (1-\alpha) Q^A_{i-1}(s_t, a_t) + \alpha \mathrm{vec}\left( 1_{s, a} + \gamma \mathbf{H}^B_{i-1}(s_{t+1}, m_{t+1}) \right) ^\top \mathrm{vec}(\mathbf{R})\\
& = (1-\alpha) Q^A_{i-1}(s_t, a_t) + \alpha(r_t + \gamma Q^B_{i-1}(s_{t+1}, m_{t+1})) \\
&= Q^A_{i}(s_t, a_t).
\end{align*}
as required.
\end{proof}
\section{Agent Description}
\label{sec:agent-description}
\paragraph{Blackjack}
In both Monte Carlo control and Q-learning we share the same training settings. We set the learning rate $\alpha = 0.1$, discount factor $\gamma = 1$. We set an initial exploration probability $\epsilon = 1$ which is exponentially decreased to $\epsilon = 0.05$ with a decay rate of $0.9999$ throughout the training.

\paragraph{Cartpole}
Similar to blackjack, we share the same training settings in cartpole. We set learning rate $\alpha = 0.1$ and discount factor $\gamma = 1$. Episode terminates when the length of the episode reaches 200 timesteps. We initially set exploration probability $\epsilon = 1$ which is linearly decreased to $\epsilon = 0.1$ throughout the first $500$ episodes.

In DQN training, we use Adam \cite{kingma2014adam} optimizer with $\epsilon = 1e-8$, exponential decays $\beta_1 = 0.9, \beta_2 = 0.999$. The learning rate is $\alpha = 0.0001$. We use Huber loss with discount factor $\gamma = 1$. We clip gradients to be in the range of $[-1, 1]$. For each learning iteration, we batch 16 experience together for optimisation. We initially set exploration probability $\epsilon = 1$ which is linearly decreased to $\epsilon = 0.1$ throughout the first $500$ episodes. Full details of the neural network architectures can be found in Table \ref{table:cartpole-dqn-arch} and \ref{table:cartpole-dbn-arch}.

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Layer} & \textbf{Type} & \textbf{Input} & \textbf{Size}     \\ \hline
input          & N/A           & N/A            & 4                 \\ \hline
fc1            & MLP           & input          & 128 \\ \hline
fc2            & MLP           & fc1            & 512 \\ \hline
output         & MLP           & fc2            & 2   \\ \hline
\end{tabular}
\caption{DQN neural network architecture}
\label{table:cartpole-dqn-arch}
\end{table}

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Layer} & \textbf{Type} & \textbf{Input} & \textbf{Size}     \\ \hline
input          & N/A           & N/A            & 162               \\ \hline
fc1            & MLP           & input          & 512 \\ \hline
fc2            & MLP           & fc1            & 1024 \\ \hline
fc3            & MLP           & fc2            & 2048   \\ \hline
output         & MLP           & fc3            & 2*2*162   \\ \hline
\end{tabular}
\caption{DBN neural network architecture. Output is reshaped to $\mathbb{R}^{A \times S \times A}$ before return.}
\label{table:cartpole-dbn-arch}
\end{table}

\paragraph{Taxi}
We first describe the training details for Q-learning. We set the learning rate $\alpha = 0.4$ and discount factor $\gamma = 0.9$. An episode terminates when the agent completes the episode or reaches a threshold of 200 timesteps. We initially set exploration probability $\epsilon = 1$ which is linearly decreased to $\epsilon = 0.1$ in the first $250$ episodes.

In DQN training, we use Adam \cite{kingma2014adam} optimizer with $\epsilon = 1e-8$, exponential decays $\beta_1 = 0.9, \beta_2 = 0.999$. The learning rate is $\alpha = 0.0001$. We use Huber loss with discount factor $\gamma = 1$. We clip gradients to be in the range of $[-1, 1]$. For each learning iteration, we batch 16 experience together for optimisation. We initially set exploration probability $\epsilon = 1$ which is linearly decreased to $\epsilon = 0.1$ in the first $250$ episodes. In order to speed up training of DBN, we made a small modification by splitting belief update and action into two inputs: belief update is a binary indicator function $1$ at position $s$, and action is converted into one-hot encoding before being fed into the neural network. Full details of the neural network architectures can be found in Table \ref{table:taxi-dqn-arch} and \ref{table:taxi-dbn-arch}.

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Layer} & \textbf{Type} & \textbf{Input} & \textbf{Size}     \\ \hline
input          & N/A           & N/A            & 4                 \\ \hline
fc1            & MLP           & input          & 500 \\ \hline
fc2            & MLP           & fc1            & 2000 \\ \hline
output         & MLP           & fc2            & 6   \\ \hline
\end{tabular}
\caption{DQN neural network architecture}
\label{table:taxi-dqn-arch}
\end{table}

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Layer} & \textbf{Type} & \textbf{Input}             & \textbf{Size}     \\ \hline
input\_belief   & N/A           & N/A                        & 500               \\ \hline
input\_action   & N/A           & N/A                        & 6              \\ \hline
belief\_stream  & MLP           & input\_belief               & 1024            \\ \hline
action\_stream  & MLP           & input\_action               & 128           \\ \hline
concat         & N/A           & input\_belief, input\_action & 1024+128 = 1152 \\ \hline
fc3            & MLP           & concat                     & 2048           \\ \hline
output         & MLP           & fc3                        & 500   \\ \hline
\end{tabular}
\caption{DBN neural network architecture. Output is reshaped to $\mathbb{R}^{A \times S \times A}$ before return.}
\label{table:taxi-dbn-arch}
\end{table}

\section{Further Results}
\label{sec:further-results}
Suppose we have the best action $a^0$ and second best action $a^1$ at state $s$, we can compute a contrastive explanation $G$ by subtracting the belief of $a^0$ against $a^1$:
\begin{equation}
    G(s, a^0, a^1) = \mathbf{H}(s, a^0) - \mathbf{H}(s, a^1)
    \label{eq:contrastive}
\end{equation}
Since $\mathbf{H}(s, a)$ is a decomposition of $Q(s, a)$, the subtraction will tell us which future states constitute the overall goodness of $a^0$ over $a^1$ in $s$. We apply equation \ref{eq:contrastive} to generate the contrastive explanations below.

\paragraph{Blackjack}
For conciseness, the visualisations of blackjack have been trimmed to only show reachable states. Figure~\ref{fig:bj-belief-maps} shows a belief map computed for an agent trained using Monte-Carlo control. This can be compared against that of the tabular Q-learning agent in Figure 3 of the main paper.

In Figure \ref{fig:bj-td-q} and Figure \ref{fig:bj-mc-q} we verify that we can recover the Q-table from learned beliefs via a proxy reward map, thus our theorem in the main paper holds. In figure \ref{fig:bj-contrastive1} and \ref{fig:bj-contrastive2} we show that different contrastive explanations can be extracted by applying equation \ref{eq:contrastive}.

\paragraph{Cartpole}
We verify in figure \ref{fig:cartpole-mc-q} and \ref{fig:cartpole-td-q} that our theorem in the main paper holds without the aid of a proxy map. We also give a demonstration of contrastive explanations in figure \ref{fig:cartpole-mc-contrastive-explantion} and \ref{fig:cartpole-td-contrastive-explantion}.

\paragraph{Taxi}
We provide animations of Figure 4 and 5 in the main text; see attached video for the animation. We also give contrastive explanations from DQN agent in Figure \ref{fig:taxi-dqn-contrastive}. Although the contrastive explanations are noticeably fuzzier, the extracted intention is similar to our results for Q-learning.

\newpage

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.23\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/bj_10,7,False_stick.png}
		\caption{Belief map for sticking}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.23\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/bj_10,7,False_hit.png}
		\caption{Belief map for hitting}
	\end{subfigure}
	~
	\begin{minipage}[b][0.25\textheight][s]{0.3\textwidth}
		\begin{subfigure}[b]{\textwidth}
			\centering
			\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/accum_bj_10,7,False_stick.png}
			\caption{Reward belief for sticking}
		\end{subfigure}
		\vfill
		\begin{subfigure}[b]{\textwidth}
			\centering
			\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/accum_bj_10,7,False_hit.png}
			\caption{Reward belief for hitting}
		\end{subfigure}
	\end{minipage}
	\caption{Monte Carlo control blackjack simulation: here we show the visualisations of belief maps and proxy action-reward map when player sum $=10$, dealer card $=7$ with no usable ace.}
	\label{fig:bj-belief-maps-mc}
\end{figure}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/q_noace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/rcvq_noace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/q_noace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/rcvq_noace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/q_ace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/rcvq_ace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/q_ace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/td/rcvq_ace_hit.png}
	\end{subfigure}
	\caption{Q-learning: visualisations of ground truth Q-table and recovered Q-table from learned belief.}
	\label{fig:bj-td-q}

\end{figure}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/q_noace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/rcvq_noace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/q_noace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/rcvq_noace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/q_ace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/rcvq_ace_stick.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/q_ace_hit.png}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.25\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/rcvq_ace_hit.png}
	\end{subfigure}
	\caption{Monte Carlo control: visualisations of ground truth Q-table and recovered Q-table from learned belief.}
	\label{fig:bj-mc-q}

\end{figure}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.28\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/contrastive_belief_10,7,False.png}
		\caption{}
		\label{subfig:bj-mc-contrastive-belief1}
	\end{subfigure}
	\qquad
	\begin{subfigure}[b]{0.3\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/contrastive_proxy_10,7,False.png}
		\caption{}
		\label{subfig:bj-mc-contrastive-proxy1}
	\end{subfigure}
	\caption{Monte Carlo control, blackjack simulation: contrastive explanation when player sum $=10$, dealer card $=7$ with no usable ace. Figure \ref{subfig:bj-mc-contrastive-belief1} shows that hitting would grant access to various future states. This is reflected in the figure \ref{subfig:bj-mc-contrastive-proxy1}, as the red block (consequence of sticking) indicates sticking is undesirable since the action is more likely to generate $-1$ reward.}
	\label{fig:bj-contrastive1}

\end{figure}

\begin{figure}
	\centering
	\begin{subfigure}[b]{0.28\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/contrastive_belief_17,9,False.png}
		\caption{}
		\label{subfig:bj-mc-contrastive-belief2}
	\end{subfigure}
	\qquad
	\begin{subfigure}[b]{0.3\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/blackjack/mc/contrastive_proxy_17,9,False.png}
		\caption{}
		\label{subfig:bj-mc-contrastive-proxy2}
	\end{subfigure}
	\caption{Monte Carlo control, blackjack simulation: contrastive explanation when player sum $=17$, dealer card $=9$ with no usable ace. In contrast to figure \ref{subfig:bj-mc-contrastive-belief1}, hitting is more likely to yield a more negative consequnce than twisting, as evidenced by \ref{subfig:bj-mc-contrastive-belief2} that the red block (consequence of hitting) at $(1, -1)$ clearly outweights the cumulative sum of blue blocks (consequence of sticking).}
	\label{fig:bj-contrastive2}

\end{figure}

\begin{figure}
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/cartpole/mc/belief_s79_a1.png}
		\caption{Belief map}
	\end{subfigure}
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=\textwidth]{supp_figs/cartpole/mc/trajectory.png}
		\caption{Forward Simulation}
	\end{subfigure}
	\caption{Monte Carlo control, cartpole simulation: belief map and trajectory visualisations.}
\end{figure}

 \begin{figure}
 	\begin{subfigure}[b]{\textwidth}
 		\centering
 		\includegraphics[width=\textwidth]{supp_figs/cartpole/mc/q.png}
 	\end{subfigure}
 	\begin{subfigure}[b]{\textwidth}
 		\centering
 		\includegraphics[width=\textwidth]{supp_figs/cartpole/mc/rcvq.png}
 	\end{subfigure}
 	\caption{Monte Carlo control, cartpole simulation: Q-table and recovered Q-table from learned beliefs. Each value is numerically identical except for floating-point errors.}
 	\label{fig:cartpole-mc-q}
 \end{figure}

  \begin{figure}
 	\begin{subfigure}[b]{\textwidth}
 		\centering
 		\includegraphics[width=\textwidth]{supp_figs/cartpole/q/q.png}
 	\end{subfigure}
 	\begin{subfigure}[b]{\textwidth}
 		\centering
 		\includegraphics[width=\textwidth]{supp_figs/cartpole/q/rcvq.png}
 	\end{subfigure}
 	\caption{Q-learning, cartpole simulation: Q-table and recovered Q-table from learned beliefs. Each value is numerically identical except for floating-point errors.}
 	\label{fig:cartpole-td-q}
 \end{figure}

\begin{figure}
 	\centering
 	\includegraphics[width=\textwidth]{supp_figs/cartpole/mc/contrastive_s79.png}
 	\caption{Monte Carlo control, cartpole simulation: selected contrastive explanations at state 79. We can intuitively reason and justify the agent's decision of moving to the right since it offers a greater stability. Moving to the left risks heading towards the edges highlighted by the red blocks.}
 	\label{fig:cartpole-mc-contrastive-explantion}
 \end{figure}

\begin{figure}
 	\includegraphics[width=\textwidth]{supp_figs/cartpole/q/contrastive_s79.png}
 	\caption{Q-learning, cartpole simulation: selected contrastive explanations at state 79. The figure shows that moving to the right is undesirable since it will lead the agent to visit states which can cause possible failure, whereas moving to the left can offer more guarantee of staying in the middle.}
 	\label{fig:cartpole-td-contrastive-explantion}
 \end{figure}

\begin{figure}
	\centering
	\hsize=0.7\textwidth
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=\hsize]{supp_figs/taxi/dqn/contrast_009.png}
		\caption{Value ambiguity. $a^0$: move south, $a^1$: move west. $Q(s, a^0) = 14.9908$, $Q(s, a^1) = 14.9858$.}
		\label{subfig:contrastive-dqn-value-ambiguity}
	\end{subfigure}
	~
	\begin{subfigure}[b]{\textwidth}
		\centering
		\includegraphics[width=\hsize]{supp_figs/taxi/dqn/contrast_012.png}
		\caption{Bounce back. $a^0$: move south, $a^1$: move west. $Q(s, a^0) = 17.9323$, $Q(s, a^1) = 15.9981$.}
		\label{subfig:contrastive-dqn-bounce-back}
	\end{subfigure}
    \caption{DQN, taxi simulation: contrastive explanations of taxi DQN agent. Refer to Figure 7 in main text for descriptions of each subplot.}
	\label{fig:taxi-dqn-contrastive}
\end{figure}

\clearpage
\ifstandalone
\bibliographystyle{plainnat}