\section{Introduction}

Planning algorithms based on lookahead search have achieved remarkable successes in artificial intelligence. Human world champions have been defeated in classic games such as checkers \cite{schaeffer:chinook}, chess \cite{campbell:deep-blue}, Go \cite{Silver16AG} and poker \cite{brown2018superhuman, deepstack}, and planning algorithms have had real-world impact in applications from logistics \cite{vlahavas2013planning} to chemical synthesis \cite{alphachem}.
However, these planning algorithms all rely on knowledge of the environment's dynamics, such as the rules of the game or an accurate simulator, preventing their direct application to real-world domains like robotics, industrial control, or intelligent assistants.

Model-based reinforcement learning (RL) \cite{sutton:book} aims to address this issue by first learning a model of the environment's dynamics, and then planning with respect to the learned model. Typically, these models have either focused on reconstructing the true environmental state \cite{pilco:deisenroth,heess:stochastic_value_gradients,levine:learning_guided_policy}, or the sequence of full observations \cite{hafner:planet, kaiser:simple}. However, prior work \cite{state_space_models, hafner:planet, kaiser:simple} remains far from the state of the art in visually rich domains, such as Atari 2600 games \cite{ALE}.
Instead, the most successful methods are based on model-free RL \cite{impala, r2d2, apex} -- i.e. they estimate the optimal policy and/or value function directly from interactions with the environment. However, model-free algorithms are in turn far from the state of the art in domains that require precise and sophisticated lookahead, such as chess and Go.

In this paper, we introduce \emph{MuZero}{}, a new approach to model-based RL that achieves state-of-the-art performance in Atari 2600, a visually complex set of domains, while maintaining superhuman performance in precision planning tasks such as chess, shogi and Go.
\emph{MuZero}{} builds upon \emph{AlphaZero}{}'s \cite{Silver18AZ} powerful search and search-based policy iteration algorithms, but incorporates a learned model into the training procedure. \emph{MuZero}{} also extends \emph{AlphaZero}{} to a broader set of environments including single agent domains and non-zero rewards at intermediate time-steps.

The main idea of the algorithm (summarized in Figure 1) is to predict those aspects of the future that are directly relevant for planning. The model receives the observation (e.g. an image of the Go board or the Atari screen) as an input and transforms it into a hidden state. The hidden state is then updated iteratively by a recurrent process that receives the previous hidden state and a hypothetical next action. At every one of these steps the model predicts the policy (e.g. the move to play), value function (e.g. the predicted winner), and immediate reward (e.g. the points scored by playing a move). The model is trained end-to-end, with the sole objective of accurately estimating these three important quantities, so as to match the improved estimates of policy and value generated by search as well as the observed reward. There is no direct constraint or requirement for the hidden state to capture all information necessary to reconstruct the original observation, drastically reducing the amount of information the model has to maintain and predict; nor is there any requirement for the hidden state to match the unknown, true state of the environment; nor any other constraints on the semantics of state. Instead, the hidden states are free to represent state in whatever way is relevant to predicting current and future values and policies. Intuitively, the agent can invent, internally, the rules or dynamics that lead to most accurate planning.

\section{Prior Work}

Reinforcement learning may be subdivided into two principal categories: model-based, and model-free \cite{sutton:book}. Model-based RL constructs, as an intermediate step, a model of the environment. Classically, this model is represented by a Markov-decision process (MDP) \cite{puterman:MDP} consisting of two components: a state transition model, predicting the next state, and a reward model, predicting the expected reward during that transition. The model is typically conditioned on the selected action, or a temporally abstract behavior such as an option \cite{sutton:between}. Once a model has been constructed, it is straightforward to apply MDP planning algorithms, such as value iteration \cite{puterman:MDP} or Monte-Carlo tree search (MCTS) \cite{coulom:mcts}, to compute the optimal value or optimal policy for the MDP. In large or partially observed environments, the algorithm must first construct the state representation that the model should predict. This tripartite separation between representation learning, model learning, and planning is potentially problematic since the agent is not able to optimize its representation or model for the purpose of effective planning, so that, for example modeling errors may compound during planning.

A common approach to model-based RL focuses on directly modeling the observation stream at the pixel-level. It has been hypothesized that deep, stochastic models may mitigate the problems of compounding error \cite{hafner:planet, kaiser:simple}. However, planning at pixel-level granularity is not computationally tractable in large scale problems. Other methods build a latent state-space model that is sufficient to reconstruct the observation stream at pixel level \cite{wahlstrom:pixels_to_torques,watter:embed_to_control}, or to predict its future latent states \cite{ha:world_model,gelada:deepmdp}, which facilitates more efficient planning but still focuses the majority of the model capacity on potentially irrelevant detail. None of these prior methods has constructed a model that facilitates effective planning in visually complex domains such as Atari; results lag behind well-tuned, model-free methods, even in terms of data efficiency \cite{hado:replay}.

A quite different approach to model-based RL has recently been developed, focused end-to-end on predicting the value function \cite{silver:predictron}. The main idea of these methods is to construct an abstract MDP model such that planning in the abstract MDP is equivalent to planning in the real environment. This equivalence is achieved by ensuring \emph{value equivalence}, i.e. that, starting from the same real state, the cumulative reward of a trajectory through the abstract MDP matches the cumulative reward of a trajectory in the real environment.

The predictron \cite{silver:predictron} first introduced value equivalent models for predicting value (without actions). Although the underlying model still takes the form of an MDP, there is no requirement for its transition model to match real states in the environment. Instead the MDP model is viewed as a hidden layer of a deep neural network. The unrolled MDP is trained such that the expected cumulative sum of rewards matches the expected value with respect to the real environment, e.g. by temporal-difference learning.

Value equivalent models were subsequently extended to optimising value (with actions). TreeQN \cite{farquhar:treeqn} learns an abstract MDP model, such that a tree search over that model (represented by a tree-structured neural network) approximates the optimal value function. Value iteration networks \cite{aviv:vin} learn a local MDP model, such that value iteration over that model (represented by a convolutional neural network) approximates the optimal value function. Value prediction networks \cite{oh:vpn} are perhaps the closest precursor to \emph{MuZero}{}: they learn an MDP model grounded in real actions; the unrolled MDP is trained such that the cumulative sum of rewards, conditioned on the actual sequence of actions generated by a simple lookahead search, matches the real environment. Unlike \emph{MuZero}{} there is no policy prediction, and the search only utilizes value prediction.

\section{\emph{MuZero}{} Algorithm}

\begin{figure}
\includegraphics[width=\textwidth]{images/learned_model_search_play_train}
\caption{
\label{fig:recurrent_search}
\textbf{Planning, acting, and training with a learned model.}
\textbf{(A)} How \emph{MuZero}{} uses its model to plan. The model consists of three connected components for representation, dynamics and prediction. %
Given a previous hidden state $s^{k-1}$ and a candidate action $a^{k}$, the \emph{dynamics} function $g$ produces an immediate reward $r^{k}$ and a new hidden state $s^{k}$. The policy $p^k$ and value function $v^k$ are computed from the hidden state $s^k$ by a \emph{prediction} function $f$. The initial hidden state $s^0$ is obtained by passing the past observations (e.g. the Go board or Atari screen) into a \emph{representation} function $h$.
\textbf{(B)} How \emph{MuZero}{} acts in the environment. A Monte-Carlo Tree Search is performed at each timestep $t$, as described in A. An action $a_{t+1}$ is sampled from the search policy $\pi_t$, which is proportional to the visit count for each action from the root node. The environment receives the action and generates a new observation $o_{t+1}$ and reward $u_{t+1}$. At the end of the episode the trajectory data is stored into a replay buffer.
\textbf{(C)} How \emph{MuZero}{} trains its model. A trajectory is sampled from the replay buffer. For the initial step, the representation function $h$ receives as input the past observations $o_1, ..., o_t$ from the selected trajectory. The model is subsequently unrolled recurrently for $K$ steps. %
At each step $k$, the dynamics function $g$ receives as input the hidden state $s^{k-1}$ from the previous step and the real action $a_{t+k}$.
The parameters of the representation, dynamics and prediction functions are jointly trained, end-to-end by backpropagation-through-time, to predict three quantities: the policy $\mathbf{p}^k \approx \pi_{t+k}$, value function $v^k \approx z_{t+k}$, and reward $r_{t+k} \approx u_{t+k}$, where $z_{t+k}$ is a sample return: either the final reward (board games) or $n$-step return (Atari).
}
\end{figure}

We now describe the \emph{MuZero}{} algorithm in more detail. Predictions are made at each time-step $t$, for each of $k = 1...K$ steps, by a model $\mu_\theta$, with parameters $\theta$, conditioned on past observations $o_1, ..., o_t$ and future actions $a_{t+1}, ..., a_{t+k}$. The model predicts three future quantities: the policy $\mathbf{p}^k_t \approx \pi(a_{t+k+1} | o_1, ..., o_t, a_{t+1}, ..., a_{t+k})$, the value function $v^k_t \approx \expect{u_{t+k+1} + \gamma u_{t+k+2} + ... | o_1, ..., o_t, a_{t+1}, ..., a_{t+k}}$, and the immediate reward $r^k_t \approx u_{t+k}$, where $u_{.}$ is the true, observed reward, $\pi$ is the policy used to select real actions, and $\gamma$ is the discount function of the environment.

Internally, at each time-step $t$ (subscripts $_t$ suppressed for simplicity), the model is represented by the combination of a \emph{representation} function, a \emph{dynamics} function, and a \emph{prediction} function. The dynamics function, $r^k, s^k = g_\theta(s^{k-1}, a^k)$, is a recurrent process that computes, at each hypothetical step $k$, an immediate reward $r^k$ and an internal state $s^k$. It mirrors the structure of an MDP model that computes the expected reward and state transition for a given state and action \cite{puterman:MDP}. However, unlike traditional approaches to model-based RL \cite{sutton:book}, this internal state $s^k$ has no semantics of environment state attached to it -- it is simply the hidden state of the overall model, and its sole purpose is to accurately predict relevant, future quantities: policies, values, and rewards. In this paper, the \emph{dynamics} function is represented deterministically; the extension to stochastic transitions is left for future work. The policy and value functions are computed from the internal state $s^k$ by the prediction function, $\mathbf{p}^k, v^k = f_\theta(s^k)$, akin to the joint policy and value network of \emph{AlphaZero}{}. The ``root" state $s^0$ is initialized using a representation function that encodes past observations, $s^0 = h_\theta(o_1, ..., o_t)$; again this has no special semantics beyond its support for future predictions.

Given such a model, it is possible to search over hypothetical future trajectories $a^1, ..., a^k$ given past observations $o_1, ..., o_t$. For example, a naive search could simply select the $k$ step action sequence that maximizes the value function. %
More generally, we may apply any MDP planning algorithm to the internal rewards and state space induced by the dynamics function. Specifically, we use an MCTS algorithm similar to \emph{AlphaZero}{}'s search, generalized to allow for single agent domains and intermediate rewards (see Methods). At each internal node, it makes use of the policy, value and reward estimates produced by the current model parameters $\theta$. The MCTS algorithm outputs a recommended policy $\pi_t$ and estimated value $\nu_t$. An action $a_{t+1} \sim \pi_t$ is then selected.

All parameters of the model are trained jointly to accurately match the policy, value, and reward, for every hypothetical step $k$, to corresponding target values observed after $k$ actual time-steps have elapsed. Similarly to \emph{AlphaZero}{}, the improved policy targets are generated by an MCTS search; the first objective is to minimise the error between predicted policy $\mathbf{p}_t^k$ and search policy $\pi_{t+k}$. Also like \emph{AlphaZero}{}, the improved value targets are generated by playing the game or MDP. However, unlike \emph{AlphaZero}{}, we allow for long episodes with discounting and intermediate rewards by \emph{bootstrapping} $n$ steps into the future from the search value, $z_t = u_{t+1} + \gamma u_{t+2} + ... + \gamma^{n-1} u_{t+n} + \gamma^n \nu_{t+n}$. Final outcomes $\{lose, draw, win\}$ in board games are treated as rewards $u_t \in \{ -1, 0, +1 \}$ occuring at the final step of the episode. Specifically, the second objective is to minimize the error between the predicted value $v^k_t$ and the value target, $z_{t+k}$ \footnote{For chess, Go and shogi, the same squared error loss as \emph{AlphaZero}{} is used for rewards and values. A cross-entropy loss was found to be more stable than a squared error when encountering rewards and values of variable scale in Atari. Cross-entropy was used for the policy loss in both cases.}. The reward targets are simply the observed rewards; the third objective is therefore to minimize the error between the predicted reward $r^k_t$ and the observed reward $u_{t+k}$. Finally, an L2 regularization term is also added, leading to the overall loss:

\begin{align}
l_t(\theta) &= \sum_{k=0}^K l^r (u_{t+k}, r_t^k) + l^v(z_{t+k}, v^k_t) + l^p(\pi_{t+k}, \mathbf{p}^k_t) + c ||\theta||^2
\label{muzero_eqn}
\end{align}
where $l^r$, $l^v$, and $l^p$ are loss functions for reward, value and policy respectively.
Supplementary Figure \ref{fig:muzero_equations} summarizes the equations governing how the \emph{MuZero}{} algorithm plans, acts, and learns.

\section{Results}

\begin{figure} %
\begin{tabularx}{\textwidth}{c c c c}

\textbf{Chess}
& \textbf{Shogi}
& \textbf{Go}
& \textbf{Atari} \\

\newgame
\scalebox{0.62}{\newgame \notationoff \showboard} &
\scalebox{0.745}{
\begin{CJK}{UTF8}{min}
\begin{TAB}(e,0.5cm,0.5cm){|c|c|c|c|c|c|c|c|c|}{|c|c|c|c|c|c|c|c|c|}
\rotatebox[origin=c]{180}{香} & \rotatebox[origin=c]{180}{桂} & \rotatebox[origin=c]{180}{銀} & \rotatebox[origin=c]{180}{金} & \rotatebox[origin=c]{180}{玉} & \rotatebox[origin=c]{180}{金} & \rotatebox[origin=c]{180}{銀} & \rotatebox[origin=c]{180}{桂} & \rotatebox[origin=c]{180}{香}\\
 & \rotatebox[origin=c]{180}{飛} &  &  &  &  &  & \rotatebox[origin=c]{180}{角} & \\
\rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩} & \rotatebox[origin=c]{180}{歩}\\
 &  &  &  &  &  &  &  & \\
 &  &  &  &  &  &  &  & \\
 &  &  &  &  &  &  &  & \\
歩 & 歩 & 歩 & 歩 & 歩 & 歩 & 歩 & 歩 & 歩\\
 & 角 &  &  &  &  &  & 飛 & \\
香 & 桂 & 銀 & 金 & 玉 & 金 & 銀 & 桂 & 香\\
\end{TAB}
\end{CJK}
}%
&
\scalebox{0.195}{
\begin{tikzpicture}
\draw[help lines] (0,0) grid (18,18);
\draw[fill=gray] (3,3) circle[radius=0.1];
\draw[fill=gray] (3,9) circle[radius=0.1];
\draw[fill=gray] (3,15) circle[radius=0.1];
\draw[fill=gray] (9,3) circle[radius=0.1];
\draw[fill=gray] (9,9) circle[radius=0.1];
\draw[fill=gray] (9,15) circle[radius=0.1];
\draw[fill=gray] (15,3) circle[radius=0.1];
\draw[fill=gray] (15,9) circle[radius=0.1];
\draw[fill=gray] (15,15) circle[radius=0.1];
\end{tikzpicture}
}
&
\includegraphics[width=0.219\textwidth,height=0.219\textwidth]{images/mspacman}
\\

\multicolumn{4}{c}{
  \hspace{-3.25em}
  \includegraphics[width=\textwidth + 2.6em]{generated/training_plots}
}

\\

\end{tabularx}

\caption{
\label{fig:results}
\textbf{Evaluation of \emph{MuZero}{} throughout training in chess, shogi, Go and Atari.} The x-axis shows millions of training steps. For chess, shogi and Go, the y-axis shows Elo rating, established by playing games against \emph{AlphaZero}{} using 800 simulations per move for both players. \emph{MuZero}{}'s Elo is indicated by the blue line, \emph{AlphaZero}{}'s Elo by the horizontal orange line. For Atari, mean (full line) and median (dashed line) human normalized scores across all 57 games are shown on the y-axis. The scores for R2D2 \cite{r2d2}, (the previous state of the art in this domain, based on model-free RL) are indicated by the horizontal orange lines. Performance in Atari was evaluated using 50 simulations every fourth time-step, and then repeating the chosen action four times, as in prior work \cite{dqn}.
}
\end{figure} %

We applied the \emph{MuZero}{} algorithm to the classic board games Go, chess and shogi~\footnote{Imperfect information games such as Poker are not directly addressed by our method.}, as benchmarks for challenging planning problems, and to all 57 games in the Atari Learning Environment \cite{ALE}, as benchmarks for visually complex RL domains.

In each case we trained \emph{MuZero}{} for $K=5$ hypothetical steps. Training proceeded for 1 million mini-batches of size 2048 in board games and of size 1024 in Atari. During both training and evaluation, \emph{MuZero}{} used 800 simulations for each search in board games, and 50 simulations for each search in Atari. The representation function uses the same convolutional \cite{convnet} and residual \cite{he:resnet} architecture as \emph{AlphaZero}{}, but with 16 residual blocks instead of 20. The dynamics function uses the same architecture as the representation function and the prediction function uses the same architecture as \emph{AlphaZero}{}. All networks use 256 hidden planes (see Methods for further details).

Figure \ref{fig:results} shows the performance throughout training in each game. In Go, \emph{MuZero}{} slightly exceeded the performance of \emph{AlphaZero}{}, despite using less computation per node in the search tree (16 residual blocks per evaluation in \emph{MuZero}{} compared to 20 blocks in \emph{AlphaZero}{}). This suggests that \emph{MuZero}{} may be caching its computation in the search tree and using each additional application of the dynamics model to gain a deeper understanding of the position.

In Atari, \emph{MuZero}{} achieved a new state of the art for both mean and median normalized score across the 57 games of the Arcade Learning Environment, outperforming the previous state-of-the-art method R2D2 \cite{r2d2} (a model-free approach) in 42 out of 57 games, and outperforming the previous best model-based approach SimPLe \cite{kaiser:simple} in all games (see Table \ref{tab:atari-results-at30}).

We also evaluated a second version of \emph{MuZero}{} that was optimised for greater sample efficiency. Specifically, it reanalyzes old trajectories by re-running the MCTS using the latest network parameters to provide fresh targets (see Appendix \ref{reanalyze}). When applied to 57 Atari games, using 200 million frames of experience per game, \emph{MuZero Reanalyze}{} achieved 731\% median normalized score, compared to 192\%, 231\% and 431\% for previous state-of-the-art model-free approaches IMPALA \cite{impala}, Rainbow \cite{rainbow} and LASER \cite{laser} respectively.

\begin{table}

\begin{tabularx}{\textwidth}{c c c c c c}

\toprule
Agent &   Median &     Mean & Env. Frames & Training Time & Training Steps \\
\midrule
Ape-X \cite{apex} &  434.1\% & 1695.6\% & 22.8B & 5 days & 8.64M\\
R2D2 \cite{r2d2} & 1920.6\% & 4024.9\% & 37.5B & 5 days & 2.16M \\
\emph{MuZero}{} & \textbf{2041.1\%
} & \textbf{4999.2\%
} & 20.0B & 12 hours & 1M \\
\midrule
IMPALA \cite{impala}  & 191.8\% & 957.6\% & 200M & -- & -- \\
Rainbow \cite{rainbow} &  231.1\% & -- & 200M & 10 days & -- \\
UNREAL\textsuperscript{a} \cite{unreal} & 250\%\textsuperscript{a} & 880\%\textsuperscript{a} & 250M & -- & -- \\
LASER \cite{laser} & 431\% & -- & 200M & -- & -- \\
\emph{MuZero Reanalyze}{} & \textbf{731.1\%} & \textbf{2168.9\%} & 200M & 12 hours & 1M \\
\bottomrule
\end{tabularx}

\caption{
\label{tab:atari-comparison}
\textbf{Comparison of \emph{MuZero}{} against previous agents in Atari}. We compare separately against agents trained in large (top) and small (bottom) data settings; all agents other than \emph{MuZero}{} used model-free RL techniques. Mean and median scores are given, compared to human testers. The best results are highlighted in \textbf{bold}. \emph{MuZero}{} sets a new state of the art in both settings. \textsuperscript{a}Hyper-parameters were tuned per game.
}
\end{table}

To understand the role of the model in \emph{MuZero}{} we also ran several experiments, focusing on the board game of Go and the Atari game of Ms.~Pacman.

First, we tested the scalability of planning (Figure \ref{fig:ablations}A), in the canonical planning problem of Go.  We compared the performance of search in \emph{AlphaZero}{}, using a perfect model, to the performance of search in \emph{MuZero}, using a learned model. Specifically, the fully trained \emph{AlphaZero}{} or \emph{MuZero}{} was evaluated by comparing MCTS with different thinking times. \emph{MuZero}{} matched the performance of a perfect model, even when doing much larger searches (up to 10s thinking time) than those from which the model was trained (around 0.1s thinking time, see also Figure \ref{fig:extended_ablations}A).

We also investigated the scalability of planning across all Atari games (see Figure \ref{fig:ablations}B). We compared MCTS with different numbers of simulations, using the fully trained \emph{MuZero}{}. The improvements due to planning are much less marked than in Go, perhaps because of greater model inaccuracy; performance improved slightly with search time, but plateaued at around 100 simulations. Even with a single simulation -- i.e. when selecting moves solely according to the policy network -- \emph{MuZero}{} performed well, suggesting that, by the end of training, the raw policy has learned to internalise the benefits of search (see also Figure \ref{fig:extended_ablations}B).

Next, we tested our model-based learning algorithm against a comparable model-free learning algorithm (see Figure \ref{fig:ablations}C). We replaced the training objective of \emph{MuZero}{} (Equation 1) with a model-free Q-learning objective (as used by R2D2), and the dual value and policy heads with a single head representing the Q-function $Q(\cdot|s_t)$. Subsequently, we trained and evaluated the new model without using any search. When evaluated on Ms.~Pacman, our model-free algorithm achieved identical results to R2D2, but learned significantly slower than \emph{MuZero}{} and converged to a much lower final score. We conjecture that the search-based policy improvement step of \emph{MuZero}{} provides a stronger learning signal than the high bias, high variance targets used by Q-learning.

To better understand the nature of \emph{MuZero}{}'s learning algorithm, we measured how \emph{MuZero}{}'s training scales with respect to the amount of search it uses \emph{during} training. Figure \ref{fig:ablations}D shows the performance in Ms.~Pacman, using an MCTS of different simulation counts per move throughout training. Surprisingly, and in contrast to previous work \cite{negative_mcts_results}, even with only 6 simulations per move -- fewer than the number of actions -- \emph{MuZero}{} learned an effective policy and improved rapidly. With more simulations performance jumped significantly higher. For analysis of the policy improvement during each individual iteration, see also Figure \ref{fig:extended_ablations} C and D.

\begin{figure} %
\includegraphics[width=0.5\textwidth]{generated/go_scaling_elos}
\includegraphics[width=0.5\textwidth]{generated/atari_scaling_score}
\includegraphics[width=\textwidth]{generated/pacman_ablations_main}
\caption{
\label{fig:ablations}
\textbf{Evaluations of \emph{MuZero}{} on Go (A), all 57 Atari Games (B) and Ms.~Pacman (C-D)}. (\textbf{A}) Scaling with search time per move in Go, comparing the learned model with the ground truth simulator. Both networks were trained at 800 simulations per search, equivalent to 0.1 seconds per search. Remarkably, the learned model is able to scale well to up to two orders of magnitude longer searches than seen during training. (\textbf{B}) Scaling of final human normalized mean score in Atari with the number of simulations per search. The network was trained at 50 simulations per search. Dark line indicates mean score, shaded regions indicate 25th to 75th and 5th to 95th percentiles. The learned model's performance increases up to 100 simulations per search. Beyond, even when scaling to much longer searches than during training, the learned model's performance remains stable and only decreases slightly.
This contrasts with the much better scaling in Go (A), presumably due to greater model inaccuracy in Atari than Go.
(\textbf{C}) Comparison of MCTS based training with Q-learning in the \emph{MuZero}{} framework on Ms.~Pacman, keeping network size and amount of training constant. The state of the art Q-Learning algorithm R2D2 is shown as a baseline. Our Q-Learning implementation reaches the same final score as R2D2, but improves slower and results in much lower final performance compared to MCTS based training. (\textbf{D}) Different networks trained at different numbers of simulations per move, but all evaluated at 50 simulations per move. Networks trained with more simulations per move improve faster, consistent with ablation (B), where the policy improvement is larger when using more simulations per move. Surprisingly, \emph{MuZero}{} can learn effectively even when training with less simulations per move than are enough to cover all 8 possible actions in Ms.~Pacman.
}
\end{figure} %

\section{Conclusions}

Many of the breakthroughs in artificial intelligence have been based on either high-performance planning \cite{campbell:deep-blue, Silver16AG, Silver18AZ} or model-free reinforcement learning methods \cite{dqn, openai_dota, alphastar}. In this paper we have introduced a method that combines the benefits of both approaches. Our algorithm, \emph{MuZero}{}, has both matched the superhuman performance of high-performance planning algorithms in their favored domains -- logically complex board games such as chess and Go -- and outperformed state-of-the-art model-free RL algorithms in their favored domains -- visually complex Atari games. Crucially, our method does not require any knowledge of the game rules or environment dynamics, potentially paving the way towards the application of powerful learning and planning methods to a host of real-world domains for which there exists no perfect simulator.

\section{Acknowledgments}

Lorrayne Bennett, Oliver Smith and Chris Apps for organizational assistance; Koray Kavukcuoglu for reviewing the paper; Thomas Anthony, Matthew Lai, Nenad Tomasev, Ulrich Paquet, Sumedh Ghaisas for many fruitful discussions; and the rest of the DeepMind team for their support.

\bibliographystyle{plain}