text:
\section{Introduction}
\label{Introduction}
Machine learning, and deep reinforcement learning (deep-RL) in particular, is a promising
approach for learning controllers or action policies for complex systems where analytic methods are elusive.
The use of robot simulators for machine learning is driven by a compromise between the large data requirements of Deep-RL and the cost of collecting experience on physical robots systems. Except for simple robot systems in controlled environments, however, real robot experience may not correspond to situations that can be simulated; an issue known as the \emph{reality gap} ~\cite{jakobi1995noise}. This gap limits the utility of simulation-based learning.

\emph{Domain randomization} (DR) aims to address the reality gap by training policies to maximize performance over a diverse set of simulation models, where the parameters of each model are sampled randomly from a given task distribution. The effectiveness of DR has been demonstrated by training controllers purely in simulation that subsequently transfer successfully to real robot systems~\cite{peng2018sim, openai2018learning} and by fine-tuning the DR distributions with real world data ~\cite{chen2018hardware}.

While successful, an aspect that has not been addressed in depth is the selection of the domain randomization distribution.
The DR distribution should be selected carefully to ensure the ability to generalize within the simulated experience. Selecting a distribution that is too wide may result in an under-performing policy (see Fig.~\ref{fig:hopper_fixed_dr}) or slow convergence (see Fig.~\ref{fig:epopt_minmax}) . Conversely, training  on  a distribution  that  is  too  narrow  leads  to  policies  that  don't generalize well~\cite{rajeswaran2016epopt,packer2018assessing}. Most of the existing methods~\cite{rajeswaran2016epopt, chebotar2018closing, ramos2019bayessim}, assume the availability of real robot data. In this work, we study the extreme case where no real robot data is available, similar to~\cite{openai2018learning}, in which the appropriate selection of a DR distribution is crucial.

\begin{figure}[t!]
\centering
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid-crop.png}
        \caption{\small Average rewards with independently trained policies per context}
        \label{fig:hopper_target_rewards}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_fixed_dr-crop.png}
        \caption{\small Average rewards with fixed, uniform domain randomization}
        \label{fig:hopper_fixed_dr}
    \end{subfigure}
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_learned_dr-crop.png}
        \caption{\small Average rewards with learned Gaussian domain randomization}
        \label{fig:hopper_learned_dr}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_gaussian_confidence-crop.png}
        \caption{\small Confidence regions of the learned distribution for 1: 68\%, 2: 95\%, and 3: 99\% confidence}
        \label{fig:hopper_confidence}
    \end{subfigure}
    \caption{\small Figure~\ref{fig:hopper_target_rewards} shows the maximum rewards obtained by exhaustively running PPO on a grid of simulation parameters, ~\ref{fig:hopper_fixed_dr} shows the performance of a policy trained with uniform domain randomization over the grid, ~\ref{fig:hopper_learned_dr} results from our method when learning the parameters of a Gaussian domain randomization distribution (whose confidence regions are shown in ~\ref{fig:hopper_confidence}). For fixed and learned DR the reward averages were obtained by evaluating the policy with 10 different rollouts.}

    \label{fig:hopper_grid_comparison}
\end{figure}

We propose to learn the parameters of the simulator distribution, such that the policy is trained under an evolving sequence of simulators that gives priority to environments where the current policy can achieve high performance. Our proposed method aims to find a compromise between variety and performance, by optimizing the DR distribution to focus on environments where high rewards are likely, while regularizing the distribution parameters via a KL-divergence objective (explained in Sec.~\ref{sec:method}) and using more data from the worst performing simulators (see the experiments in Sec~\ref{sec:epopt}). Our algorithm learns the domain randomization distribution while simultaneously optimizing the policy to maximize performance over the learned distribution. Figure~\ref{fig:hopper_learned_dr} shows our resuls towards finding the \emph{sweet-spot} distribution.
We evaluate our method on a variety of control problems from the OpenAI Gym suite of benchmarks~\cite{gym2016}. We find that our method is able to improve on the performance of fixed domain randomization. Furthermore, we demonstrate our model's robustness to initial simulator distribution parameters, showing that our method repeatably converges to similar domain randomization distributions across different experiments.

\section{Related Work}

Developing robust or generalizable controllers has a long and rich history beyond the scope of this paper~\cite{Zames1981,zhou1998robustControl}.
Caruana was on of the first to deeply explore the advantages of multi-task learning in a robotics context~\cite{Caruana93multitasklearning}. Baxter considered multi-task learning in a Bayesian/information-theoretic context~\cite{Baxter1997}.

Packer et al~\cite{packer2018assessing} present an empirical study of generalization in Deep-RL, testing interpolation and extrapolation performance of state-of-the-art algorithms when varying simulation parameters in control tasks. The authors provide an experimental assessment of generalization under varying training and testing distributions. Our work extends these results by providing results for the case when the training distribution parameters are learned and change during policy training.

Chebotar et al~\cite{chebotar2018closing} propose training policies on a distribution of simulators, whose parameters are fit to real-world data. Their proposed algorithm switches back and forth between optimizing the policy under the DR distribution and updating the DR distribution by minimizing the discrepancy between simulated and real world trajectories. In contrast, we aim to learn policies that maximize performance over a diverse distribution of environments where the task is feasible, as a way of minimizing the interactions with the real robot system. Similarly, Ramos et al~\cite{ramos2019bayessim} adopt a likelihood-free inference method that computes the posterior distribution of simulation parameter using real-world trajectories.  Rajeswaran et al~\cite{rajeswaran2016epopt} propose a related approach for learning robust policies over a distribution of simulator models. The proposed approach, based on the the $\epsilon$-percentile conditional value at risk (CVaR) ~\cite{tamar2015optimizing} objective, improves the policy performance on a small proportion of environments where the policy performs the worst. The authors propose to update the distribution of simulation models to maximize the likelihood of real-world trajectories, via Bayesian inference.
In Section~\ref{sec:result} we combine our method with the CVaR objective to encourage diversity of the learned DR distribution.

Related to learning the DR distribution, Paul et al~\cite{fpo_opt} propose using Bayesian Optimization (BO) to update from the simulation model distribution. This is done by evaluating the improvement over the current policy by using a policy gradient algorithm with data sampled from the current simulator distribution. The parameters of the simulator distribution for the next iteration are selected to maximize said improvement. Mehta et al~\cite{mehta2019adr} direct the policy search towards harder instances given the domain randomization ranges.
Given a set reference environment, the domain randomization distribution is modified in order to produce simulated environments within which the learned policy behaves differently than the original policy. Their method explores the parameter space by learning a discriminative reward computed from discrepancies in policy roll-outs, generated in randomized and reference environments.

Other related methods rely on policies that are conditioned on \emph{context}: variables used to represent unobserved physical properties that can be varied in the simulator, either explicitly or implicitly. For example, Chen et al~\cite{chen2018hardware} propose learning a policy conditioned on the hardware properties of the robot, encoded as a vector $v_h$. These represent variations on the dynamics of the environment (i.e. friction, mass), drawn from a fixed simulator distribution. When explicit, the context is equal to the simulator parameters. When implicit, the mapping between context vectors and simulator environments is learned during training, using policy optimization. At test time, when the true context is unknown, the context vector $v_h$ that is fed as input to the policy is obtained by gradient descent on the task performance objective. Similarly, Yu et al~\cite{strategy_optim} propose training policies conditioned on simulator parameters (explicitly), then optimizing the context vector alone to maximize performance at test time. Training is done by collecting trajectories on a fixed simulation model distribution. The argument of the authors is that searching over context vectors is easier than searching over policy parameters. The proposed method relies on population-based gradient-free search for optimizing the context vector to maximize task performance. Our method follows a similar approach, but we focus on learning the training distribution. Rakelly et al~\cite{rakelly2019efficient} also use context-conditioned policies, where the context is implicitly encoded into a vector $z$. During the training phase, their proposed algorithm improves on the performance of the policy while learning a probabilistic mapping from trajectory data to context vectors. At test time, the learned mapping is used for online inference of the context vector. This is similar in spirit to the Universal Policies with Online System Identification method~\cite{yu2017preparing}, which instead uses deterministic context inference with an explicit context encoding. Again, these methods use a fixed DR distribution and could benefit from adapting it during training, as we propose in this work.

\section{Problem Statement}
We consider parametric Markov Decision Processes (MDPs) ~\cite{sutton2018reinforcement}.
An MDP $\mathcal{M}$ is defined by the tuple
$\langle \mathcal{S},\mathcal{A}, p, r,\gamma, \rho_0\rangle$, where $\mathcal{S}$ is the set of possible states and $\mathcal{A}$ is the set of actions, $p$ : $\mathcal{S} \times \mathcal{A} \times \mathcal{S} \xrightarrow{} \mathbb{R}$, encodes the state transition dynamics, $r$ :
$\mathcal{S} \times \mathcal{A} \xrightarrow{} \mathbb{R}$
is the task-dependent reward function, $\gamma$ is a discount factor, and
$\rho_0$ : $\mathcal{S} \xrightarrow{} \mathbb{R} $ is the initial state distribution.
Let $s_t$ and $a_t$ be the state and action taken at time $t$. At the beginning of each episode, $s_0 \sim \rho_0(.)$. Trajectories $\tau$ are obtained by iteratively sampling actions using the current policy, $\pi$, $a_t \sim \pi (a_t | s_t)$ and evaluating next states according to the transition dynamics $s_{t+1} \sim p(s_{t+1} | s_t, a_t, z)$, where $z$ are the parameters of the dynamics.
Given an MDP $\mathcal{M}$, the goal is then to learn policy $\pi$ to maximize the expected sum of rewards $J_{\mathcal{M}} (\pi) = \mathbb{E}_{\tau}\left[R(\tau) | \pi \right] = \mathbb{E}_{\tau}\left[\sum_{t=0}^\infty \gamma^t r_t\right]$, where $r_t=r(s_t,a_t)$.

In our work, we aim to maximize performance over a \emph{distribution} of MDPs, each described by a \emph{context} vector $z$ representing the variables that change over the distribution: changes in transition dynamics, rewards, initial state distribution, etc. Thus, our objective is to maximize $\mathbb{E}_{z \sim p_(z)}\left[J_{\mathcal{M}_{z}} (\pi)\right]$, where $p(z)$ is the domain randomization distribution. Similar to~\cite{strategy_optim,chen2018hardware, rakelly2019efficient}, we condition the policy on the context vector, $\pi({a_t|s_t, z})$. In the experiments reported in this paper, we let $z$ encode the parameters of the transition model in a physically based simulator; e.g. mass, friction or damping.

\section{Proposed Method}
\label{sec:method}
We introduce \textbf{LSDR} (Learning the Sweet-spot Distribution Range) algorithm for concurrently learning a domain randomization distribution and a robust policy that maximizes performance over it. LSDR requires a prior distribution $p(z)$, which is a guess of what the domain randomization ranges should be. Instead of directly sampling from $p(z)$, we use a surrogate distribution $p_{\phi}(z)$, with trainable parameters $\phi$. Our goal is to find appropriate parameters $\phi$ to optimize $\pi(\cdot| s, z)$. LSDR proceeds by updating the policy with trajectories sampled from $p_{\phi}(z)$, and updating the $\phi$ based on the performance of the policy $p(z)$. To avoid the collapse of the learned distribution, we use a regularizer that encourages the distribution to be diverse. The idea is to sample more data from environments where improvement of the policy is possible, without collapsing to environments that are trivial to solve.
We summarize our training and testing procedure in Algorithm~\ref{alg:LSDR-policy} and ~\ref{alg:LSDR-distr}.
In our experiments, we use Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} for the $\textrm{UpdatePolicy}$ procedure in Algorithm~(\ref{alg:LSDR-policy}).

\begin{algorithm}[th]
   \caption{\small Learning the policy and training distribution}
   \label{alg:LSDR-policy}
\begin{algorithmic}
   \REQUIRE testing distribution $p(z)$, initial parameters of the learned distribution $\phi$, initial policy $\pi$, buffer size $B$, total iterations $N$
    \FOR {$i \in \{1,...,N\}$}
    \STATE     $z \sim p_{\phi}(z)$
    \STATE     $s_0 \sim \rho_0(s)$
    \STATE     $\mathcal{B} = \{\}$
    \WHILE{    $|\mathcal{B}| < B$}
    \STATE         $a_{t} \sim \pi(a_{t}| s_{t}, z)$
    \STATE         $s_{t+1}, r_{t} \sim p(s_{t+1}, r_{t} | s_{t}, a_{t}, z)$
    \STATE         append $(s_{t}, z_k, a_{t}, r_{t}, s_{t+1})$ to $\mathcal{B}$
    \IF{         $s'$ is terminal}
    \STATE             $z \sim p_{\phi}(z)$
    \STATE             $s_{t+1} \sim \rho_0(s)$
    \ENDIF
    \STATE       $s_t \leftarrow s_{t+1}$
    \ENDWHILE
    \STATE     $\phi \leftarrow \textrm{UpdateDistribution}(\phi, p(z), \pi)$
    \STATE     $\pi \leftarrow \textrm{UpdatePolicy}(\pi, \mathcal{B})$
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\begin{algorithm}[th]
   \caption{\small UpdateDistribution}
   \label{alg:LSDR-distr}
\begin{algorithmic}
   \REQUIRE learned distribution parameters $\phi$, testing distribution $p(z)$, policy $\pi$, total iterations $M$, total trajectory samples $K$
    \FOR {$i \in \{1,...,M\}$}
    \STATE sample $z_{1:K}$ from $p(z)$
    \STATE Obtain Monte-Carlo estimate of $\mathcal{L}_{DR}(\phi)$ by executing $\pi$ on environments with $z_{1:K}$
    \STATE $\phi \leftarrow \phi + \lambda \nabla_{\phi} (\mathcal{L}_{DR}(\phi) - \alpha D_{KL}(p(z)||p_{\phi}(z)))$
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Learning the Sweet-spot Distribution Range}
The goal of our method is to find a training distribution $p_{\phi}(z)$ to maximize the expected reward of the policy over $p(z)$, while gradually reducing the sampling frequency of environments where the task is not solvable\footnote{In this work, we consider a task solvable if there exists a policy that brings the environment to a set of desired goal states.}. Such a situation is common in physics-based simulations, where a suboptimal selection of simulation parameters may lead to environments where the task is impossible due to physical limits or unstable simulations.

We start by assuming that the prior distribution $p(z)$ has wide but bounded support, such that we get a distribution of solvable and unsolvable tasks. To update the training distribution, we use an objective of the following form
\begin{equation}
    \underset{\phi}{\arg\max\,}\mathcal{L}_{DR}(\phi) - \\ \alpha D_{KL}(p(z)||p_{\phi}(z))
    \label{eq:dr_objective}
\end{equation}
where the first term is designed to encourage improvement on environments that are more likely to be solvable, while the second term is a regularizer that keeps the distribution from collapsing. In our experiments, we set $\mathcal{L}_{DR}(\phi) = \mathbb{E}_{z \sim p(z)} [J_{\mathcal{M}_{z}} (\pi)\log(p_{\phi}(z))]$ so that the distribution is driven towards high reward regions.
If we use the performance of the policy as a way of determining whether the task is solvable for a given context $z$, then a trivial solution would be to make $p_{\phi}(z)$ concentrate on few easy environments. The second term in Eq.~(\ref{eq:dr_objective}) helps to avoid this issue by penalizing distributions that deviate too much from $p(z)$, which is assumed to be wide.

To estimate the gradient of Eq.~(\ref{eq:dr_objective}) with respect to $\phi$, we use the log-derivative score function gradient estimator~\cite{fu2006gradient}, resulting in the following Monte-Carlo update :

\begin{equation}
\begin{split}
\phi \leftarrow \phi + \lambda \bigg[ \frac{1}{K} \sum_{i=1}^K \bigg( J_{\mathcal{M}_{z_i}}(\pi) \nabla_{\phi}\log(p_{\phi}(z_i))  \bigg)  - \\
\alpha \nabla_{\phi}D_{KL}(p(z)||p_{\phi}(z))  \bigg]
\end{split}
\end{equation}

where $z_{i} \sim p(z)$. Updating $\phi$ with samples from the distribution we are learning, can be problematic as we may not get information about the performance of the policy in low probability contexts under $p_{\phi}(z)$.

\begin{figure}[t!]
\centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_no_entropy.pdf}
        \caption{}
        \label{fig:no_entropy}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_train_sampl.pdf}
        \caption{}
        \label{fig:train_sampl}
    \end{subfigure}
    \caption{\small Learned torso size distribution for Hopper. Figure ~\ref{fig:no_entropy} shows distribution learned without the KL divergence regularizer and Figure~\ref{fig:train_sampl} shows the distribution learned while sampling from train distribution.}
    \label{fig:not-learning}
\end{figure}

Empirically, we observed this issue in our single dimensional experiments with discrete distributions, where the distribution collapses as it never sees low-probability samples (see Figure~\ref{fig:not-learning}). An option in that case would be to use samples from $p(z)$ to evaluate the first term of our objective, which biases the gradients but avoids the distribution collapse. For the experiments with multivariate Gaussian distributions changing the sampling distribution was not required.

To ensure that the two terms in Eq.~(\ref{eq:dr_objective}) have similar scale, we standardize the evaluations of $J_{\mathcal{M}_{z_i}}$ with exponentially averaged batch statistics and set $\alpha$ via hyper-parameter search.

\section{Experimental Results}
\label{sec:result}
We evaluate the impact of learning the DR distribution on two standard benchmark locomotion tasks: Hopper and Half-cheetah, see Fig.~\ref{fig:mujoco-viz}, from the MuJoCo tasks in the OpenAI Gym suite~\cite{gym2016}. We use an explicit encoding of the context vector $z$, corresponding to the torso size, density, foot friction and joint damping of the environments. Our experiments study both single-dimensional as well as multidimensional domain randomization contexts. For the first, we use discrete distributions and run experiments for each context dimension separately. For latter, we parametrize the domain randomization distribution with multivariate Gaussians.

We selected the prior $p(z)$ as an uniform distribution over ranges that include both solvable and unsolvable environments. In the case of discrete distributions, we discretize the support of $p(z)$ with 100 bins, and initialize $p_{\phi}(z)$ to be uniform; i.e. equivalent to $p(z)$. When sampling from this distribution, we first select a bin according to the discrete probabilities, then select a continuous context value uniformly at random from the ranges of the corresponding bin. For the multidimensional context experiments, the mean of the Gaussian distribution is initialized with the mean of test range, and the variance is initialized to $1/10^{th}$ of the variance of $p(z)$, so that most of the mass of the distribution falls within its support.

We compare the test-time jump-start and asymptotic performance of policies learned with $p_{\phi}(z)$ (learned domain randomization) and $p(z)$ (fixed domain randomization). At test time, we sample (uniformly at random) a test set of samples from the support of $p(z)$, $50$ for single-dimensional and $100$ for multidimensional contexts, and fine-tune the policy with the parameters obtained at training time. The questions we aim to answer with our experiments are: 1) does learning policies with wide DR distributions affect the performance of the policy in the environments where the task is solvable? 2) does learning the DR distribution converge to the actual ranges where the task is solvable? 3) Is learning the DR distribution beneficial?

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.15\textwidth}
        \includegraphics[width=\textwidth]{Figures/hophop.png}
        \caption{}
        \label{fig:hophop}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.23\textwidth}
        \includegraphics[width=\textwidth]{Figures/half-cheetah.png}
        \caption{}
        \label{fig:halfcheetah}
    \end{subfigure}
    \caption{\small Illustrations of the 2D simulated robot models used in the experiments. The hopper (a) and half-cheetah (b) tasks, present more challenging environments when varying dynamics.}\label{fig:mujoco-viz}
\end{figure}

\subsection{Results}
\noindent\textbf{Learned Distribution Ranges: }
Table~\ref{table:learned_ranges} shows the ranges for $p(z)$ and the final equivalent ranges for the distributions found by our method, for the single-dimensional discrete distribution learning experiments.
Figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna} \ref {fig:hop-dyna-gaussian} and \ref {fig:cheet-dyna-gaussian} show the evolution of $p_{\phi}(z)$ during training, using our method. In figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna}, each plot corresponds to a separate domain randomization experiment, where we randomized one different simulator parameter while keeping the rest fixed.
\begin{table*}[t!]
\centering
\resizebox{0.65\textwidth}{!}{
\begin{tabular}{c c c | c }
\multicolumn{3}{c}{} \\
\midrule
Environment & Parameters & Initial Train/Test Distribution & Converged Ranges \\
\midrule
\multirow{4}{*}{Hopper}    & Torso size    & $[-0.1, 0.2]$    &  $[0.0015, 0.09]$\\
                           & Density       & $[10, 5000]$     &  $[400, 4000]$\\
                           & Friction      & $[-3.0, 10.0]$   &  $[-3.0, 3.0]$\\
                           & Joint Damping & $[-10.0, 30.0]$  &  $[-1.0, 26]$\\
\bottomrule
\multirow{4}{*}{Half-Cheetah} & Torso size    & $[-0.1, 2.0]$    &  $[0.055, 0.52]$\\
                           & Density       & $[10, 5000]$     &  $[285, 2420]$\\
                           & Friction      & $[-3.0, 10.0]$   &  $[0.65, 4.39]$\\
                           & Joint Damping & $[-10.0, 30.0]$  &  $[-1.65, 12.9]$\\
\bottomrule
\end{tabular}
}
\caption{\small Ranges of parameters for each environment, in the beginning of training and the equivalent ranges found by the algorithm, obtained by fitting an uniform distribution to the final learned distribution.}
\label{table:learned_ranges}
\end{table*}

Initially, each of these distribution is uniform. As the agent becomes better over the training distribution, it becomes easier to discriminate between promising environments (where the task is solvable) and impossible ones where rewards stay at low values. After around $1500$ epochs, the distributions have converged to their final distributions. For Hopper, the learned distributions corresponds closely with the environments where we can find a policy using vanilla policy gradient methods from scratch. To determine the consistency of these results, we ran the Hopper torso size experiment 7 times, and fitted the parameters of a uniform distribution to the resulting $p_{\phi}(z)$. The mean ranges ($\pm$ one standard deviation) across the 7 experiments were $[0.00086 \pm 0.00159, 0.09275 \pm 0.00342]$, which provides some evidence for the reproducibility of our method.
For figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna}, all plots correspond to a multidimensional context experiment. Since the contexts are being learned jointly, the final distributions correspond to the joint distribution of all contexts. These plots correspond to the 2D projection of each context distribution.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_distr.pdf}
        \label{fig:hop_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_density_distr.pdf}
        \label{fig:hop_density_distr}
        \vspace{-1.5em}
        \caption{\small Density}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_friction_distr.pdf}
        \label{fig:hop_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_damping_distr.pdf}
        \label{fig:hop_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned domain randomization distribution $p_{\phi}(z)$ over time for \textbf{Hopper}.  Each plot corresponds to a different experiments where we kept the other simulator parameters fixed at their default values. Lighter color corresponds to higher probabilities.}\label{fig:hop-dyna}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_torso_distr.pdf}
        \label{fig:cheet_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_density_distr.pdf}
        \label{fig:cheet_density_distr}
        \vspace{-1.5em}
        \caption{\small Density}
    \end{subfigure}
        ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_friction_distr.pdf}
        \label{fig:cheet_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_damping_distr.pdf}
        \label{fig:cheet_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned domain randomization distribution $p_{\phi}(z)$  over time for \textbf{Half-Cheetah}. Note how the learned distribution assigns low probability to physically implausible environments (negative or large mass, large friction etc)}\label{fig:cheet-dyna}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Torso.pdf}
        \label{fig:hop_multidim_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Fric.pdf}
        \label{fig:hop_multidim_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Damp.pdf}
        \label{fig:hop_multidim_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned multidimensional domain randomization distribution $p_{\phi}(z)$ for \textbf{Hopper}.}
    \label{fig:hop-dyna-gaussian}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Torso.pdf}
        \label{fig:cheet_multi_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Fric.pdf}
        \label{fig:cheet_multi_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Damp.pdf}
        \label{fig:cheet_multi_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned multidimensional domain randomization distribution $p_{\phi}(z)$ for \textbf{Half-Cheetah}.}
    \label{fig:cheet-dyna-gaussian}
\end{figure}

\noindent\textbf{Learned vs Fixed Domain Randomization: }
We compare the jumpstart and asymptotic performance between learning the domain randomization distribution and keeping it fixed. Our results show our method, using PPO as the policy optimizer (\textbf{LSDR}) vs keeping the domain randomization distribution fixed (\textbf{Fixed-DR}) which corresponds to keeping the domain randomization distribution fixed.
We ran the same experiments for \textit{Hopper} and \textit{Half-Cheetah}.

Figure~\ref{fig:hop-cheet-generalization} depict learning curves when fine tuning the policy at test-time, for torso size randomization. All the methods start with the same random seed at training time across $100$ seeds. The policies are trained for $3000$ epochs, where we collect $B=4000$ samples per epoch for the policy update. For the distribution update, we collect $K=10$ additional trajectories and run the gradient update for $M=10$ steps (without re-sampling new trajectories).
To ensure fairness in terms of the sampled contexts, we report the comparison over the full test range (the support of $p(z)$). From these results, it is clear that learning the domain randomization distribution improves on the jump-start and asymptotic performance over using fixed domain randomization. In this case we test the policy generalization on the wide test distribution, which includes contexts that are not solvable; i.e. the optimal policy found by vanilla-PPO does not result in successful locomotion on the such extreme range\footnote{Successful policies on Hopper obtain cumulative rewards of at least $2500$. For Half-Cheetah, the rewards are greater than 0 when the robot successfully moves forward.}. For the multidimensional Hopper experiments, the results are shown in Figure~\ref{fig:hopper_grid_comparison}. In this case, we evaluated the performance of the policy at the last training iteration, with 10 rollouts per context. For the independently trained policies, we ran PPO for each context in the grid for $1000$ epochs, each with $4000$ steps of experience. While our method does not match the performance of the exhaustive grid training, it is clearly better than training on an uniform DR distribution. Furthermore, our method observes considerably less experience data ($1.2e7$ steps) than the exhaustive grid training ($4e9$ steps). We expect to get better performance by increasing the per-epoch experience, as done in~\cite{yu2017preparing}.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.4\textwidth}
        \includegraphics[width=\textwidth]{Figures/generalization_hopper_multidim.pdf}
        \label{fig:hopper-generalization}
        \vspace{-1em}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.4\textwidth}
        \includegraphics[width=\textwidth]{Figures/generalization_cheetah_multidim.pdf}
        \label{fig:cheetah-generalization}
        \vspace{-1em}
    \end{subfigure}
    \caption{\small Comparison of test-time performance on Hopper (top) and Half-Cheetah (bottom) between fixed vs learned domain randomization in the policy inputs. Note that running experiments on the test range, will result in environments that may not necessarily be solvable and this results in additional variance on the performance across seeds.}
    \label{fig:hop-cheet-generalization}
\end{figure}

\noindent\textbf{Using a different policy optimizer}
\label{sec:epopt}
We also experimented with using EPOpt-PPO~\cite{rajeswaran2016epopt} as the policy optimizer in Algorithm~(\ref{alg:LSDR-policy}). The motivation for this is to mitigate the bias towards environments with higher cumulative rewards $J_{M_z}(\pi)$ early during training. EPOpt encourages improving the policy on the worst performing environments, at the expense of collecting more data per epoch. At each training epoch, we obtain $100$ samples from the training distribution and obtain trajectories by executing the policy once on each of the corresponding environments. From the resulting $100$ trajectories, we use the $10\%$ trajectories that resulted in the lowest rewards to fill the buffer for a PPO policy update, discarding the rest of the trajectories.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.4\textwidth]{Figures/epopt_minmax-cropped.pdf}
    \caption{\small Learning curves for torso DR on Hopper using EPOpt. Lines represent mean performance, while the shaded regions correspond to the maximum and minimum performance (smoothed with a 5th order Savitzky-Golay filter and a window of 10 epochs)}
    \label{fig:epopt_minmax}
\end{figure}

Figure~\ref{fig:epopt_minmax} compares the effect of learning the domain randomization distribution vs using a fixed wide range in this setting. We found that learning the domain randomization distribution resulted in faster convergence to high reward policies over the evaluation range [0.01, 0.09]: our method attains cumulative rewards greater than 3000 in about 1/2 the epochs required with uniform DR. Our method also results in better asymptotic performance: our method finds policies with better and more consistent performance (less variance) as seen in Figure~\ref{fig:epopt_minmax}. We believe this could be a consequence of lower variance in the policy gradient estimates, as the the learned $p_{\phi}(z)$ has lower variance than $p(z)$. Interestingly, using EPOpt resulted in a distribution with a wider torso size range than vanilla PPO, from approximately $0.0$ to $0.14$, demonstrating that optimizing worst case performance does help in alleviating the bias towards high reward environments.

\section{Discussion}
By allowing the agent to learn a good representative distribution, we are able to learn to solve difficult control tasks that heavily rely on a good initial domain randomization range.
Our main experimental validation of domain randomization distribution learning is in the domain of simulated robotic locomotion.
As shown in our experiments, our method is not sensitive to the initial domain randomization distribution and is able to converge to a more diverse range, while staying within the feasible range. We experimented with single dimensional discrete and multivariate Gaussian distributions for learning the DR distributions. Our results show the benefit of learning a domain randomization distribution when compared to the previously accepted practice: keeping it fixed. Further extension of this work will explore better objectives for learning the distribution and more flexible parameterizations for the DR distribution.

Unscaled rewards, non-linear dynamics and many other complex factors make the manual setting of DR ranges ill-suited for human intuition. Automating this step is crucial to improving the robustness of simulation-to-real transfer and solving today's wide range of robotics challenges; our method is a new step in this fruitful direction.

\section{Introduction}
\label{Introduction}
Machine learning, and deep reinforcement learning (deep-RL) in particular, is a promising
approach for learning controllers or action policies for complex systems where analytic methods are elusive.
The use of robot simulators for machine learning is driven by a compromise between the large data requirements of Deep-RL and the cost of collecting experience on physical robots systems. Except for simple robot systems in controlled environments, however, real robot experience may not correspond to situations that can be simulated; an issue known as the \emph{reality gap} ~\cite{jakobi1995noise}. This gap limits the utility of simulation-based learning.

\emph{Domain randomization} (DR) aims to address the reality gap by training policies to maximize performance over a diverse set of simulation models, where the parameters of each model are sampled randomly from a given task distribution. The effectiveness of DR has been demonstrated by training controllers purely in simulation that subsequently transfer successfully to real robot systems~\cite{peng2018sim, openai2018learning} and by fine-tuning the DR distributions with real world data ~\cite{chen2018hardware}.

While successful, an aspect that has not been addressed in depth is the selection of the domain randomization distribution.
The DR distribution should be selected carefully to ensure the ability to generalize within the simulated experience. Selecting a distribution that is too wide may result in an under-performing policy (see Fig.~\ref{fig:hopper_fixed_dr}) or slow convergence (see Fig.~\ref{fig:epopt_minmax}) . Conversely, training  on  a distribution  that  is  too  narrow  leads  to  policies  that  don't generalize well~\cite{rajeswaran2016epopt,packer2018assessing}. Most of the existing methods~\cite{rajeswaran2016epopt, chebotar2018closing, ramos2019bayessim}, assume the availability of real robot data. In this work, we study the extreme case where no real robot data is available, similar to~\cite{openai2018learning}, in which the appropriate selection of a DR distribution is crucial.

\begin{figure}[t!]
\centering
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid-crop.png}
        \caption{\small Average rewards with independently trained policies per context}
        \label{fig:hopper_target_rewards}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_fixed_dr-crop.png}
        \caption{\small Average rewards with fixed, uniform domain randomization}
        \label{fig:hopper_fixed_dr}
    \end{subfigure}
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_learned_dr-crop.png}
        \caption{\small Average rewards with learned Gaussian domain randomization}
        \label{fig:hopper_learned_dr}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.21\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_grid_gaussian_confidence-crop.png}
        \caption{\small Confidence regions of the learned distribution for 1: 68\%, 2: 95\%, and 3: 99\% confidence}
        \label{fig:hopper_confidence}
    \end{subfigure}
    \caption{\small Figure~\ref{fig:hopper_target_rewards} shows the maximum rewards obtained by exhaustively running PPO on a grid of simulation parameters, ~\ref{fig:hopper_fixed_dr} shows the performance of a policy trained with uniform domain randomization over the grid, ~\ref{fig:hopper_learned_dr} results from our method when learning the parameters of a Gaussian domain randomization distribution (whose confidence regions are shown in ~\ref{fig:hopper_confidence}). For fixed and learned DR the reward averages were obtained by evaluating the policy with 10 different rollouts.}

    \label{fig:hopper_grid_comparison}
\end{figure}

We propose to learn the parameters of the simulator distribution, such that the policy is trained under an evolving sequence of simulators that gives priority to environments where the current policy can achieve high performance. Our proposed method aims to find a compromise between variety and performance, by optimizing the DR distribution to focus on environments where high rewards are likely, while regularizing the distribution parameters via a KL-divergence objective (explained in Sec.~\ref{sec:method}) and using more data from the worst performing simulators (see the experiments in Sec~\ref{sec:epopt}). Our algorithm learns the domain randomization distribution while simultaneously optimizing the policy to maximize performance over the learned distribution. Figure~\ref{fig:hopper_learned_dr} shows our resuls towards finding the \emph{sweet-spot} distribution.
We evaluate our method on a variety of control problems from the OpenAI Gym suite of benchmarks~\cite{gym2016}. We find that our method is able to improve on the performance of fixed domain randomization. Furthermore, we demonstrate our model's robustness to initial simulator distribution parameters, showing that our method repeatably converges to similar domain randomization distributions across different experiments.

\section{Related Work}

Developing robust or generalizable controllers has a long and rich history beyond the scope of this paper~\cite{Zames1981,zhou1998robustControl}.
Caruana was on of the first to deeply explore the advantages of multi-task learning in a robotics context~\cite{Caruana93multitasklearning}. Baxter considered multi-task learning in a Bayesian/information-theoretic context~\cite{Baxter1997}.

Packer et al~\cite{packer2018assessing} present an empirical study of generalization in Deep-RL, testing interpolation and extrapolation performance of state-of-the-art algorithms when varying simulation parameters in control tasks. The authors provide an experimental assessment of generalization under varying training and testing distributions. Our work extends these results by providing results for the case when the training distribution parameters are learned and change during policy training.

Chebotar et al~\cite{chebotar2018closing} propose training policies on a distribution of simulators, whose parameters are fit to real-world data. Their proposed algorithm switches back and forth between optimizing the policy under the DR distribution and updating the DR distribution by minimizing the discrepancy between simulated and real world trajectories. In contrast, we aim to learn policies that maximize performance over a diverse distribution of environments where the task is feasible, as a way of minimizing the interactions with the real robot system. Similarly, Ramos et al~\cite{ramos2019bayessim} adopt a likelihood-free inference method that computes the posterior distribution of simulation parameter using real-world trajectories.  Rajeswaran et al~\cite{rajeswaran2016epopt} propose a related approach for learning robust policies over a distribution of simulator models. The proposed approach, based on the the $\epsilon$-percentile conditional value at risk (CVaR) ~\cite{tamar2015optimizing} objective, improves the policy performance on a small proportion of environments where the policy performs the worst. The authors propose to update the distribution of simulation models to maximize the likelihood of real-world trajectories, via Bayesian inference.
In Section~\ref{sec:result} we combine our method with the CVaR objective to encourage diversity of the learned DR distribution.

Related to learning the DR distribution, Paul et al~\cite{fpo_opt} propose using Bayesian Optimization (BO) to update from the simulation model distribution. This is done by evaluating the improvement over the current policy by using a policy gradient algorithm with data sampled from the current simulator distribution. The parameters of the simulator distribution for the next iteration are selected to maximize said improvement. Mehta et al~\cite{mehta2019adr} direct the policy search towards harder instances given the domain randomization ranges.
Given a set reference environment, the domain randomization distribution is modified in order to produce simulated environments within which the learned policy behaves differently than the original policy. Their method explores the parameter space by learning a discriminative reward computed from discrepancies in policy roll-outs, generated in randomized and reference environments.

Other related methods rely on policies that are conditioned on \emph{context}: variables used to represent unobserved physical properties that can be varied in the simulator, either explicitly or implicitly. For example, Chen et al~\cite{chen2018hardware} propose learning a policy conditioned on the hardware properties of the robot, encoded as a vector $v_h$. These represent variations on the dynamics of the environment (i.e. friction, mass), drawn from a fixed simulator distribution. When explicit, the context is equal to the simulator parameters. When implicit, the mapping between context vectors and simulator environments is learned during training, using policy optimization. At test time, when the true context is unknown, the context vector $v_h$ that is fed as input to the policy is obtained by gradient descent on the task performance objective. Similarly, Yu et al~\cite{strategy_optim} propose training policies conditioned on simulator parameters (explicitly), then optimizing the context vector alone to maximize performance at test time. Training is done by collecting trajectories on a fixed simulation model distribution. The argument of the authors is that searching over context vectors is easier than searching over policy parameters. The proposed method relies on population-based gradient-free search for optimizing the context vector to maximize task performance. Our method follows a similar approach, but we focus on learning the training distribution. Rakelly et al~\cite{rakelly2019efficient} also use context-conditioned policies, where the context is implicitly encoded into a vector $z$. During the training phase, their proposed algorithm improves on the performance of the policy while learning a probabilistic mapping from trajectory data to context vectors. At test time, the learned mapping is used for online inference of the context vector. This is similar in spirit to the Universal Policies with Online System Identification method~\cite{yu2017preparing}, which instead uses deterministic context inference with an explicit context encoding. Again, these methods use a fixed DR distribution and could benefit from adapting it during training, as we propose in this work.

\section{Problem Statement}
We consider parametric Markov Decision Processes (MDPs) ~\cite{sutton2018reinforcement}.
An MDP $\mathcal{M}$ is defined by the tuple
$\langle \mathcal{S},\mathcal{A}, p, r,\gamma, \rho_0\rangle$, where $\mathcal{S}$ is the set of possible states and $\mathcal{A}$ is the set of actions, $p$ : $\mathcal{S} \times \mathcal{A} \times \mathcal{S} \xrightarrow{} \mathbb{R}$, encodes the state transition dynamics, $r$ :
$\mathcal{S} \times \mathcal{A} \xrightarrow{} \mathbb{R}$
is the task-dependent reward function, $\gamma$ is a discount factor, and
$\rho_0$ : $\mathcal{S} \xrightarrow{} \mathbb{R} $ is the initial state distribution.
Let $s_t$ and $a_t$ be the state and action taken at time $t$. At the beginning of each episode, $s_0 \sim \rho_0(.)$. Trajectories $\tau$ are obtained by iteratively sampling actions using the current policy, $\pi$, $a_t \sim \pi (a_t | s_t)$ and evaluating next states according to the transition dynamics $s_{t+1} \sim p(s_{t+1} | s_t, a_t, z)$, where $z$ are the parameters of the dynamics.
Given an MDP $\mathcal{M}$, the goal is then to learn policy $\pi$ to maximize the expected sum of rewards $J_{\mathcal{M}} (\pi) = \mathbb{E}_{\tau}\left[R(\tau) | \pi \right] = \mathbb{E}_{\tau}\left[\sum_{t=0}^\infty \gamma^t r_t\right]$, where $r_t=r(s_t,a_t)$.

In our work, we aim to maximize performance over a \emph{distribution} of MDPs, each described by a \emph{context} vector $z$ representing the variables that change over the distribution: changes in transition dynamics, rewards, initial state distribution, etc. Thus, our objective is to maximize $\mathbb{E}_{z \sim p_(z)}\left[J_{\mathcal{M}_{z}} (\pi)\right]$, where $p(z)$ is the domain randomization distribution. Similar to~\cite{strategy_optim,chen2018hardware, rakelly2019efficient}, we condition the policy on the context vector, $\pi({a_t|s_t, z})$. In the experiments reported in this paper, we let $z$ encode the parameters of the transition model in a physically based simulator; e.g. mass, friction or damping.

\section{Proposed Method}
\label{sec:method}
We introduce \textbf{LSDR} (Learning the Sweet-spot Distribution Range) algorithm for concurrently learning a domain randomization distribution and a robust policy that maximizes performance over it. LSDR requires a prior distribution $p(z)$, which is a guess of what the domain randomization ranges should be. Instead of directly sampling from $p(z)$, we use a surrogate distribution $p_{\phi}(z)$, with trainable parameters $\phi$. Our goal is to find appropriate parameters $\phi$ to optimize $\pi(\cdot| s, z)$. LSDR proceeds by updating the policy with trajectories sampled from $p_{\phi}(z)$, and updating the $\phi$ based on the performance of the policy $p(z)$. To avoid the collapse of the learned distribution, we use a regularizer that encourages the distribution to be diverse. The idea is to sample more data from environments where improvement of the policy is possible, without collapsing to environments that are trivial to solve.
We summarize our training and testing procedure in Algorithm~\ref{alg:LSDR-policy} and ~\ref{alg:LSDR-distr}.
In our experiments, we use Proximal Policy Optimization (PPO)~\cite{schulman2017proximal} for the $\textrm{UpdatePolicy}$ procedure in Algorithm~(\ref{alg:LSDR-policy}).

\begin{algorithm}[th]
   \caption{\small Learning the policy and training distribution}
   \label{alg:LSDR-policy}
\begin{algorithmic}
   \REQUIRE testing distribution $p(z)$, initial parameters of the learned distribution $\phi$, initial policy $\pi$, buffer size $B$, total iterations $N$
    \FOR {$i \in \{1,...,N\}$}
    \STATE     $z \sim p_{\phi}(z)$
    \STATE     $s_0 \sim \rho_0(s)$
    \STATE     $\mathcal{B} = \{\}$
    \WHILE{    $|\mathcal{B}| < B$}
    \STATE         $a_{t} \sim \pi(a_{t}| s_{t}, z)$
    \STATE         $s_{t+1}, r_{t} \sim p(s_{t+1}, r_{t} | s_{t}, a_{t}, z)$
    \STATE         append $(s_{t}, z_k, a_{t}, r_{t}, s_{t+1})$ to $\mathcal{B}$
    \IF{         $s'$ is terminal}
    \STATE             $z \sim p_{\phi}(z)$
    \STATE             $s_{t+1} \sim \rho_0(s)$
    \ENDIF
    \STATE       $s_t \leftarrow s_{t+1}$
    \ENDWHILE
    \STATE     $\phi \leftarrow \textrm{UpdateDistribution}(\phi, p(z), \pi)$
    \STATE     $\pi \leftarrow \textrm{UpdatePolicy}(\pi, \mathcal{B})$
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\begin{algorithm}[th]
   \caption{\small UpdateDistribution}
   \label{alg:LSDR-distr}
\begin{algorithmic}
   \REQUIRE learned distribution parameters $\phi$, testing distribution $p(z)$, policy $\pi$, total iterations $M$, total trajectory samples $K$
    \FOR {$i \in \{1,...,M\}$}
    \STATE sample $z_{1:K}$ from $p(z)$
    \STATE Obtain Monte-Carlo estimate of $\mathcal{L}_{DR}(\phi)$ by executing $\pi$ on environments with $z_{1:K}$
    \STATE $\phi \leftarrow \phi + \lambda \nabla_{\phi} (\mathcal{L}_{DR}(\phi) - \alpha D_{KL}(p(z)||p_{\phi}(z)))$
    \ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Learning the Sweet-spot Distribution Range}
The goal of our method is to find a training distribution $p_{\phi}(z)$ to maximize the expected reward of the policy over $p(z)$, while gradually reducing the sampling frequency of environments where the task is not solvable\footnote{In this work, we consider a task solvable if there exists a policy that brings the environment to a set of desired goal states.}. Such a situation is common in physics-based simulations, where a suboptimal selection of simulation parameters may lead to environments where the task is impossible due to physical limits or unstable simulations.

We start by assuming that the prior distribution $p(z)$ has wide but bounded support, such that we get a distribution of solvable and unsolvable tasks. To update the training distribution, we use an objective of the following form
\begin{equation}
    \underset{\phi}{\arg\max\,}\mathcal{L}_{DR}(\phi) - \\ \alpha D_{KL}(p(z)||p_{\phi}(z))
    \label{eq:dr_objective}
\end{equation}
where the first term is designed to encourage improvement on environments that are more likely to be solvable, while the second term is a regularizer that keeps the distribution from collapsing. In our experiments, we set $\mathcal{L}_{DR}(\phi) = \mathbb{E}_{z \sim p(z)} [J_{\mathcal{M}_{z}} (\pi)\log(p_{\phi}(z))]$ so that the distribution is driven towards high reward regions.
If we use the performance of the policy as a way of determining whether the task is solvable for a given context $z$, then a trivial solution would be to make $p_{\phi}(z)$ concentrate on few easy environments. The second term in Eq.~(\ref{eq:dr_objective}) helps to avoid this issue by penalizing distributions that deviate too much from $p(z)$, which is assumed to be wide.

To estimate the gradient of Eq.~(\ref{eq:dr_objective}) with respect to $\phi$, we use the log-derivative score function gradient estimator~\cite{fu2006gradient}, resulting in the following Monte-Carlo update :

\begin{equation}
\begin{split}
\phi \leftarrow \phi + \lambda \bigg[ \frac{1}{K} \sum_{i=1}^K \bigg( J_{\mathcal{M}_{z_i}}(\pi) \nabla_{\phi}\log(p_{\phi}(z_i))  \bigg)  - \\
\alpha \nabla_{\phi}D_{KL}(p(z)||p_{\phi}(z))  \bigg]
\end{split}
\end{equation}

where $z_{i} \sim p(z)$. Updating $\phi$ with samples from the distribution we are learning, can be problematic as we may not get information about the performance of the policy in low probability contexts under $p_{\phi}(z)$.

\begin{figure}[t!]
\centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_no_entropy.pdf}
        \caption{}
        \label{fig:no_entropy}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_train_sampl.pdf}
        \caption{}
        \label{fig:train_sampl}
    \end{subfigure}
    \caption{\small Learned torso size distribution for Hopper. Figure ~\ref{fig:no_entropy} shows distribution learned without the KL divergence regularizer and Figure~\ref{fig:train_sampl} shows the distribution learned while sampling from train distribution.}
    \label{fig:not-learning}
\end{figure}

Empirically, we observed this issue in our single dimensional experiments with discrete distributions, where the distribution collapses as it never sees low-probability samples (see Figure~\ref{fig:not-learning}). An option in that case would be to use samples from $p(z)$ to evaluate the first term of our objective, which biases the gradients but avoids the distribution collapse. For the experiments with multivariate Gaussian distributions changing the sampling distribution was not required.

To ensure that the two terms in Eq.~(\ref{eq:dr_objective}) have similar scale, we standardize the evaluations of $J_{\mathcal{M}_{z_i}}$ with exponentially averaged batch statistics and set $\alpha$ via hyper-parameter search.

\section{Experimental Results}
\label{sec:result}
We evaluate the impact of learning the DR distribution on two standard benchmark locomotion tasks: Hopper and Half-cheetah, see Fig.~\ref{fig:mujoco-viz}, from the MuJoCo tasks in the OpenAI Gym suite~\cite{gym2016}. We use an explicit encoding of the context vector $z$, corresponding to the torso size, density, foot friction and joint damping of the environments. Our experiments study both single-dimensional as well as multidimensional domain randomization contexts. For the first, we use discrete distributions and run experiments for each context dimension separately. For latter, we parametrize the domain randomization distribution with multivariate Gaussians.

We selected the prior $p(z)$ as an uniform distribution over ranges that include both solvable and unsolvable environments. In the case of discrete distributions, we discretize the support of $p(z)$ with 100 bins, and initialize $p_{\phi}(z)$ to be uniform; i.e. equivalent to $p(z)$. When sampling from this distribution, we first select a bin according to the discrete probabilities, then select a continuous context value uniformly at random from the ranges of the corresponding bin. For the multidimensional context experiments, the mean of the Gaussian distribution is initialized with the mean of test range, and the variance is initialized to $1/10^{th}$ of the variance of $p(z)$, so that most of the mass of the distribution falls within its support.

We compare the test-time jump-start and asymptotic performance of policies learned with $p_{\phi}(z)$ (learned domain randomization) and $p(z)$ (fixed domain randomization). At test time, we sample (uniformly at random) a test set of samples from the support of $p(z)$, $50$ for single-dimensional and $100$ for multidimensional contexts, and fine-tune the policy with the parameters obtained at training time. The questions we aim to answer with our experiments are: 1) does learning policies with wide DR distributions affect the performance of the policy in the environments where the task is solvable? 2) does learning the DR distribution converge to the actual ranges where the task is solvable? 3) Is learning the DR distribution beneficial?

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.15\textwidth}
        \includegraphics[width=\textwidth]{Figures/hophop.png}
        \caption{}
        \label{fig:hophop}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.23\textwidth}
        \includegraphics[width=\textwidth]{Figures/half-cheetah.png}
        \caption{}
        \label{fig:halfcheetah}
    \end{subfigure}
    \caption{\small Illustrations of the 2D simulated robot models used in the experiments. The hopper (a) and half-cheetah (b) tasks, present more challenging environments when varying dynamics.}\label{fig:mujoco-viz}
\end{figure}

\subsection{Results}
\noindent\textbf{Learned Distribution Ranges: }
Table~\ref{table:learned_ranges} shows the ranges for $p(z)$ and the final equivalent ranges for the distributions found by our method, for the single-dimensional discrete distribution learning experiments.
Figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna} \ref {fig:hop-dyna-gaussian} and \ref {fig:cheet-dyna-gaussian} show the evolution of $p_{\phi}(z)$ during training, using our method. In figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna}, each plot corresponds to a separate domain randomization experiment, where we randomized one different simulator parameter while keeping the rest fixed.
\begin{table*}[t!]
\centering
\resizebox{0.65\textwidth}{!}{
\begin{tabular}{c c c | c }
\multicolumn{3}{c}{} \\
\midrule
Environment & Parameters & Initial Train/Test Distribution & Converged Ranges \\
\midrule
\multirow{4}{*}{Hopper}    & Torso size    & $[-0.1, 0.2]$    &  $[0.0015, 0.09]$\\
                           & Density       & $[10, 5000]$     &  $[400, 4000]$\\
                           & Friction      & $[-3.0, 10.0]$   &  $[-3.0, 3.0]$\\
                           & Joint Damping & $[-10.0, 30.0]$  &  $[-1.0, 26]$\\
\bottomrule
\multirow{4}{*}{Half-Cheetah} & Torso size    & $[-0.1, 2.0]$    &  $[0.055, 0.52]$\\
                           & Density       & $[10, 5000]$     &  $[285, 2420]$\\
                           & Friction      & $[-3.0, 10.0]$   &  $[0.65, 4.39]$\\
                           & Joint Damping & $[-10.0, 30.0]$  &  $[-1.65, 12.9]$\\
\bottomrule
\end{tabular}
}
\caption{\small Ranges of parameters for each environment, in the beginning of training and the equivalent ranges found by the algorithm, obtained by fitting an uniform distribution to the final learned distribution.}
\label{table:learned_ranges}
\end{table*}

Initially, each of these distribution is uniform. As the agent becomes better over the training distribution, it becomes easier to discriminate between promising environments (where the task is solvable) and impossible ones where rewards stay at low values. After around $1500$ epochs, the distributions have converged to their final distributions. For Hopper, the learned distributions corresponds closely with the environments where we can find a policy using vanilla policy gradient methods from scratch. To determine the consistency of these results, we ran the Hopper torso size experiment 7 times, and fitted the parameters of a uniform distribution to the resulting $p_{\phi}(z)$. The mean ranges ($\pm$ one standard deviation) across the 7 experiments were $[0.00086 \pm 0.00159, 0.09275 \pm 0.00342]$, which provides some evidence for the reproducibility of our method.
For figures~\ref{fig:hop-dyna}, \ref{fig:cheet-dyna}, all plots correspond to a multidimensional context experiment. Since the contexts are being learned jointly, the final distributions correspond to the joint distribution of all contexts. These plots correspond to the 2D projection of each context distribution.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_torso_distr.pdf}
        \label{fig:hop_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_density_distr.pdf}
        \label{fig:hop_density_distr}
        \vspace{-1.5em}
        \caption{\small Density}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_friction_distr.pdf}
        \label{fig:hop_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_learned_damping_distr.pdf}
        \label{fig:hop_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned domain randomization distribution $p_{\phi}(z)$ over time for \textbf{Hopper}.  Each plot corresponds to a different experiments where we kept the other simulator parameters fixed at their default values. Lighter color corresponds to higher probabilities.}\label{fig:hop-dyna}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_torso_distr.pdf}
        \label{fig:cheet_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_density_distr.pdf}
        \label{fig:cheet_density_distr}
        \vspace{-1.5em}
        \caption{\small Density}
    \end{subfigure}
        ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_friction_distr.pdf}
        \label{fig:cheet_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.2\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_learned_damping_distr.pdf}
        \label{fig:cheet_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned domain randomization distribution $p_{\phi}(z)$  over time for \textbf{Half-Cheetah}. Note how the learned distribution assigns low probability to physically implausible environments (negative or large mass, large friction etc)}\label{fig:cheet-dyna}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Torso.pdf}
        \label{fig:hop_multidim_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Fric.pdf}
        \label{fig:hop_multidim_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hopper_multidim_Damp.pdf}
        \label{fig:hop_multidim_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned multidimensional domain randomization distribution $p_{\phi}(z)$ for \textbf{Hopper}.}
    \label{fig:hop-dyna-gaussian}
\end{figure}

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Torso.pdf}
        \label{fig:cheet_multi_torso_distr}
        \vspace{-1.5em}
        \caption{\small Torso size}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Fric.pdf}
        \label{fig:cheet_multi_friction_distr}
        \vspace{-1.5em}
        \caption{\small Friction}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.14\textwidth}
        \includegraphics[width=\textwidth]{Figures/hcheetah_multidim_Damp.pdf}
        \label{fig:cheet_multi_damping_distr}
        \vspace{-1.5em}
        \caption{\small Damping}
    \end{subfigure}
    \caption{\small Evolution of the learned multidimensional domain randomization distribution $p_{\phi}(z)$ for \textbf{Half-Cheetah}.}
    \label{fig:cheet-dyna-gaussian}
\end{figure}

\noindent\textbf{Learned vs Fixed Domain Randomization: }
We compare the jumpstart and asymptotic performance between learning the domain randomization distribution and keeping it fixed. Our results show our method, using PPO as the policy optimizer (\textbf{LSDR}) vs keeping the domain randomization distribution fixed (\textbf{Fixed-DR}) which corresponds to keeping the domain randomization distribution fixed.
We ran the same experiments for \textit{Hopper} and \textit{Half-Cheetah}.

Figure~\ref{fig:hop-cheet-generalization} depict learning curves when fine tuning the policy at test-time, for torso size randomization. All the methods start with the same random seed at training time across $100$ seeds. The policies are trained for $3000$ epochs, where we collect $B=4000$ samples per epoch for the policy update. For the distribution update, we collect $K=10$ additional trajectories and run the gradient update for $M=10$ steps (without re-sampling new trajectories).
To ensure fairness in terms of the sampled contexts, we report the comparison over the full test range (the support of $p(z)$). From these results, it is clear that learning the domain randomization distribution improves on the jump-start and asymptotic performance over using fixed domain randomization. In this case we test the policy generalization on the wide test distribution, which includes contexts that are not solvable; i.e. the optimal policy found by vanilla-PPO does not result in successful locomotion on the such extreme range\footnote{Successful policies on Hopper obtain cumulative rewards of at least $2500$. For Half-Cheetah, the rewards are greater than 0 when the robot successfully moves forward.}. For the multidimensional Hopper experiments, the results are shown in Figure~\ref{fig:hopper_grid_comparison}. In this case, we evaluated the performance of the policy at the last training iteration, with 10 rollouts per context. For the independently trained policies, we ran PPO for each context in the grid for $1000$ epochs, each with $4000$ steps of experience. While our method does not match the performance of the exhaustive grid training, it is clearly better than training on an uniform DR distribution. Furthermore, our method observes considerably less experience data ($1.2e7$ steps) than the exhaustive grid training ($4e9$ steps). We expect to get better performance by increasing the per-epoch experience, as done in~\cite{yu2017preparing}.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[b]{0.4\textwidth}
        \includegraphics[width=\textwidth]{Figures/generalization_hopper_multidim.pdf}
        \label{fig:hopper-generalization}
        \vspace{-1em}
    \end{subfigure}
    ~
    \begin{subfigure}[b]{0.4\textwidth}
        \includegraphics[width=\textwidth]{Figures/generalization_cheetah_multidim.pdf}
        \label{fig:cheetah-generalization}
        \vspace{-1em}
    \end{subfigure}
    \caption{\small Comparison of test-time performance on Hopper (top) and Half-Cheetah (bottom) between fixed vs learned domain randomization in the policy inputs. Note that running experiments on the test range, will result in environments that may not necessarily be solvable and this results in additional variance on the performance across seeds.}
    \label{fig:hop-cheet-generalization}
\end{figure}

\noindent\textbf{Using a different policy optimizer}
\label{sec:epopt}
We also experimented with using EPOpt-PPO~\cite{rajeswaran2016epopt} as the policy optimizer in Algorithm~(\ref{alg:LSDR-policy}). The motivation for this is to mitigate the bias towards environments with higher cumulative rewards $J_{M_z}(\pi)$ early during training. EPOpt encourages improving the policy on the worst performing environments, at the expense of collecting more data per epoch. At each training epoch, we obtain $100$ samples from the training distribution and obtain trajectories by executing the policy once on each of the corresponding environments. From the resulting $100$ trajectories, we use the $10\%$ trajectories that resulted in the lowest rewards to fill the buffer for a PPO policy update, discarding the rest of the trajectories.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.4\textwidth]{Figures/epopt_minmax-cropped.pdf}
    \caption{\small Learning curves for torso DR on Hopper using EPOpt. Lines represent mean performance, while the shaded regions correspond to the maximum and minimum performance (smoothed with a 5th order Savitzky-Golay filter and a window of 10 epochs)}
    \label{fig:epopt_minmax}
\end{figure}

Figure~\ref{fig:epopt_minmax} compares the effect of learning the domain randomization distribution vs using a fixed wide range in this setting. We found that learning the domain randomization distribution resulted in faster convergence to high reward policies over the evaluation range [0.01, 0.09]: our method attains cumulative rewards greater than 3000 in about 1/2 the epochs required with uniform DR. Our method also results in better asymptotic performance: our method finds policies with better and more consistent performance (less variance) as seen in Figure~\ref{fig:epopt_minmax}. We believe this could be a consequence of lower variance in the policy gradient estimates, as the the learned $p_{\phi}(z)$ has lower variance than $p(z)$. Interestingly, using EPOpt resulted in a distribution with a wider torso size range than vanilla PPO, from approximately $0.0$ to $0.14$, demonstrating that optimizing worst case performance does help in alleviating the bias towards high reward environments.

\section{Discussion}
By allowing the agent to learn a good representative distribution, we are able to learn to solve difficult control tasks that heavily rely on a good initial domain randomization range.
Our main experimental validation of domain randomization distribution learning is in the domain of simulated robotic locomotion.
As shown in our experiments, our method is not sensitive to the initial domain randomization distribution and is able to converge to a more diverse range, while staying within the feasible range. We experimented with single dimensional discrete and multivariate Gaussian distributions for learning the DR distributions. Our results show the benefit of learning a domain randomization distribution when compared to the previously accepted practice: keeping it fixed. Further extension of this work will explore better objectives for learning the distribution and more flexible parameterizations for the DR distribution.

Unscaled rewards, non-linear dynamics and many other complex factors make the manual setting of DR ranges ill-suited for human intuition. Automating this step is crucial to improving the robustness of simulation-to-real transfer and solving today's wide range of robotics challenges; our method is a new step in this fruitful direction.

redpajama_set_name: RedPajamaArXiv


So, if you would like acquire these fantastic graphics regarding (Chord Progressions Guitar), just click save icon to store these pics for your pc. They are ready for download, if you'd rather and wish to have it, click save badge in the page, and it will be instantly saved in your notebook computer. Finally if you desire to obtain new and the recent picture related to (Chord Progressions Guitar), please follow us on google plus or save the site, we attempt our best to give you daily update with all new and fresh shots. Hope you love keeping right here. For many upgrades and recent news about (Chord Progressions Guitar) images, please kindly follow us on tweets, path, Instagram and google plus, or you mark this page on bookmark section, We try to give you update periodically with fresh and new shots, love your surfing, and find the best for you.
Thanks for visiting our website, articleabove (Chord Progressions Guitar) published by at . At this time we are delighted to declare we have found an awfullyinteresting contentto be discussed, namely (Chord Progressions Guitar) Many individuals looking for specifics of(Chord Progressions Guitar) and certainly one of these is you, is not it?


Tell us about your favorite childhood meal — the one that was always a treat. That meant "celebration," or that comforted you and has deep roots in your memory.
Again I am going to change the parameters of this challenge, as my blog is all about 2014. But I also want to say at the outset, and to strengthen my anti-childhood memory stance. It was lousy back there: Food in late 1950s and early 1960s in Perth working class post war re-patronized suburbs was very conservative: Lamb roasts, baked potatoes and two veges – usually lamb with peas and carrots.
Oh, and what about that dreadful pumpkin baked with orange juice my Nanna used to make. Fish and chips an essential Friday catholic thing. My mother did a Continental cookery class at night classes down the hill at Tuart Hill High School to learn how to prepare the wonderful new things Italian and Greek immigrants were bringing into the country.
But I want to move forward, a whole generation and a new –ism (multiculturalism). Instead of remembering the world of a 14 year old I am going to focus on 2014.
Being on the road in 2014 has been imbibed with a necessity for simplicity, pasta and stir through sauce. Fish and Chips (see other blog entries about this challenge) Soup and toast. Grilled meat and a salad which might depending upon what is in the larder sometimes only sliced tomatoes with artful seasoning (sachets of salt and pepper released from a MacDonald's restaurant).
Every kitchen should have one, a crepe machine.
Imagine my delight at being introduced to a stocked pantry and gourmet appliances in my sister's house – Our home for an 8 week house sitting term. We have been instructed to use up foodstuffs, to avail ourselves of the facilities, but leave the treasured record collection alone. In return our tasks are to water an out of control garden, and walk an extremely well behaved dog.
On one early morning walk out along the boundaries of Bribie Island National Park I find a Paw-paw – Papaya tree. A few unripe fruit have been victims of overnight strong winds, they come home with me. I can make Green Papaya Salad, a delicacy I encountered once at a Cooking class.
Thank you Google for the recipe: I reflect that electronics has impacted on our use of books.
Amongst products in my sister's kitchen I can find fish sauce which usually draws the – 'can I have mine without fish sauce' comment from my husband. Rice wine, dried shallots, sweet soy sauce, chilies (another prohibited item) and sesame seeds. I cut loose in a frenzy of chopping and slicing. He can have the meat pie from Beefy Pies we didn't eat yesterday.
'I didn't think you would like it.' Unspoken is – because I know what is in here and its al; the things you don't like me to use at home.
'Of course.' Again I am silently sure it will be instantly rejected. But nope!
So next time I find green papaya there is to be two bowls of green papaya salad made. I might even be able to sneak some exotic spices, sauces and basic ingredients into the shopping trolley when we are done with our travels.
Have I met the challenge of a distinctive voice by keeping this first person, but changing the boundaries?


text:
KPMG severely criticised by Financial Reporting Council
The accountancy watchdog singles out the firm for an 'unacceptable deterioration' of quality in its audit work
KPMG was also recently fined £4.5m by the watchdog over its audit of insurance firm Quindell
KPMG has been severely criticised by the accountancy watchdog for an "unacceptable deterioration" of quality in its audit work.
The Financial Reporting Council's review of audit quality singled out the accountancy giant, which will be subject to increased scrutiny as a result.
KPMG fined £4.5m over audit of insurance firm Quindell
"Fifty per cent of KPMG's FTSE 350 audits required more than just limited improvements, compared to 35 per cent in the previous year.
"As a result, KPMG will be subject to increased scrutiny by the FRC," the watchdog said.
KPMG came in for stinging criticism over its audit of collapsed firm Carillion earlier this year, and the FRC has opened an investigation into the group under the Audit Enforcement Procedure.
KPMG was also recently fined £4.5 million by the watchdog over its audit of insurance firm Quindell.
The increased FRC scrutiny of KPMG includes inspecting 25 per cent more of the firm's audits over 2018/19 and monitoring the implementation of its audit quality plan.
KPMG's new leadership will be made to account for any shortcomings, the watchdog added.
The FRC also took aim collectively at the so-called Big Four - Deloitte, PwC, EY and KPMG - for an overall decline in audit inspection results, urging them to "act swiftly".
It cited a failure by auditors to challenge management and show "appropriate scepticism".
Results from the most recent inspections of eight firms - including the big four, BDO, Grant Thornton, Mazars, Moore Stephens - by the FRC show that in 2017/18, 72 per cent of audits required no more than limited improvements compared with 78 per cent in 2016/17.
Among FTSE 350 company audits, 73 per cent required no more than limited improvements against 81 per cent in the prior year.
FRC chief executive Stephen Haddrill said: "At a time when public trust in business and in audit is in the spotlight, the Big Four must improve the quality of their audits and do so quickly.
"They must address urgently several factors that are vital to audit, including the level of challenge and scepticism by auditors, in particular in their bank audits.
Watchdog to probe KPMG over Carillion figures
"We also expect improvements in group audits and in the audit of pension balances. Firms must strenuously renew their efforts to improve audit quality to meet the legitimate expectation of investors and other stakeholders."

redpajama_set_name: RedPajamaCommonCrawl


text:
Za or ZA may refer to:

Business
 Za (guilds), former Japanese feudal trade guilds; also a Japanese term (座), usually meaning "seat" or "platform"
 ZoneAlarm, an internet security software company
 ZA, IATA airline code for Interavia Airlines
 Zest Airways, a defunct airline in the Philippines

Country codes
 ZA (from Dutch Zuid-Afrika), ISO 3166 country code for South Africa
 ZA, FIPS 10-4 country code for Zambia

Language
 Za (Armenian letter) (Զ զ), the sixth letter of the Armenian alphabet
 Za (cuneiform), a sign of the ancient Near Eastern cuneiform script
 Ẓā' (ظ), the penultimate and rarest modern Arabic letter
 Zhuang languages, spoken in southern China (ISO 639-1: za)

Science and technology
 .za, South Africa's Internet top-level domain
 Zeptoampere (zA), SI unit of electric current, equal to 10−21 amperes
 Zettampere (ZA), SI unit of electric current, equal to 1021 amperes
 Zinc aluminium (ZA), a family of metal alloys

Other uses
 "Za", a song by Supergrass from their 2002 album Life on Other Planets
 Za, American slang for pizza
 Za (plant), the common name in Malagasy for Adansonia za

See also

Za Za Za, album by Grupo Climax
Za-Za album by BulletBoys

redpajama_set_name: RedPajamaWikipedia


text:
"Baumwipfelpfad " treetop path at Bad Iburg horticultural show
Bad Iburg (Lower Saxony)
Designed as a highlight of the horticultural show in Bad Iburg, the treetop path is expected to enhance the attractiveness of the town and the Teutoburg Forest region as a tourist destination in the long term. With a total length of 439 metres and a height of up to 28 metres, the bridges of the treetop path wind through the trees of an ancient mixed woodland forest.
Bad Iburg treetop path has 2 access towers and thus closes the gap in a circular route concept. Main access is via the taller of the 2 towers (32 metres high). From its viewing platform, visitors can enjoy a stunning panoramic view of the town and the surrounding region. The climb up to the platform involves 158 steps that wind around the tower up to the top. A lift is available inside the tower for people with disabilities and visitors with pushchairs. The treetop path is accessed via the lowest tower platform at a height of 17 metres.
As the project manager, we were responsible for the treetop path from the initial idea through to planning and construction. The general contractor Biedenkapp (Wangen) executed the large-scale project.
Situated along the treetop path are a variety of environmental education stations which focus on a range of topics, including forest ecosystems and the geology of the Teutoburg Forest, in 10 different subject areas.
Type: Treetop path
Constructed: 2017/18
Entrance tower: 32 m
Path length: 439 m
Special feature: Fully accessible
www.laga2018-badiburg.de
Determining the height
Photos of the structure


text:
Ken Block Blows Up The Internet Again With A 914HP EcoBoost F-150
By Steve Turner October 29, 2018
To say Ken Block is an internet sensation would be an understatement. His Gymkhana video series showcases his driving skills in a number of amazing machines. For the tenth film in the series, the head Hoonigan turned to Detroit Speed in Mooresville, North Carolina, to construct an all-wheel-drive, EcoBoost-powered version of the same model truck he learned to drive in — a 1977 Ford F-150.
…It's simply one of the most badass and unique vehicles I've seen in a long time… — Ken Block, Hoonigan Racing Division
Believe it or not, the 1977 Ford F-150 Hoonitruck shares some traits with Ford's modern pickups, including the aluminum used for its bodywork. It even has a backup camera and remote-start capabilities via the Ford Pass smartphone app. (Photo Credit: Hoonigan)
"This project has been in the works for about two years," Ken explained. "I'm beyond pleased with how it turned out, it's simply one of the most badass and unique vehicles I've seen in a long time and I think it complements my Hoonicorn perfectly. It's also crazy to be throwing something around that's this size when doing Gymkhana-style driving. But, it looks absolutely wild on camera, and that was our goal from the start. I can't wait for the fans to be able to see it in action, I think they're gonna love it!"
Under the hood of the Hoonitruck is a 900-plus horsepower 3.5-liter EcoBoost engine, similar to the powerplant used in the modern Raptor and Ford GT.
You will be able to see the truck's build, testing, and filming of Gymkhana Ten as part of Ken's new Amazon Prime series, The Gymkhana Files. It is on that platform where Gymkhana Ten will first be shown on December 7, 2018 before eventually appearing on the Hoonigan YouTube channel on December 17, 2018. Before all that, however, attendees of the SEMA show will be able to peep this monster machine in the Toyo Tires booth.
1977 Ford F-150 Hoonitruck Specs
• 3.5-liter Ford Performance/Roush Yates EcoBoost V6
• 914 horsepower
• 702 lb-ft of torque
• Chassis design, engineering and bodywork by Detroit Speed Engineering
• Custom 20-inch fifteen52 Turbomac HD wheels
• Hand-hammered military-grade aluminum bodywork
• Sadev six-speed, all-wheel-drive gearbox
• Styling by Block and his team at Hoonigan
• Toyo Tires Proxes ST III
Steve Turner
Steve Turner brings decades of passion and knowledge in the world of Ford performance, having covered it for over 20 years. From the swan song of the Fox Mustang to the birth of the Coyote, Steve had a front-row seat.
Edelbrock And Pat Musi Expand Line Of High-Performance Crate Engines
EcoBoost Battle: Tuned 2.7 F-150 vs. Stock 3.5 F-150
Livernois Releases Firestorm Fuel Pump Kit for 3.5L EcoBoost F-150
Tuning The EcoBoost F-150 With SCT's Livewire TS


Under the fly-and-buy program, clients will be reimbursed the cost of the airfare and accommodations in Panama for anyone who visits during the month of April and eventually purchases a home within the Beach Club Residences. Similar to island projects in Dubai and the Caribbean, Ocean Reef offers a small gated community of luxury homes and a 170-slip marina on two man-made islands, connected to the city by a 160-meter (525-foot) bridge.
The Fly-and-Buy program coincides with the April 1 start of construction on Beach Club Residences, which is the first development to offer 2 or 3 bedroom units on Ocean Reef. Located on the second island, steps away from the planned Marina & Yacht Club, the Residences offer the amenities of a five-star beach resort, with pools, a sandy beach and private watersports dock for its residents.
Sales in Beach Club Residences launched last year and sold quickly as buyers took advantage of the only development on Ocean Reef Islands offering residences for under $1 million, said PPR President Duncan McGowan.
With the start of construction in Beach Club Residences, the developer is releasing the floor plans for residences in the third tower, with some special units that will include private plunge pools.
Progress continues on many areas of Ocean Reef Islands. In addition to the start of construction on Beach Club Residences, developer Grupo Los Pueblos has announced that work is set to begin on the marina in the next three months with the delivery of the first slips at the end of 2019. Several more families have taken occupancy of homes on the first island in recent months, as the Ocean Reef Island's community takes shape.


Horses and children, I often think, have a lot of the good sense there is in the world.
Horses are a lot of fun. Our Equiboosters don't always have to do 'serious stuff' – they can enjoy themselves too. One of the things they enjoy the most, is being pampered like princes: being brushed, being caressed, being talked to, being spoiled. They love it when they can play too. And they love playing with people. And they love people playing with them.
This teambuilding is not about coaching your team. This teambuilding is about having fun with your team. And with the horses. We have lots of fun activities with horses for you in store. Let us know what you are looking for and we'll send you a proposal that meets your requirements. The Just for fun Teambuildings are organized in cooperation with Event-Solutions.
Our Equiboosters are happy horses. Happy horses are relaxed horses. That makes them a perfect friend for Seniors who want to get out of the house or the home they're staying at and spend some quality time with these noble animals. The interaction with the horses brings a sense of overall wellbeing and energizes the visitors.
Equiboost receives small groups of children (max. 10) who want to celebrate their birthday with the horses. An adapted afternoon program can be arranged, together with a birthday cake. This birthday party will be remembered for a long time!


text:
Q: Make Up/Down arrow in input boxes not do anything How can I make <input> elements not react to pressing the Up arrow (keyCode 38) or the Down arrow (keyCode 40), while they are focused? I'm using jQuery for the project, but have no qualms against writing it in raw JS if that's easier.

A: Like this:
$('.yourinputclass').keypress(function(e) {
    if(e.which == 38 or e.which == 40) return false; // or you can use e.preventDefault(); like it was mentioned in the comments
});

Documentation here

redpajama_set_name: RedPajamaStackExchange


Subscribe to Natalie's mailing list and be notified when the travelog is updated!
No longer want to receive updates by email? Was your email mistakenly entered? No worries... Unsubscribe from Natalie's mailing list!


Bosch is a leading technology company, it offers Public Address Systems for inddustrial and comercial applications, Amplifiers and Conference Systems.
In 1886, Robert Bosch founded the 'Workshop for Precision Mechanics and Electrical Engineering' in Stuttgart. This was the birth of today's globally active Robert Bosch GmbH. From the very beginning, the company's history has been characterized by innovative drive and social commitment. We have compiled the highlights from over 130 years in our archives.
JBL is an American audio electronics company founded in 1946 by James Bullough Lansing. Their primary products are loudspeakers and associated electronics.
There are two independent divisions within the company - JBL Consumer and JBL Professional. The former produces audio equipment for the consumer home market while the latter produces professional equipment for the studio, installed sound, tour sound, portable sound (production and DJ), and cinema markets.
Mackie is a brand of the US based company LOUD Technologies. The Mackie brand is used on professional music and recording equipment, such as mixing consoles, loudspeakers, studio monitors and DAWcontrol surfaces, digital recording equipment and more.
P.Audio designs and produces high performance powered and passive sound reinforcement systems. The product lines are optimized for portable and fixed applications and fixed installations.
Many of the powered designs feature onboard digital signal processing (DSP) and optimize the integration of high performance components and system enclosures.PAudio product offerings range from medium format concert systems and controlled pattern column systems to permanent installation designs and foreground/background and ceiling systems.
DNM has 20 years of experience in the manufacturing of audio equipments for homes, offices, restaurants, hotels, places of worship, schools, health spas, gardens, garment showrooms, coffee shops etc.
Since its founding, Dynatech has been devoted to meeting the needs of sound reinforcement professionals with the finest products available.
Group of engineers from different parts of the world including Canada, Taiwan, Thailand, India, Indonesia, Malaysia and China came together to create and support high-quality products for sound reinforcement, designed to meet the real challenges faced by audio practitioners.
With a history of audio innovation that began in 1925, Shure has turned a passion for making great microphones and audio electronics into an obsession.
Shure continues to set the worldwide industry standard for superior microphones and audio electronics.