text:
Feature | August 2017
Implementation is Key
Formulating policies is only the first step, the success of smart grids depends on the time-bound implementation of projects.
 Technology enabled systems and solutions have become the driving force for electricity segment in the last few year. It may not be wrong to say, smart metering, smart grid applications, two-way metering, growing generation of power from renewable energy sources have driven the power grid market. However, India is still away from rolling out the smart grid. What is lacking? In one word, the answer is implementation.

In the past decade, India was successful in formulating and taking baby steps towards the technology-intensive smart grid roll out. The country is on par with its peers as it was quick in adopting the technology. In the recent times, the price sensitive Indian market saw products that are designed for its geographical and climatic conditions. But, yet to see any sizeable investments in research and development.

On the policy front, the country has made noteworthy imprints. In the year 2010, the India Smart Grid Forum (ISGF) and India Smart Grid Task Force (ISGTF) were set up. In less than two years, the government had allotted 14 smart grid pilot projects. 2013 saw release of Smart Grid Vision and Roadmap. The National Smart Grid Mission (NSGM) was constituted in 2015. In the same year the Forum of Regulators (FoR) approved the Model Smart Grid Regulations, an enabling document. The states have started drafting smart grid policies for the implementation. Interestingly, in the last three years, 27 state electricity regulatory commissions have issued Net Metering Regulations. And also the Electricity Tariff Policy of 2016 mandated smart meters for all customers with a monthly consumption above 200 kWh by the end of 2019.

Smart grids are the best answer to a country which is looking at enhancing energy efficiency, reducing load shedding, efficient use of the existing networks, ensuring reliable and quality power and reducing transmission and distribution (T&D) losses. Power produced has to be consumed and canot be stored even today, that makes it even more challenging. A technologically advanced system, smart grids not only enables the state power distribution companies (discoms) in load management, but also help consumers to realign their consumption patterns. In turn, it helps the user to bring down their energy cost.

All appears to be smooth and appealing regarding the steps taken from the government side. But what happened to the 14 smart grid pilot projects which were sanctioned in 2012? 'Owing to various reasons, we could not do much on the implementation side. None of the pilots could be completed till date. Four of the 14 projects have been cancelled. We see two or three projects may be completed in 2017,' said Reji Kumar Pillai, President, ISGF.

The pan India roll out of smart grids to a greater extent hinged on the success of the pilot projects and the data collected from them. As these projects too failed, the natural question is how far are we are realistically from developing smart grids in the country. The industry experts are of the opinion that the pilot projects were location/state specific. And for a pan-India roll out you need to look at projects that can generate sample data which can be used by any state in India. And, they also feel that the states are yet to be ready with solid implementation plans.

Agreed that even the bigger countries and the developed nations implemented smart grids in phases and for a country like India, with a vast geography and geography-specific challenges, a mass roll out may not be a good idea. Realistically, it will take three to five years before India sees a sizeable smart grid network in place.

Why Smart Grids?
 Indian discoms and the transmission entities face a lot of challenges. For the transmission companies, it is primarily the fight against the Aggregate Transmission and Commercial (ATC) losses. And for the discoms it is a combination of financial issues to operational efficiencies, fight against power theft and upgrading itself to smart grid, i.e., deployment of smart metering.

UDAY (Ujwal Discom Assurance Yojana), a scheme was approved in November 2015 to restructure the state discoms' outstanding loans to the tune of Rs 4.3 laKh crore. Also, aimed at strengthening the financial health of discoms, the scheme also mandates discoms to follow the following four steps (i) Improving operational efficiencies; (ii) Reduction of cost of power; (iii) Reduction in interest cost; and (iv) Enforcing financial discipline. For improving operational efficiency, it mandates compulsory deployment of smart meters, upgradation of transformers, meters etc. It also mandates discoms to bring down the transmission and distribution (T&D) losses to 15 per cent levels in 2018-19 from the current 21 per cent levels.

India has recently shifted its focus from conventional energy to renewable energy. It has set an ambitious target of adding 175 GW of renewable energy to the grid by 2022. Of which 40 GW is going to come from solar PV on 20 million rooftops. Which will be connected to low voltage distribution grids straight away. This means the discoms need to look at technologies that can enable flexibility in power systems that can integrate the increasing share of renewable energy. This flexibility should be both in handling supply and demand.

 Impediments
 In an earlier interaction, Dr Pramod Deo, former chairman of Central Electricity Regulatory Commission (CERC), pointed out, ôThe present structure of cross-subsidy will be the major hurdle for smart grid implementation. In states like Maharashtra and Gujarat, the revenue from agriculture is only 13 per cent, whereas the revenue from industrial consumption of Maharashtra 47 per cent and in Gujarat it is 67 per cent. But agriculture consumes electricity which is double the cost and pays just half. So, how do you tackle this? It again trickles down to the political will.'

On the other hand, Pillai was candid in pointing out the lack of trained manpower in discoms; the existing procurement frame work etc. as challenges. He said, 'Implementing smart grid systems are not like buying electrical products and equipment. These are sophisticated technologies integrated with two-way communication systems, which cannot be procured on lowest bidder route.'

The Indian market is price sensitive, at the same time it is also a big volume market. So, for product manufactures it is challenging. However, the prices of smart meters have come down to Rs 70 a year. And this cost according to Pillai should be shared between the consumer and the utility at equal proportions.

A global player like Sensus admits that they bet big on India. Amit Vaidya, Director- Strategic Customer Team, Sensus India, said, 'The Indian Prime Minister's initiative -Make In India - is helping MNCs like us to plan for local manufacturing and bring the cost of products down.'

An integrated smart grid will consist of a network of devices that are connected to the central controlling point. These include asset management system, control systems at various points, automation of distribution network, a management system for the distribution grid, energy management system, information and communication devices, power quality and power management devices, security systems, and a two-way smart meter at the consumer end.

To give you an example, Sensus network technology offers near real-time data from the network, which helps one to monitor and manage the services to identify any issues on the distribution network and address them quickly.

 Cyber Security
 Cyber security plays a critical role in a smart grid. The recent cyber attacks witnessed globally have stalled the operations of aviation and logistics segments. In May 2017, an attack had left the British Airways services in chaos worldwide. And in June 2017, one of the largest container shipping company had a tough time when Patya virus attacked their system. Same was the case when FedEx holding company TNT Express was thrown out of gear due to cyber attack. If a similar attack were to happen on an electricity network, the whole grid could collapse in no time. So, building a foolproof system and managing it, and safeguarding it from such threats would be a challenge.

 Way Forward
 The policy is in place, the price of products are suitable for the Indian market conditions, countries like US, European Commission, Canada, Sweden and Japan are extending their support in terms of funding and technology. So, what is preventing the implementation of smart grids?

Apart from the procurement and discoms' financial health, the regulatory commissions' approvals also are essential for the implementation of smart grids. The introduction of dynamic tariff structures based on time of use (ToU)/time of the day (ToD). These are absent in many of the states currently.

The state regulators in the normal course approve investments if the benefits outweigh the cost. However, in the case of smart grids, the benefits could be intangible and call for higher capital investments. A supportive investment approval framework that promotes technological innovations is required.

An interoperability framework in place will also be a requirement for the large scale deployment. The discom employees should be given adequate training to handle the technology-intensive grid. And last but not the least, the consumer should be given ample awareness on the benefits of smart metering in reducing their electricity bills. Now, discoms have to pull up their socks and act. For the mass roll out, push from the Ministry of Power in the form of encouraging incentives may be quite imminent.

 -Renjini Liza Varghese

Formulating policies, Smart grids, Power grid, Discoms, Transmission, Distribution, ISGF, UDAY, Transformers, Meters
MATLAB-Simulink helps discoms in load, demand & price planning
Need conducive environment to transform discom processes
CIL to launch coal auction with flexible lifting
The Future of LED Lighting
Kerala has a potential for 10K mw solar rooftop energy'
ReNew Power in talks with PE firm to raise $200 million
MP's Sagar gets round the clock power supply scheme


text:
\section{Introduction}
One of the major tenets of reinforcement learning states that, when not otherwise constrained in its behaviour, an agent should aim to maximize its expected utility $Q$, or \emph{value} \citep{sutton98reinforcement}. Bellman's equation succintly describes this value in terms of the expected reward and expected outcome of the random transition $(x, a) \hspace{-0.1em} \to \hspace{-0.1em}  (X', A')$:
\begin{equation*}
Q(x,a) = \expect R(x,a) + \gamma \expect Q(X', A') .
\end{equation*}
In this paper, we aim to go beyond the notion of value and argue in favour of a distributional perspective on reinforcement learning.
Specifically, the main object of our study is the random return $Z$ whose expectation is the value $Q$. This random return is also described by a recursive equation, but one of a distributional nature:
\begin{equation*}
Z(x,a) \overset{D}{=} R(x,a) + \gamma Z(X', A').
\end{equation*}
The \emph{distributional Bellman equation} states that the distribution of $Z$ is characterized by the interaction of three random variables: the reward $R$, the next state-action $(X', A')$, and its random return $Z(X', A')$. By analogy with the well-known case, we call this quantity the \emph{value distribution}.

Although the distributional perspective is almost as old as Bellman's equation itself \citep{jaquette73markov,sobel82variance,white88mean}, in reinforcement learning it has thus far been subordinated to specific purposes: to model parametric uncertainty \citep{dearden98bayesian}, to design risk-sensitive algorithms \citep{morimura2010nonparametric,morimura10parametric}, or for theoretical analysis \citep{azar12sample,lattimore12pac}.
By contrast, we believe the value distribution has a central role to play in reinforcement learning.

\textbf{Contraction of the policy evaluation Bellman operator.} Basing ourselves on results by \citet{rosler92fixed} we show that, for a fixed policy, the Bellman operator over value distributions is a contraction in a maximal form of the Wasserstein (also called Kantorovich or Mallows) metric. Our particular choice of metric matters: the same operator is not a contraction in total variation, Kullback-Leibler divergence, or Kolmogorov distance.

\textbf{Instability in the control setting.} We will demonstrate an instability in the distributional version of Bellman's optimality equation, in contrast to the policy evaluation case. Specifically, although the optimality operator is a contraction in expected value (matching the usual optimality result), it is not a contraction in any metric over distributions.
These results provide evidence in favour of learning algorithms that model the effects of nonstationary policies.

\textbf{Better approximations.} From an algorithmic standpoint, there are many benefits to learning an approximate distribution rather than its approximate expectation.
The distributional Bellman operator preserves multimodality in value distributions, which we believe leads to more stable learning. Approximating the full distribution also mitigates the effects of learning from a nonstationary policy. As a whole, we argue that this approach makes approximate reinforcement learning significantly better behaved.%

We will illustrate the practical benefits of the distributional perspective in the context of the Arcade Learning Environment \citep{bellemare13arcade}. By modelling the value distribution within a DQN agent \citep{mnih15nature}, we obtain considerably increased performance across the gamut of benchmark Atari 2600 games, and in fact achieve state-of-the-art performance on a number of games. Our results echo those of \citet{veness15compress}, who obtained extremely fast learning by predicting Monte Carlo returns.

From a supervised learning perspective, learning the full value distribution might seem obvious: why restrict ourselves to the mean? The main distinction, of course, is that in our setting there are no given targets. Instead, we use Bellman's equation to make the learning process tractable; we must, as \citet{sutton98reinforcement} put it, ``learn a guess from a guess''. It is our belief that this guesswork ultimately carries more benefits than costs.

\section{Setting}

We consider an agent interacting with an environment in the standard fashion: at each step, the agent selects an action based on its current state, to which the environment responds with a reward and the next state. We model this interaction as a time-homogeneous Markov Decision Process $(\mathcal{X}, \mathcal{A}, R, P, \gamma)$. As usual, $\mathcal{X}$ and $\mathcal{A}$ are respectively the state and action spaces, $P$ is the transition kernel $P(\cdot \, | \, x, a)$, $\gamma \in [0, 1]$ is the discount factor, and
$R$ is the reward function, which in this work we explicitly treat as a random variable. A stationary policy $\pi$ maps each state $x \in \mathcal{X}$ to a probability distribution over the action space $\mathcal{A}$.

\subsection{Bellman's Equations}

The \emph{return} $Z^\pi$ is the sum of discounted rewards along the agent's trajectory of interactions with the environment. The value function $Q^\pi$ of a policy $\pi$ describes the expected return from taking action $a \in \mathcal{A}$ from state $x \in \mathcal{X}$, then acting according to $\pi$:
\begin{align}
Q^\pi(x,a) &:= \expect Z^\pi(x,a) = \expect \left [ \sum_{t=0}^\infty \gamma^t R(x_t, a_t) \right ], \label{eqn:random_return} \\
x_t \sim P(\cdot &\, | \, x_{t-1}, a_{t-1}), a_t \sim \pi(\cdot \, | \, x_t), x_0 = x, a_0 = a . \nonumber
\end{align}
Fundamental to reinforcement learning is the use of Bellman's equation  \citep{bellman57dynamic} to describe the value function:
\begin{equation*}
Q^\pi(x,a) = \expect R(x,a) + \gamma \expect_{P, \pi} Q^\pi(x', a') .
\end{equation*}
In reinforcement learning we are typically interested in acting so as to maximize the return. The most common approach for doing so involves the optimality equation
\begin{equation*}
Q^*(x,a) = \expect R(x,a) + \gamma \expect\nolimits_P \max_{a' \in \mathcal{A}} Q^*(x', a') .
\end{equation*}
This equation has a unique fixed point $Q^*$, the optimal value function, corresponding to the set of optimal policies $\Pi^*$ ($\pi^*$ is optimal if $\expect_{a \sim \pi^*} Q^*(x, a) = \max_a Q^*(x,a)$).

We view value functions as vectors in $\mathbb{R}^{\mathcal{X} \times \mathcal{A}}$, and the expected reward function as one such vector. In this context, the \emph{Bellman operator} $\mathcal{T}^\pi$ and \emph{optimality operator} $\mathcal{T}$ are
\begin{align}
\mathcal{T}^\pi Q(x,a) &:= \expect R(x,a) + \gamma \expect_{P, \pi} Q(x',a')\label{eqn:bellman_operator_pe} \\
\mathcal{T} Q(x,a) &:= \expect R(x,a) + \gamma \expect\nolimits_{P} \max_{a' \in \mathcal{A}} Q(x', a') . \label{eqn:bellman_operator_optimality}
\end{align}
These operators are useful as they describe the expected behaviour of popular learning algorithms such as SARSA and Q-Learning. In particular they are both contraction mappings, and their repeated application to some initial $Q_0$ converges exponentially to $Q^\pi$ or $Q^*$, respectively \citep{bertsekas96neurodynamic}.

\section{The Distributional Bellman Operators}\label{sec:theory}
\begin{figure}
\begin{center}
\includegraphics[width=0.48\textwidth]{figures/distributional_bellman.pdf}
\end{center}
\caption{A distributional Bellman operator with a deterministic reward function: (a) Next state distribution under policy $\pi$, (b) Discounting shrinks the distribution towards 0, (c) The reward shifts it, and (d) Projection step (Section \ref{sec:algorithm}). \label{fig:bellman_dist}}
\end{figure}

In this paper we take away the expectations inside Bellman's equations and consider instead the full distribution of the random variable $Z^\pi$. From here on, we will view $Z^\pi$ as a mapping from state-action pairs to distributions over returns, and call it the \emph{value distribution}.

Our first aim is to gain an understanding of the theoretical behaviour of the distributional analogues of the Bellman operators, in particular in the less well-understood control setting. The reader strictly interested in the algorithmic contribution may choose to skip this section.

\subsection{Distributional Equations}

It will sometimes be convenient to make use of the probability space $(\Omega, \mathcal{F}, \Pr)$. The reader unfamiliar with measure theory may think of $\Omega$ as the space of all possible outcomes of an experiment \citep{billingsley95probability}.
We will write $\pnorm{\mathbf{u}}$ to denote the $L_p$ norm of a vector $\mathbf{u} \in \mathbb{R}^\mathcal{X}$ for $1 \le p \le \infty$; the same applies to vectors in $\mathbb{R}^{\mathcal{X} \times \mathcal{A}}$. The $L_p$ norm of a random vector $U : \Omega \to \mathbb{R}^\mathcal{X}$ (or $\mathbb{R}^{\mathcal{X} \times \mathcal{A}}$) is then $\pnorm{U} := \left [ \expect \left [ \pnorm{U(\omega)}^p \right ] \right ]^{1/p}$, and for $p = \infty$ we have $\| U \|_\infty = \textrm{ess}\sup{\|U(\omega)\|_\infty}$ (we will omit the dependency on $\omega \in \Omega$ whenever unambiguous).
We will denote the c.d.f. of a random variable $U$ by $F_U(y) := \Pr\{ U \le y \}$, and its inverse c.d.f. by $F^{-1}_U(q) := \inf \{ y : F_U(y) \ge q \}$.

A distributional equation $U \overset{D}{:=} V$ indicates that the random variable $U$ is distributed according to the same law as $V$. Without loss of generality, the reader can understand the two sides of a distributional equation as relating the distributions of two independent random variables.
Distributional equations have been used in reinforcement learning by \citet{engel05reinforcement,morimura10parametric} among others, and in operations research by \citet{white88mean}.

\subsection{The Wasserstein Metric}

The main tool for our analysis is the Wasserstein metric $d_p$ between cumulative distribution functions \citep[see e.g.][where it is called the Mallows metric]{bickel81asymptotic}. For $F$, $G$ two c.d.fs over the reals, it is defined as
\begin{equation*}
d_p(F, G) := \inf_{U, V} \pnorm{U - V},
\end{equation*}
where the infimum is taken over all pairs of random variables $(U, V)$ with respective cumulative distributions $F$ and $G$. The infimum is attained by the inverse c.d.f. transform of a random variable $\mathcal{U}$ uniformly distributed on $[0, 1]$:
\begin{equation*}
d_p(F, G) = \| F^{-1}(\mathcal{U}) - G^{-1}(\mathcal{U}) \|_p .
\end{equation*}
For $p < \infty$ this is more explicitly written as
\begin{equation*}
d_p(F, G) = \left ( \int_0^1 \big | F^{-1}(u) - G^{-1}(u) \big |^p du \right )^{1/p} .
\end{equation*}
Given two random variables $U$, $V$ with c.d.fs $F_U$, $F_V$, we will write $d_p(U, V) := d_p(F_U, F_V)$. We will find it convenient to conflate the random variables under consideration with their versions under the $\inf$, writing
\begin{equation*}
d_p(U, V) = \inf_{U,V} \pnorm{U - V} .
\end{equation*}
whenever unambiguous; we believe the greater legibility justifies the technical inaccuracy.
Finally, we extend this metric to vectors of random variables, such as value distributions, using the corresponding $L_p$ norm.

Consider a scalar $a$ and a random variable $A$ independent of $U, V$. The metric $d_p$ has the following properties:
\begin{align*}
d_p(aU, aV) &\le |a| d_p(U, V) \tag{P1} \\
d_p(A + U, A + V) &\le d_p(U, V) \tag{P2} \\
d_p(AU, AV) & \le \pnorm{A} d_p(U, V). \tag{P3}
\end{align*}
We will need the following additional property, which makes no independence assumptions on its variables.
Its proof, and that of later results, is given in the appendix.
\begin{lem}[Partition lemma]\label{lem:partition_lemma}
Let $A_1, A_2, \dots$ be a set of random variables describing a partition of $\Omega$, i.e. $A_i(\omega) \in \{0, 1\}$ and for any $\omega$ there is exactly one $A_i$ with $A_i(\omega) = 1$. Let $U, V$ be two random variables. Then
\begin{equation*}
d_p\big (U, V \big) \le \sum\nolimits_i d_p (A_i U, A_i V) .
\end{equation*}
\end{lem}
Let $\mathcal{Z}$ denote the space of value distributions with bounded moments.
For two value distributions $Z_1, Z_2 \in \mathcal{Z}$ we will make use of a maximal form of the Wasserstein metric:
\begin{equation*}
\bar d_p(Z_1, Z_2) := \sup_{x, a} d_p(Z_1(x,a), Z_2(x,a)) .
\end{equation*}
We will use $\bar d_p$ to establish the convergence of the distributional Bellman operators.
\begin{lem}
$\bar d_p$ is a metric over value distributions.
\end{lem}

\subsection{Policy Evaluation}\label{sec:policy_evaluation}

In the \emph{policy evaluation} setting \citep{sutton98reinforcement} we are interested in the value function $V^\pi$ associated with a given policy $\pi$. The analogue here is the value distribution $Z^\pi$. In this section we characterize $Z^\pi$ and study the behaviour of the policy evaluation operator $\cT^\pi$. We emphasize that $Z^\pi$ describes the intrinsic randomness of the agent's interactions with its environment, rather than some measure of uncertainty about the environment itself.

We view the reward function as a random vector $R \in \mathcal{Z}$, and define the transition operator $P^\pi : \mathcal{Z} \to \mathcal{Z}$
\begin{align}
P^\pi Z(x, a) &\overset{D}{:=} Z(X', A') \label{eqn:policy_operator} \\
X' &\sim P(\cdot \, | \, x, a), \, A' \sim \pi(\cdot \, | \, X'), \nonumber
\end{align}
where we use capital letters to emphasize the random nature of the next state-action pair $(X', A')$.
We define the distributional Bellman operator $\cT^\pi : \mathcal{Z} \to \mathcal{Z}$ as
\begin{equation}
\cT^\pi Z(x,a) \overset{D}{:=} R(x,a) + \gamma P^\pi Z(x,a). \label{eqn:distributional_bellman_operator_pe}
\end{equation}
While $\cT^\pi$ bears a surface resemblance to the usual Bellman operator \eqnref{bellman_operator_pe}, it  is fundamentally different. In particular, three sources of randomness define the compound distribution $\cT^\pi Z$:
\begin{enumerate}[noitemsep,label=\alph*)]
	\item The randomness in the reward $R$,
	\item The randomness in the transition $P^\pi$, and
	\item The next-state value distribution $Z(X', A')$.
\end{enumerate}
In particular, we make the usual assumption that these three quantities are independent.
In this section we will show that \eqnref{distributional_bellman_operator_pe} is a contraction mapping whose  unique fixed point is the random return $Z^\pi$.

\subsubsection{Contraction in $\bar d_p$}
Consider the process $Z_{k+1} := \cT^\pi Z_k$, starting with some $Z_0 \in \mathcal{Z}$. We may expect the limiting expectation of $\{ Z_k \}$ to converge exponentially quickly, as usual, to $Q^\pi$. As we now show, the process converges in a stronger sense: $\cT^\pi$ is a contraction in $\bar d_p$, which implies that all moments also converge exponentially quickly.

\begin{lem}\label{lem:contraction_pe}
$\cT^\pi : \mathcal{Z} \to \mathcal{Z}$ is a $\gamma$-contraction in $\bar d_p$.
\end{lem}
Using Lemma \ref{lem:contraction_pe}, we conclude using Banach's fixed point theorem that $\cT^\pi$ has a unique fixed point. By inspection, this fixed point must be $Z^\pi$ as defined in \eqnref{random_return}. As we assume all moments are bounded, this is sufficient to conclude that the sequence $\{ Z_k \}$ converges to $Z^\pi$ in $\bar d_p$ for $1 \le p \le \infty$.

To conclude, we remark that not all
distributional metrics are equal; for example, \citet{chung87discounted} have shown that $\cT^\pi$ is not a contraction in total variation distance. Similar results can be derived for the Kullback-Leibler divergence and the Kolmogorov distance.

\subsubsection{Contraction in Centered Moments}
Observe that $d_2(U, V)$ (and more generally, $d_p$) relates to a coupling $C(\omega) := U(\omega) - V(\omega)$, in the sense that
\begin{align*}
d_2^2(U, V) &\le \expect [ (U - V)^2 ] = \mathbb{V} \big (C \big ) + \big ( \expect C \big )^2 .
\end{align*}
As a result, we cannot directly use $d_2$ to bound the variance difference $| \mathbb{V}(\cT^\pi Z(x,a)) - \mathbb{V}(Z^\pi(x,a)) |$. However, $\cT^\pi$ is in fact a contraction in variance \citep[][see also appendix]{sobel82variance}.
In general, $\cT^\pi$ is not a contraction in the $p^{th}$ centered moment, $p > 2$, but the centered moments of the iterates $\{ Z_k \}$ still converge exponentially quickly to those of $Z^\pi$; the proof extends the result of \citet{rosler92fixed}.

\subsection{Control}\label{sec:control}

Thus far we have considered a fixed policy $\pi$, and studied the behaviour of its associated operator $\cT^\pi$. We now set out to understand the distributional operators of the \emph{control} setting -- where we seek a policy $\pi$ that maximizes value -- and the corresponding notion of an optimal value distribution. As with the optimal value function, this notion is intimately tied to that of an optimal policy. However, while all optimal policies attain the same value $Q^*$, in our case a difficulty arises: in general there are many optimal value distributions.

In this section we show that the distributional analogue of the Bellman optimality operator converges, in a weak sense, to the set of optimal value distributions. However, this operator is not a contraction in any metric between distributions, and is in general much more temperamental than the policy evaluation operators. We believe the convergence issues we outline here are a symptom of the inherent instability of greedy updates, as highlighted by e.g.~\citet{tsitsiklis02convergence} and most recently \citet{harutyunyan16qlambda}.

Let $\Pi^*$ be the set of optimal policies. We begin by characterizing what we mean by an \emph{optimal value distribution}.
\begin{defn}[Optimal value distribution]
An optimal value distribution is the v.d. of an optimal policy. The set of optimal value distributions is $\mathcal{Z}^* := \{ Z^{\pi^*} : \pi^* \in \Pi^* \}$.
\end{defn}
We emphasize that not all value distributions with expectation $Q^*$ are optimal: they must match the full distribution of the return under some optimal policy.
\begin{defn}
A greedy policy $\pi$ for $Z \in \mathcal{Z}$ maximizes the expectation of $Z$. The set of greedy policies for $Z$ is
\begin{equation*}
\mathcal{G}_Z := \{ \pi : \sum\nolimits_a \pi(a \, | \, x) \expect Z(x, a) = \max_{a' \in \mathcal{A}} \expect Z(x, a') \}.
\end{equation*}
\end{defn}
Recall that the expected Bellman optimality operator $\mathcal{T}$ is
\begin{equation}
\mathcal{T} Q(x,a) = \expect R (x,a) + \gamma \expect\nolimits_P \max_{a' \in \mathcal{A}} Q(x', a') . \label{eqn:bellman_optimality_operator}
\end{equation}
The maximization at $x'$ corresponds to some greedy policy. Although this policy is implicit in \eqnref{bellman_optimality_operator}, we cannot ignore it in the distributional setting.
We will call a \emph{distributional Bellman optimality operator} any operator $\mathcal{T}$ which implements a greedy selection rule, i.e.
\begin{equation*}
\mathcal{T} Z = \cT^\pi Z\text{ for some } \pi \in \mathcal{G}_Z .
\end{equation*}
As in the policy evaluation setting, we are interested in the behaviour of the iterates $Z_{k+1} := \mathcal{T} Z_k$, $Z_0 \in \mathcal{Z}$. Our first result is to assert that $\expect Z_k$ behaves as expected.
\begin{lem}\label{lem:exponential_convergence_of_mean}
Let $Z_1, Z_2 \in \mathcal{Z}$. Then
\begin{equation*}
\infnorm{\expect \mathcal{T} Z_1 - \expect \mathcal{T} Z_2} \le \gamma \infnorm{\expect Z_1 - \expect Z_2},
\end{equation*}
and in particular $\expect Z_k \to Q^*$ exponentially quickly.
\end{lem}
By inspecting Lemma \ref{lem:exponential_convergence_of_mean}, we might expect that $Z_k$ converges quickly in $\bar d_p$ to some fixed point in $\mathcal{Z}^*$. Unfortunately, convergence is neither quick nor assured to reach a fixed point. In fact, the best we can hope for is pointwise convergence, not even to the set $\mathcal{Z}^*$ but to the larger set of \emph{nonstationary optimal value distributions}.
\begin{defn}
A nonstationary optimal value distribution $Z^{**}$ is the value distribution corresponding to a sequence of optimal policies. The set of n.o.v.d. is $\mathcal{Z}^{**}$.
\end{defn}
\begin{thm}[Convergence in the control setting]\label{thm:control_convergence}
Let $\mathcal{X}$ be measurable and suppose that $\mathcal{A}$ is finite. Then
\begin{equation*}
\lim_{k \to \infty} \inf_{Z^{**} \in \mathcal{Z}^{**}} d_p (Z_k(x,a), Z^{**}(x,a)) = 0 \quad \forall x, a .
\end{equation*}
If $\mathcal{X}$ is finite, then $Z_k$ converges to $\mathcal{Z}^{**}$ uniformly. Furthermore, if there is a total ordering $\prec$ on $\Pi^*$, such that for any $Z^* \in \mathcal{Z}^*$,
\begin{equation*}
\mathcal{T} Z^* = \mathcal{T}^\pi Z^* \; \text{with} \; \pi \in \mathcal{G}_{Z^*}, \; \pi \prec \pi' \; \; \forall \pi' \in \mathcal{G}_{Z^*} \setminus \{\pi \} .
\end{equation*}
Then $\mathcal{T}$ has a unique fixed point $Z^* \in \mathcal{Z}^*$.
\end{thm}
Comparing Theorem \ref{thm:control_convergence} to Lemma \ref{lem:exponential_convergence_of_mean} reveals a significant difference between the distributional framework and the usual setting of expected return. While the mean of $Z_k$ converges exponentially quickly to $Q^*$, its distribution need not be as well-behaved! To emphasize this difference, we now provide a number of negative results concerning $\mathcal{T}$.
\begin{prop}\label{prop:control_noncontraction}
The operator $\mathcal{T}$ is not a contraction.
\end{prop}
Consider the following example (Figure \ref{fig:control_noncontraction}, left). There are two states, $x_1$ and $x_2$; a unique transition from $x_1$ to $x_2$; from $x_2$, action $a_1$ yields no reward, while the optimal action $a_2$ yields $1 + \epsilon$ or $-1 + \epsilon$ with equal probability. Both actions are terminal. There is a unique optimal policy and therefore a unique fixed point $Z^*$. Now consider $Z$ as given in Figure \ref{fig:control_noncontraction} (right), and its distance to $Z^*$:
\begin{equation*}
\bar d_1(Z, Z^*) = d_1(Z(x_2, a_2), Z^*(x_2, a_2)) = 2 \epsilon , \\
\end{equation*}
where we made use of the fact that $Z = Z^*$ everywhere except at $(x_2, a_2)$. When we apply $\mathcal{T}$ to $Z$, however, the greedy action $a_1$ is selected and $\mathcal{T} Z(x_1) = Z(x_2, a_1)$. But
\begin{align*}
d_1(\mathcal{T} Z, \mathcal{T} Z^*) &= d_1(\mathcal{T} Z(x_1), Z^*(x_1)) \\
&= \tfrac{1}{2} | 1 - \epsilon | + \tfrac{1}{2}  | 1 + \epsilon | > 2 \epsilon
\end{align*}
for a sufficiently small $\epsilon$. This shows that the undiscounted update is not a nonexpansion: $\bar d_1(\mathcal{T} Z, \mathcal{T} Z^*) > \bar d_1(Z, Z^*)$. With $\gamma < 1$, the same proof shows it is not a contraction. Using a more technically involved argument, we can extend this result to any metric which separates $Z$ and $\mathcal{T} Z$.

\begin{figure}
\begin{center}
\begin{minipage}{0.1\textwidth}
\hspace{-3em} \includegraphics[width=0.85in]{figures/Noncontraction-Example.pdf}
\end{minipage}
\begin{minipage}{0.2\textwidth}
$\begin{array}[c]{lccc}
\multicolumn{1}{c}{} & \multicolumn{1}{c}{x_1} & \multicolumn{1}{c}{x_2, a_1} & \multicolumn{1}{c}{x_2, a_2} \\
\midrule
Z^* & \cellcolor{lightsunflower}\epsilon \pm 1 & 0 & \cellcolor{lightsunflower}\epsilon \pm 1 \\
\midrule
\midrule
Z & \epsilon \pm 1 & 0 & \cellcolor{lightsunflower} -\epsilon \pm 1 \\
\midrule
\mathcal{T} Z & \cellcolor{lightsunflower}{0} & 0 & \epsilon \pm 1 \\
\bottomrule
\end{array}$
\end{minipage}
\end{center}
\caption{Undiscounted two-state MDP for which the optimality operator $\mathcal{T}$ is not a contraction, with example. The entries that contribute to $\bar d_1(Z, Z^*)$ and $\bar d_1(\mathcal{T} Z, Z^*)$ are highlighted.\label{fig:control_noncontraction}}
\end{figure}
\begin{prop}
Not all optimality operators have a fixed point $Z^* = \mathcal{T} Z^*$.
\end{prop}
To see this, consider the same example, now with $\epsilon = 0$, and a greedy operator $\mathcal{T}$ which breaks ties by picking $a_2$ if $Z(x_1) = 0$, and $a_1$ otherwise. Then the sequence $\mathcal{T} Z^*(x_1), (\mathcal{T})^2 Z^*(x_1),\, \dots$ alternates between $Z^*(x_2, a_1)$ and $Z^*(x_2, a_2)$.
\begin{prop}
That $\mathcal{T}$ has a fixed point $Z^* = \mathcal{T} Z^*$ is insufficient to guarantee the convergence of $\{ Z_k \}$ to $\mathcal{Z}^*$.
\end{prop}
Theorem \ref{thm:control_convergence} paints a rather bleak picture of the control setting. It remains to be seen whether the dynamical eccentricies highlighted here actually arise in practice. One open question is whether theoretically more stable behaviour can be derived using stochastic policies, for example from conservative policy iteration \citep{kakade02approximately}.

\section{Approximate Distributional Learning}\label{sec:algorithm}

In this section we propose an algorithm based on the distributional Bellman optimality operator. In particular, this will require choosing an approximating distribution. Although the Gaussian case has previously been considered \citep{morimura10parametric,tamar2016learning}, to the best of our knowledge we are the first to use a rich class of parametric distributions.

\subsection{Parametric Distribution}\label{sec:parametric_distribution}
We will model the value distribution using a discrete distribution parametrized by $N \in \mathbb{N}$ and $V_{\textsc{min}},V_{\textsc{max}} \in \mathbb{R}$, and whose support is the set of atoms $\{z_i = V_{\textsc{min}} + i \triangle z : 0 \le i < N \}$, $\triangle z := \tfrac{V_{\textsc{max}} - V_{\textsc{min}}}{N-1}$. In a sense, these atoms are the ``canonical returns'' of our distribution. The atom probabilities are given by a parametric model $\theta: \mathcal{X} \times \mathcal{A} \to \mathbb{R}^N$
\begin{equation*}
	Z_\theta(x, a) = z_i \quad \text{w.p. } \; p_i(x, a) := \frac{e^{\theta_i(x, a)}}{\sum\nolimits_j e^{\theta_j(x, a)}}.
\end{equation*}
The discrete distribution has the advantages of being highly expressive and computationally friendly \citep[see e.g.][]{vandenoord16pixel}.

\subsection{Projected Bellman Update}
Using a discrete distribution poses a problem: the Bellman update $\mathcal{T} Z_\theta$ and our parametrization $Z_\theta$ almost always have disjoint supports. From the analysis of Section \ref{sec:theory} it would seem natural to minimize the Wasserstein metric (viewed as a loss) between $\mathcal{T} Z_\theta$ and $Z_\theta$, which is also conveniently robust to discrepancies in support.
However, a second issue prevents this: in practice we are typically restricted to learning from sample transitions, which is not possible under the Wasserstein loss (see Prop. \ref{prop:wasserstein_sgd_rl} and toy results in the appendix).

Instead, we project the sample Bellman update $\hat \mathcal{T} Z_\theta$ onto the support of $Z_\theta$ (Figure \ref{fig:bellman_dist}, Algorithm \ref{alg:cql}), effectively reducing the Bellman update to multiclass classification. Let $\pi$ be the greedy policy w.r.t. $\expect Z_\theta$.
Given a sample transition $(x, a, r, x')$, we compute the Bellman update $\hat \mathcal{T} z_j := r + \gamma z_j$ for each atom $z_j$, then distribute its probability $p_j(x', \pi(x'))$ to the immediate neighbours of $\hat \mathcal{T} z_j$.
The $i^{th}$ component of the projected update $\Phi \hat \mathcal{T} Z_\theta(x,a)$ is
\begin{small}
\begin{equation}
(\Phi \hat \mathcal{T} Z_\theta(x,a))_i = \sum_{j=0}^{N-1} \left[ 1 - \frac{| [ \hat \mathcal{T} z_j ]^{V_{\textsc{max}}}_{V_{\textsc{min}}} - z_i|}{\triangle z} \right]^1_0 p_j(x', \pi(x')),
\label{eqn:cat_proj}
\end{equation}
\end{small}
where $[ \cdot ]^b_a$ bounds its argument in the range $[a, b]$.\footnote{Algorithm \ref{alg:cql} computes this projection in time linear in $N$.} As is usual, we view the next-state distribution as parametrized by a fixed parameter $\tilde \theta$. The sample loss $\mathcal{L}_{x,a}(\theta)$ is the cross-entropy term of the KL divergence
\begin{equation*}
D_{\textsc{kl}}(\Phi \hat \mathcal{T} Z_{\tilde \theta}(x,a) \, \| \, Z_\theta(x,a)),
\end{equation*}
which is readily minimized e.g. using gradient descent. We call this choice of distribution and loss the \emph{categorical algorithm}. When $N = 2$, a simple one-parameter alternative is
$\Phi \hat \mathcal{T} Z_\theta(x,a) := [\expect [\hat \mathcal{T} Z_\theta(x,a)] - V_{\textsc{min}}) / \triangle z]^1_0;$
we call this the \emph{Bernoulli algorithm}.
We note that, while these algorithms appear unrelated to the Wasserstein metric, recent work \citep{bellemare17cramer} hints at a deeper connection.

\begin{algorithm}[ht]
\caption{Categorical Algorithm}\label{alg:cql}
\begin{algorithmic}
\INPUT A transition $x_t, a_t, r_t, x_{t+1}$, $\gamma_t \in [0, 1]$
\STATE $Q(x_{t+1}, a) := \sum\nolimits_i z_i p_i(x_{t+1}, a)$
\STATE $a^* \leftarrow \argmax_a Q(x_{t+1}, a)$
\STATE $m_i = 0, \quad i \in 0, \dots, N-1$
\FOR{$j \in 0, \dots, N-1$}
	\STATE \textcolor{gray}{\# Compute the projection of $\hat \mathcal{T} z_j$ onto the support $\{ z_i \}$}
	\STATE $\hat \mathcal{T} z_j \leftarrow \left [ r_t + \gamma_t z_j \right ]^{V_{\textsc{max}}}_{V_{\textsc{min}}}$
	\STATE $b_j \leftarrow (\hat \mathcal{T} z_j - V_{\textsc{min}}) / \Delta z$ \; \textcolor{gray}{\# $b_j \in [0, N-1]$}
	\STATE $l \leftarrow \lfloor b_j \rfloor$, $u \leftarrow \lceil b_j \rceil$
	\STATE \textcolor{gray}{\# Distribute probability of $\hat \mathcal{T} z_j$}
	\STATE $m_l \leftarrow m_l + p_j(x_{t+1}, a^*) (u - b_j)$
	\STATE $m_u \leftarrow m_u +  p_j(x_{t+1}, a^*) (b_j - l)$
\ENDFOR
\OUTPUT $-\sum_i m_i \log p_i(x_t, a_t)$ \; \textcolor{gray}{\# Cross-entropy loss}
\end{algorithmic}
\end{algorithm}

\begin{figure*}[htb]
\begin{center}
\includegraphics[width=.95\textwidth]{figures/Varying-Number-Canonicals.pdf}%
\caption{Categorical DQN: Varying number of atoms in the discrete distribution. Scores are moving averages over 5 million frames.}
\label{fig:atari_categorical}
\end{center}
\end{figure*}

\section{Evaluation on Atari 2600 Games}

To understand the approach in a complex setting, we applied the categorical algorithm to games from the Arcade Learning Environment \citep[ALE;][]{bellemare13arcade}. While the ALE is deterministic, stochasticity does occur in a number of guises: 1) from state aliasing, 2) learning from a nonstationary policy, and 3) from approximation errors.
We used five training games (Fig \ref{fig:atari_categorical}) and 52 testing games.

For our study, we use the DQN architecture \citep{mnih15nature}, but output the atom probabilities $p_i(x,a)$ instead of action-values, and chose $V_{\textsc{max}} = -V_{\textsc{min}} = 10$ from
preliminary experiments over the training games. We call the resulting architecture \emph{Categorical DQN}.
We replace the squared loss $(r + \gamma Q(x',\pi(x')) - Q(x,a))^2$ by $\mathcal{L}_{x,a}(\theta)$ and train the network to minimize this loss.\footnote{For $N=51$, our TensorFlow implementation trains at roughly 75\% of DQN's speed.} As in DQN, we use a simple $\epsilon$-greedy policy over the expected action-values; we leave as future work the many ways in which an agent could select actions on the basis of the full distribution. The rest of our training regime matches \citeauthor{mnih15nature}'s, including the use of a target network for $\tilde \theta$.

Figure \ref{fig:space_invaders_value_distribution} illustrates the typical value distributions we observed in our experiments. In this example, three actions (those including the button press) lead to the agent releasing its laser too early and eventually losing the game. The corresponding distributions reflect this: they assign a significant probability to 0 (the terminal value). The safe actions have similar distributions (\textsc{left}, which tracks the invaders' movement, is slightly favoured). This example helps explain
why our approach is so successful: the distributional update keeps separated the low-value, ``losing'' event from the high-value, ``survival'' event, rather than average them into one (unrealizable) expectation.\footnote{Video: \url{http://youtu.be/yFBwyPuO2Vg}.}

One surprising fact is that the distributions are not concentrated on one or two values, in spite of the ALE's determinism, but are often close to Gaussians. We believe this is due to our discretizing the diffusion process induced by $\gamma$.
\begin{figure}[htb]
\begin{center}
\includegraphics[width=3.2in]{figures/Space-Invaders-Distribution-Visualization.pdf}  %
\end{center}
\caption{Learned value distribution during an episode of \textsc{Space Invaders}. Different actions are shaded different colours. Returns below 0 (which do not occur in \textsc{Space Invaders}) are not shown here as the agent assigns virtually no probability to them.\label{fig:space_invaders_value_distribution}}
\end{figure}

\subsection{Varying the Number of Atoms}

We began by studying our algorithm's performance on the training games in relation to the number of atoms (Figure \ref{fig:atari_categorical}). For this experiment, we set $\epsilon = 0.05$. From the data, it is clear that using too few atoms can lead to poor behaviour, and that more always increases performance; this is not immediately obvious as we may have expected to saturate the network capacity. The difference in performance between the 51-atom version and DQN is particularly striking: the latter is outperformed in all five games, and in \textsc{Seaquest} we attain state-of-the-art performance. As an additional point of the comparison, the single-parameter Bernoulli algorithm performs better than DQN in 3 games out of 5, and is most notably more robust in \textsc{Asterix}.

One interesting outcome of this experiment was to find out that our method does pick up on stochasticity. \textsc{Pong} exhibits intrinsic randomness: the exact timing of the reward depends on internal registers and is truly unobservable. We see this clearly reflected in the agent's prediction (Figure \ref{fig:pong_bimodality}): over five consecutive frames, the value distribution shows two modes indicating the agent's belief that it has yet to receive a reward. Interestingly, since the agent's state does not include past rewards, it cannot even extinguish the prediction after receiving the reward, explaining the relative proportions of the modes.

\begin{figure*}[htb]
\begin{center}
\includegraphics[width=1.25in]{figures/pong_aliasing_0.png}
\includegraphics[width=1.25in]{figures/pong_aliasing_1.png}
\includegraphics[width=1.25in]{figures/pong_aliasing_2.png}
\includegraphics[width=1.25in]{figures/pong_aliasing_3.png}
\includegraphics[width=1.25in]{figures/pong_aliasing_4.png}
\caption{Intrinsic stochasticity in \textsc{Pong}.\label{fig:pong_bimodality}}
\end{center}
\vspace{-1em}
\end{figure*}

\subsection{State-of-the-Art Results}

The performance of the 51-atom agent (from here onwards, C51) on the training games, presented in the last section, is particularly remarkable given that it involved none of the other algorithmic ideas present in state-of-the-art agents. We next asked whether incorporating the most common hyperparameter choice, namely a smaller training $\epsilon$, could lead to even better results. Specifically, we set $\epsilon = 0.01$ (instead of $0.05$); furthermore, every 1 million frames, we evaluate our agent's performance with $\epsilon = 0.001$.

We compare our algorithm to DQN ($\epsilon = 0.01$), Double DQN \citep{vanhasselt16deep}, the Dueling architecture \citep{wang2016dueling}, and Prioritized Replay \citep{schaul16prioritized}, comparing the best evaluation score achieved during training.
We see that C51 significantly outperforms these other algorithms (Figures \ref{fig:perc_scores} and \ref{fig:allatari}). In fact, C51 surpasses the current state-of-the-art by a large margin in a number of games, most notably \textsc{Seaquest}. One particularly striking fact is the algorithm's good performance on sparse reward games, for example \textsc{Venture} and \textsc{Private Eye}. This suggests that value distributions are better able to propagate rarely occurring events. Full results are provided in the appendix.

We also include in the appendix (Figure \ref{fig:train_sup}) a comparison, averaged over 3 seeds, showing the number of games in which C51's training performance outperforms fully-trained DQN and human players. These results continue to show dramatic improvements, and are more representative of an agent's average performance. Within 50 million frames, C51 has outperformed a fully trained DQN agent on 45 out of 57 games. This suggests that the full 200 million training frames, and its ensuing computational cost, are unnecessary for evaluating reinforcement learning algorithms within the ALE.

The most recent version of the ALE contains a stochastic execution mechanism designed to ward against trajectory overfitting.%
Specifically, on each frame the environment rejects the agent's selected action with probability $p=0.25$.
Although DQN is mostly robust to stochastic execution, there are a few games in which its performance is reduced. On a score scale normalized with respect to the random and DQN agents, C51 obtains mean and median score improvements of $126\%$ and $21.5\%$ respectively, confirming the benefits of C51 beyond the deterministic setting.

\begin{figure}
\begin{tabular}{ l | r | r | r | r }
\multicolumn{1}{c}{} & \mbox{\textbf{Mean}} & \mbox{\textbf{Median}} & $>\, $\mbox{\textbf{H.B.}} & $>\,$\mbox{\textbf{DQN}} \\
\hline
\textsc{dqn}  &   228\% & 79\% & 24 & 0 \\
\textsc{ddqn}   &   307\% & 118\% & 33 & 43 \\
\textsc{Duel.}   &   373\% & 151\% & 37 & 50 \\
\textsc{Prior.}   &   434\% & 124\% & 39 & 48 \\
\textsc{Pr. Duel.}   &   592\% & 172\% & 39 & 44 \\
\hline
\textsc{C51}   &   \textbf{\textcolor{blue}{701\%}} & \textbf{\textcolor{blue}{178\%}} & \textbf{\textcolor{blue}{40}} & \textbf{\textcolor{blue}{50}} \\
\hline
\hline
\textsc{unreal}$^\dagger$ & 880\% & 250\% & - & - \\
\end{tabular}
\caption{Mean and median scores across 57 Atari games, measured as percentages of human baseline \citep[H.B.,][]{nair15massively}.}
\label{fig:perc_scores}
\end{figure}

\vspace{-5pt}
\begin{figure}[htb]
\begin{center}
\includegraphics[width=0.47\textwidth]{figures/all_atari_games_flat.png}
\caption{Percentage improvement, per-game, of C51 over Double DQN, computed using \citeauthor{vanhasselt16deep}'s method.}
\label{fig:allatari}
\end{center}
\vspace{-1em}
\end{figure}

\section{Discussion}\label{sec:discussion}

In this work we sought a more complete picture of reinforcement learning, one that involves value distributions.
We found that learning value distributions is a powerful notion that allows us to surpass most gains previously made on Atari 2600, without further algorithmic adjustments.

\subsection{Why does learning a distribution matter?}
\blfootnote{$^\dagger$ The UNREAL results are not altogether comparable, as they were generated in the asynchronous setting with per-game hyperparameter tuning \citep{jaderberg17reinforcement}.}
It is surprising that, when we use a policy which aims to maximize expected return, we should see any difference in performance. The distinction we wish to make is that \emph{learning distributions matters in the presence of approximation}. We now outline some possible reasons.

\textbf{Reduced chattering.} Our results from Section \ref{sec:control} highlighted a significant instability in the Bellman optimality operator. When combined with function approximation, this instability may prevent the policy from converging, what \citet{gordon95stable} called \emph{chattering}. We believe the gradient-based categorical algorithm is able to mitigate these effects by effectively averaging the different distributions, similar to conservative policy iteration \citep{kakade02approximately}. While the chattering persists, it is integrated to the approximate solution.

\textbf{State aliasing.} Even in a deterministic environment, state aliasing may result in effective stochasticity.
\citet{mccallum95reinforcement}, for example, showed the importance of coupling representation learning with policy learning in partially observable domains. We saw an example of state aliasing in \textsc{Pong}, where the agent could not exactly predict the reward timing. Again, by explicitly modelling the resulting distribution we provide a more stable learning target.

\textbf{A richer set of predictions.} A recurring theme in artificial intelligence is the idea of an agent learning from a multitude of predictions (\citealt{caruana97multitask,utgoff02many,sutton11horde,jaderberg17reinforcement}). The distributional approach naturally provides us with a rich set of auxiliary predictions, namely: the probability that the return will take on a particular value. Unlike previously proposed approaches, however, the accuracy of these predictions is tightly coupled with the agent's performance.

\textbf{Framework for inductive bias.} The distributional perspective on reinforcement learning allows a more natural framework within which we can impose assumptions about the domain or the learning problem itself. In this work we used distributions with support bounded in $[V_{\textsc{min}}, V_{\textsc{max}}]$. Treating this support as a hyperparameter allows us to change the optimization problem by treating all extremal returns (e.g. greater than $V_{\textsc{max}}$) as equivalent.
Surprisingly, a similar value clipping in DQN significantly degrades performance in most games.
To take another example: interpreting the discount factor $\gamma$ as a proper probability, as some authors have argued, leads to a different algorithm.

\textbf{Well-behaved optimization.} It is well-accepted that the KL divergence between categorical distributions is a reasonably easy loss to minimize. This may explain some of our empirical performance. Yet early experiments with alternative losses, such as KL divergence between continuous densities, were not fruitful, in part because the KL divergence is insensitive to the values of its outcomes. A closer minimization of the Wasserstein metric should yield even better results than what we presented here.

In closing, we believe our results highlight the need to account for distribution in the design, theoretical or otherwise, of algorithms.

\section*{Acknowledgements}

The authors acknowledge the important role played by their colleagues at DeepMind throughout the development of this work. Special thanks to Yee Whye Teh, Alex Graves, Joel Veness, Guillaume Desjardins, Tom Schaul, David Silver, Andre Barreto, Max Jaderberg, Mohammad Azar, Georg Ostrovski, Bernardo Avila Pires, Olivier Pietquin, Audrunas Gruslys, Tom Stepleton, Aaron van den Oord; and particularly Chris Maddison for his comprehensive review of an earlier draft. Thanks also to Marek Petrik for pointers to the relevant literature, and Mark Rowland for fine-tuning details in the final version.

\section*{Erratum}
The camera-ready copy of this paper incorrectly reported a mean score of 1010\% for C51. The corrected figure stands at 701\%, which remains higher than the other comparable baselines. The median score remains unchanged at 178\%.

The error was due to evaluation episodes in one game (Atlantis) lasting over 30 minutes; in comparison, the other results presented here cap episodes at 30 minutes, as is standard. The previously reported score on Atlantis was 3.7 million; our 30-minute score is 841,075, which we believe is close to the achievable maximum in this time frame. Capping at 30 minutes brings our human-normalized score on Atlantis from 22824\% to a mere (!) 5199\%, unfortunately enough to noticeably affect the mean score, whose sensitivity to outliers is well-documented.

\begin{small}

redpajama_set_name: RedPajamaArXiv


K'Motion Dance Studio is a dance studio in Canberra run by Karen Rutherfoord operating at Corazon Studio.
They run courses in various styles including Salsa, Rueda, Zouk Lambada, Tango. They also run short courses and workshops in other styles including Samba de Gafieira, Kizomba, Bachata and Cha Cha.
They run regular events such as Salsa @ Unipub and Zouk Revolution.


text:
Dr. Timothy Johnson
Senior Medical Contributor for ABC News
Speaker Categories: Healthcare Industry | Healthcare Policy | Medical Professional | Stress Management | Science | Life Balance | Media
Travels From: NY, United States.
Speaker Fee Range: $10,001 to $20,000*
Dr. Timothy Johnson Bio
Dr. Timothy Johnson is a Senior Medical Contributor for ABC News and one of the nation's leading medical communicators of healthcare information. Previously, Dr. Johnson was the Chief Medical Editor for ABC providing on-air medical analysis for "World News," "Nightline" and "20/20." He has provided commentary on medical problems and answers for viewers of ABC News' "Good Morning America" since the program's debut in 1975.
Healthcare Catalyst
Dr. Johnson has helped spearhead several division-wide reporting efforts on a range of key issues, including practical ways to fix the American healthcare system and the fight against cancer, diabetes and Alzheimer's disease.
Award-Winning Journalist
Dr. Johnson's programs and feature reports have won several awards, including an Emmy Award from the National Academy of Television Arts and Sciences, as well as two Emmy Awards from the Boston/New England Chapter of the same organization. In 1999 he earned a Gabriel Award in the best news story category for the two-part piece, "Alzheimer's: A Faded Memory." Dr. Johnson was selected as the recipient of the 1998 Bradford Washburn Award presented by the Museum of Science, Boston -- an honor also bestowed upon Jacques Cousteau, Walter Cronkite, Dr. Jane Goodall and Alan Alda, among others. In 1988 he received the Lewis Thomas Award for Communications from the American College of Physicians, and in 1987 and 1989 he won the Howard W. Blakeslee Award given by the American Heart Association.
Dr. Johnson is the founding editor of the Harvard Medical School Health Letter and co-editor of the Harvard Medical School Health Letter Book. He is also co-editor of the bookYour Good Health, co-author of Let's Talk and author of Dr. Timothy Johnson's On-Call Guide to Men's Heath and Finding God in the Questions: A Personal Journey.
Originally headed for the ministry, Dr. Johnson graduated in from North Park Seminary in 1963, then decided two years later to enter medicine. A Phi Beta Kappa graduate of Augustana College, he graduated summa cum laude from Albany Medical College and holds a master's degree in public health from Harvard University. He has had appointments at both Harvard Medical School and Massachusetts General Hospital.
The Truth About Getting Sick in America
In this keynote, Dr. Johnson talks specifically about Health Care Reform in America, and about four of the most commonly expressed worries over the health care legislation that was passed by Congress.
The Truth About Getting Sick in America: The Real Problems with Health Care and What We Can Do Purchase Book
Dr. Timothy Johnson's On Call Guide to Men's Health: Authoritative Answers to Your Most Important Questions Purchase Book
Finding God in the Questions: A Personal Journey Purchase Book
Edward Hughes
Eileen McDargh
Gail Wilensky
Jeff Goldsmith
Miriam Nelson
Pamela Peeke
Susan Blumenthal
LeAnn Thieman
Bowen White
John Kenagy
Christine Dumas
Cass Wheeler
Hadassah Lieberman
Chris Trimble
Rosemary Gibson
Jeff Evans
David Nash
John Nosta

redpajama_set_name: RedPajamaCommonCrawl


If all was good and right with the world, Scotland's The Machine Room would be on the tip of every indie music lover's tongue. The band have been releasing amazing music since 2011 -- if you missed out on 2012's Love From a Distance EP, do yourself a favor and remedy that now. On March 17th, The Machine Room returns with a new EP titled Sweden which is named after (and includes) the band's 2013 single. The EP will be preceded by the new single "Chances" on March 3rd, but the song is available to preview now. Stream "Chances" below along with some of the Machine Room's previously released gems below (plus grab a free download of their cover of New Order's "Bizarre Love Triangle"). Catch up with The Machine Room on Facebook and Twitter.


The new Golden Eagle will be the second richest race in Australia.
The Golden Eagle, a new and unique race with prizemoney second only to The Everest in Australian racing, is the centrepiece of an expansion of the Sydney spring carnival.
Racing NSW and Australian Turf Club this morning announced a series of significant spring carnival race programming chances including the introduction of the $7.5 million Golden Eagle, to be run over 1500m at Rosehill Gardens on November 2 next year.
The Golden Eagle, which is open to four-year-olds only, has more prizemoney than the $7.3 million Melbourne Cup and complements the $14 million The Everest, giving Sydney's spring carnival now the two richest horse races in the southern hemisphere.
Of the $7.5 million Golden Eagle prizemoney, 10% will be donated to charity. The connections of each runner will nominate a charity that has been accredited with Racing NSW to receive 10% of the prizemoney won by their horse. Accordingly, a total of $750,000 will go to charity.
For example, as prizemoney for first is $4.1 million, the charity which has the winning horse would receive $410,000.
"The Golden Eagle is the first feature race on the Australian (turf) calendar exclusively for four-year-old horses," said Racing NSW Chairman Russell Balding AO.
"We believe the Golden Eagle will be attractive in retaining our top three-year olds to race on as four-year olds and benefit the racing industry, underpinning the strength of racing in NSW and Australia more broadly.
The Golden Eagle will also be the third leg of the new "Golden Slam" at Rosehill, with a $5 million bonus offered to any horse that can win the Golden Slipper at two, the Golden Rose at three and the Golden Eagle as a four-year-old.
"The Golden Eagle is an exciting development, which will bring together the best sprinter-milers of their generation at an age when they have fully matured," said ATC Chairman Matt McGrath.
Racing NSW chief executive Peter V'landys said the Golden Eagle prizemoney will be funded entirely from new revenues, rather than drawing on existing revenues of the NSW thoroughbred racing industry. This new revenue stream will have a separate announcement.
"I am delighted that this enables the Golden Eagle to be added as an important feature race at Rosehill and will also distribute $750,000 to charities,'' V'landys said.
"Now having the second richest race on turf in the world, Sydney's west has the perfect day to celebrate."
The Golden Eagle headlines changes to the program for spring racing in Sydney which will feature a race that highlights each Saturday meeting through to November 9.
This also includes a new $1 million three-year-old and a $1 million two-year-old race highlighting Saturday race programs in October and November.
The new $1 million Bondi Stakes will be run at Royal Randwick for three-year olds over a distance of 1600m on October 26.
The race provides a new spring target for three-year olds in Sydney and follows on from races such as the Golden Rose and Flight Stakes.
The Golden Gift, 1100m for two-year olds which was first run by the ATC in 2015 will now be run for $1 million prizemoney on 9 November 2019. This race provides a focal point for two-year-old racing in the Spring, with the prizemoney on offer enough to ensure that the first and second placegetters will gain a berth in the Golden Slipper.
Should one horse win the Golden Slipper, Golden Rose and Golden Eagle connections will pocket a $5 million Golden Bonus.


In our house everywhere we look, you have toy cars, trucks and train cars lined up in rows. Along the sofa, across the coffee table, end to end in all shapes and sizes. While taking a quick about turn to exit your bedroom the other day, I noticed a line-up on top of your toy chest. Suddenly my head snapped back to take a second look "woe that's out of context – I was looking for that earlier". In doing so I also said to myself "clever little guy"! You actually had a lipstick tube lined up amongst the other vehicles.
Oh my goodness Patrick, we are just amazed at the amount of information you have managed to absorb about your new found passion. Wheels, wipers, lights, rims, trim – eyes wide and beaming with excitement like a salesperson – you've got the gab down to a science. Our adventures around the city are quite informative these days, as you announce each and every brand on the road, interspersed with "what's dat, what's dat". Not sure where this is all going, but at least like the vehicles – it looks as though we're heading in the right direction.


JOINmeUP CherryxSophie livejasmin. S3xyStud. SexyLovGames. JadenLeon.
WhippedcreamerMISSNIKKILEEHighSensationSuperMolly .JasminAndGabi3alexxasquirttohotforubabyMonterHugeCock .esmeraldasex25UrSexEscapeEmmaAngelicaClaryceXo .EmmaAngelicaAmeliSEXXXLinaBeckerASIANLUV2CUM .EriiikaDyaMissS3xyStudSexyIrinkaa .FrankBigDickLollyTeenTatitstohotforubaby .EriiikaCityBBPoisonEssencesAsianFunnyGirls .MarinahoneyAzaliDivaMarinahoneyBabyBoy555 .01MissNIKKILEECuteBoy469CarolineSpankyFrankBigDick .HeadyEvelynUrSexEscapebadnwetHannaMyers .LuxuryJhenbadnwetFoxiRoxyEmmaHazel .
VelMilerpregnantLAURALollyTeenNancyBurton .QustumaDavidSaint01MissNIKKILEEAnnamilla .DoubleSugarSuperMollyVelMilerNancyBurton .JessLittleMarinahoneySexyLovGamesHannaMyers .AsianMILKSHAKEHannaMyersRitah2AngelsCock .xLuxuriousGirlxQustumaFeminiboyRitah .UrSexEscapeBabyBoy555tsrudevsmenDivineSofi24 .EriiikaRitahLollyTeenMiaSunshinee .GhiaGoldenAsianFunnyGirlsNiraWolffeMiaSunshinee .LOVERSFANTASYEmmaAngelicaAmyRavenFuqBunnies .Angel95cuteandrydany1KudruliaDoubleSugar .


United States Map With Capital Cities Fresh Us Map States With Capitals Us Puzzle Little Save Map Us States And is just one of the many collections of Sample Travel Route Map Reference that we have on this website. We have a lot of Travel Route Map or journey maps and any other things concerning in this website. We're not just providing info about , but , you can get a lot more reference to Travel Route Map as well. So , don't forget to keep visiting gotravelingabroad.com to get the latest update about Travel Route Map , Journey Maps , Roads maps and more.
United States Map With Capital Cities Fresh Us Map States With Capitals Us Puzzle Little Save Map Us States And was posted in March 1, 2017 at 9:08 pm. United States Map With Capital Cities Fresh Us Map States With Capitals Us Puzzle Little Save Map Us States And has viewed by 39 users. Click it and download the United States Map With Capital Cities Fresh Us Map States With Capitals Us Puzzle Little Save Map Us States And.
Maps, Blank Map Of The United States was posted April 29, 2017 at 7:18 am by gotravelingabroad.com . More over Blank Map Of The United States has viewed by 1473 visitor.
Maps, County Map Alabama With Cities was posted April 16, 2017 at 7:18 am by gotravelingabroad.com . More over County Map Alabama With Cities has viewed by 2175 visitor.
Maps, North America Map Black And White was posted March 13, 2017 at 8:08 pm by gotravelingabroad.com . More over North America Map Black And White has viewed by 3073 visitor.


text:
NDA Eligibility 2023: NDA Age Limit, Qualification, Height
ByNaveen Singh | Updated on: Jan 30, 2023
Check the latest NDA eligibility 2023 including all the parameters such as qualification, NDA age limit, height, physical standards, etc. Get to know the NDA eligibility for females and males as prescribed in the official notification.
OverviewApply OnlineAdmit CardAnalysisQuestion PapersCut OffSyllabusVacancyEligibility CriteriaSelection ProcessSalary & Job ProfileBooksStudy MaterialInterviewAnswer KeyResultExam Pattern
NDA Eligibility 2023 will be released along with the official notification. There have been no significant changes in the eligibility parameters over the years except for the age limit. Candidates can check their eligibility here, including various parameters such as educational qualification, NDA age limit, nationality, marital status, and physical and medical standards. The Union Public Service Commission sets it for those who applied for the upcoming exam. It is a must for every candidate to fulfill the prescribed age limit and other NDA eligibility criteria before filling out the application form. Without that, their candidature might get cancelled at any stage of the recruitment process.
According to the NDA eligibility, female and male candidates who have completed their 12th or equivalent can apply for the exam. Before applying, interested male and female applicants must go through the detailed parameters carefully, which are provided here.
NDA Eligibility 2023
NDA Age Limit
NDA Eligibility for Female
NDA Qualification: 12th Percentage & Marks
NDA Eligibility Criteria 2023: Nationality
NDA Eligibility: Marital Status
NDA Physical Eligibility
NDA Height Weight and Chest Standards
NDA Eligibility - Vision Standards
FAQs on NDA Eligibility
NDA Eligibility Criteria 2023 is one of the main aspects to go through before applying for the NDA Exam. The candidate must meet all the eligibility requirements to move further in the recruitment process. Both male and female candidates can apply for the exam after looking at the vacancies and whether they fulfill all eligible parameters or not. Those applicants who are not born earlier than 2 July 2004 or later than 1 July 2007 can apply for the exam as per NDA eligibility.
The candidates need to fulfill all the aforementioned NDA eligibility to proceed further with the selection process for the exam. Note that the candidates applying for the Navy and the Air Force might have different eligibility requirements than those for the Army.
According to the NDA eligibility criteria, only unmarried male and female aspirants aged between 16.5 - 19.5 are eligible to apply for the exam. Those interested in applying for this exam must fall in the mentioned age range. Candidates exceeding the NDA age limit will not be considered eligible for the exam.
NDA age limit 2023 has been released with the official notification on 21 December 2022. Here are the NDA eligibility details for all candidates who wish to appear for the upcoming exam:
The NDA age limit of aspirants must be between 16.5 - 19.5 years.
The aspiring candidate must have not been born earlier than 2 July 2004 or later than 1 July 2007.
NDA Age Limit for Girls
The NDA age limit for girls is the same as that of male candidates. As per the NDA eligibility, both male and female candidates who are unmarried and born between 2 July 2004 to 1 July 2007 are eligible to apply for the exam.
After a petition filed by Kush Kalra in 2021, the Supreme Court of India passed an interim order to allow female candidates to apply for the exam. If you wish to know the NDA Eligibility for Female candidates, refer to the points listed below:
According to UPSC, the NDA eligibility for female candidates is the same as for male candidates.
The Height, Weight, and Vision requirements for female candidates applying for the exam remain the same as for male candidates.
The NDA qualification for females and marital status shall remain the same as for male candidates.
The physical standard required for female candidates shall remain the same as for male candidates.
The NDA qualification required is different for candidates applying to the other army wings. Take a look at the detailed NDA eligibility criteria of educational qualification for candidates applying for the Army, Navy, and Air Force below:
Required NDA Qualification
Candidate must be appearing/passed Class 12 or equivalent examination from a recognized board.
Air Force & Naval wings of NDA and 10+2 Cadet entry scheme at INA
Candidate must be appearing/cleared his class 12 or equivalent examination with Physics and Mathematics.
NDA Eligibility Criteria Note:
Those who are appearing in Class 12 or equivalent examination can also apply for the exam.
The candidates who are appearing in the Class 11 exam are not eligible to apply for the NDA examination.
According to NDA eligibility, an applicant who is an Indian citizen is eligible to apply for the upcoming examination. Those who are not a citizen of India and still want to apply for the exam must meet the UPSC NDA eligibility criteria listed below:
The candidate applying for the exam must be either:
A Citizen of India
A Subject of Nepal or
A subject of Bhutan, or,
Tibetan refugees who came to India before 1 Jan. 1962, with the intention of permanent settlement.
A person of Indian origin who migrated from the following countries with the intention of permanent settlement:
Pakistan, Sri Lanka
East African Countries of Kenya, Uganda, United Republic of Tanzania, Zambia, Malawi, Zaire & Ethiopia, and Vietnam.
Get a better overview of NDA eligibility by referring to the following table:
NDA Eligibility Details
Can Apply or Not?
Citizen of India
Subject of Bhutan
Subject of Nepal
Tibetan Refugee who came to India before 1 Jan 1962
Tibetan Refugee who came to India after 1 Jan 1962
Indian Origin citizen who migrated from Pakistan, Sri Lanka
Indian Origin citizen who migrated from Kenya, Uganda, United Republic of Tanzania, Zambia, Malawi, Zaire & Ethiopia, Vietnam
Indian Origin citizen who migrated from other countries in the world
NDA Nationality
Here are some other important points about NDA eligibility regarding nationality:
The candidates must fulfil the NDA nationality criteria to be able to get through the recruitment process.
It should be clear that the applicant who belongs to categories 2, 3, 4, and 5 must have a certificate of NDA eligibility that has been issued by the Government of India.
Gorkha applicants who are subjects of Nepal however, do not need to produce this certificate.
Marital status plays a crucial role in NDA eligibility criteria. Only unmarried male and female candidates can apply for the examination. For a detailed NDA eligibility, check the list mentioned below:
Only unmarried male & female candidates are allowed to apply for the exam.
All candidates who pass the exam are not allowed to marry until they complete their training in the Academy.
The candidate who marries during the training period will be discharged and will have to refund all the expenditure spent on them by the Government of India.
Physical Fitness is an important NDA eligibility criteria for the exam based on which candidates are shortlisted. As per NDA eligibility, the candidates must be medically fit to get through the selection process and closer to their dreams.
It is recommended that they go for a detailed medical examination before submitting their application. Here are the guidelines for Physical Test requirements based on the NDA Eligibility:
The applicant must have good physical and mental health and should be free from diseases and disabilities.
The candidates are advised to rectify minor defects/ailments.
Permanent body tattoos are only allowed from the wrist to the inside of the elbow and on the hand's backside. Tattoos on any other body part are not acceptable.
The NDA eligibility criteria for candidates belonging to tribes who have tattoos/marks on the face/body as part of their custom/traditions allow them relaxation. Such candidates will be allowed to appear for the exam on a special case basis.
The chest should not be less than 81 cm after full expansion. The expansion of the chest after full inspiration should be more than 5 cm. X-ray of the chest will be taken to examine any kind of disease.
Bones/Joints should not be impaired of any function. These should be developed properly.
As per the NDA eligibility 2023, the candidate should not have any history of mental breakdown/fits.
Un-operated Hernia will lead to disqualification. In the case of an operated Hernia, it should be operated on 1 year before the final medical examination.
The Liver and Spleen of the candidate should be of normal size.
The hearing should be normal. The candidate should be able to hear a whisper with each ear at a distance of 610 cm. There should not be any disease of the ear, nose, and throat.
Blood Pressure should be normal.
There should not be any functional or organic disease of the heart and blood vessels.
There should be no hydrocele, varicocele or piles, or any kind of skin disease.
Any kind of disease of the internal organs of the abdomen will lead to disqualification.
Examination of urine will also be done, and any abnormality, if found, will lead to disqualification.
The applicant should have a sufficient number of natural and sound teeth. There should be a minimum of 14 dental points.
USG abdomen examination will be performed, and any congenital structural anomaly will lead to disqualification.
NDA eligibility states that the candidate should not be of the weak constitution or overweight/underweight.
Height and weight play a major role in the NDA eligibility criteria. The candidate who is applying for the defence exam must be physically fit. NDA height and weight parameters are primary requirements of the candidates that are necessary to be fulfilled and no relaxation will be given on this ground except the ones mentioned by them. The NDA eligibility criteria for height and weight requirements are mentioned below in detail for different posts.
NDA Height & Weight for Army and Air Force
Take a look at the detailed requirements of the NDA height and weight to get complete knowledge of the eligibility criteria.
The minimum height required is 157 cm (Note for the Indian Air Force it is 162.5 cm).
There are certain exceptions for candidates based on the region from which they belong.
NDA Eligibility on Height and Weight
Height (in cms)
Weight (in kgs)
For Gorkhas and the candidates belonging to hills of North-Eastern parts of India, Garhwal, and Kumaon, the minimum height requirement is 5 cm less than the figure mentioned above.
According to the NDA eligibility 2023, there is a relaxation of 2 cm for the candidates belonging to Lakshadweep.
NDA Height and Weight for Navy
Take a look at the NDA height and weight requirements for the Indian Navy for an elaborate knowledge of the eligibility criteria:
NDA Eligibility 2023 for Pilot
The NDA physical standards for becoming a Pilot are slightly different than candidates applying for other posts. The elucidated details of NDA eligibility 2023 physical standards are mentioned here:
Leg Length: Minimum: 99 cms and Maximum: 120 cms
Thigh Length: Maximum of 64 cms
Sitting Height: Minimum - 81.50 cms and Maximum - 96 cms
A selected candidate for NDA represents our country and bears plenty of duties and responsibilities. For an applicant, it is crucially important to stand fit medical requirements. One of the major important NDA eligibility 2023 requirements is a medical test for vision standards. NDA medical test requirements for vision are:
The distance vision (corrected) should be 6/6 in the better eye and 6/9 in the worse eye.
Myopia of more than 2.5 D is not allowed.
Hypermetropia more than 3.5 D including astigmatism is not allowed.
Candidates applying for Indian Air Force must not wear spectacles.
Below are the visual standards for the Indian Navy and Indian Air Force as per the official NDA eligibility criteria:
NDA Eligibility on Vision Standards
Uncorrected without glass
Corrected with glass
6/6 (only for Hypermetropia)
Limits of Myopia
Limits of Hypermetropia
+2.00 D Sph
Limits of Colour Perception
Manifest Myopia
Nil Re
Tinoscopic Myopia
Colour Vision
CP-I (MLT)
+0.75 Cyl (within + 2.0 D.Max)
Near Vision
N-5 each eye
NDA Selection Process
What is NDA eligibility 2023?
It is essential for the candidates to get detailed knowledge of the NDA eligibility criteria 2023 and ensure satisfying all the parameters to be able to proceed further in the selection process. The details of the criteria are segregated into many stages such as nationality, NDA age limit criteria, educational qualification, marital status, height, and weight standards. The candidates can check the detailed notification for the prescribed NDA eligibility before filling in the application forms.
When will UPSC release the NDA eligibility 2023 criteria?
The Commission released the NDA eligibility 2023 on 21 December 2022 through the official notification for registration. It will contain all the details about the NDA age limit 2023, height, educational qualification required, weight, vision, etc., based on which candidates can fill out the application form.
What is the NDA age limit?
It is essential for the candidates to satisfy and fulfil every criterion for advancing further in the recruitment process. The NDA age limit for the aspirants must be between 16.5-19.5 years to apply for the exam. The candidate must have not been born earlier than 2 July 2004 and not later than 1 July 2007.
What is the marital status in accordance with the NDA eligibility criteria?
Marital status plays a crucial role in NDA eligibility. The candidate must satisfy the prescribed criteria to get through the selection process. Only unmarried male and female candidates aged between 16.5 - 19.5 are eligible to apply for the exam. For a detailed list of rules, check the list mentioned below-
Only unmarried male & female candidates can apply for the examination.
Candidates should not marry until they complete their training in the Academy.
The candidate who marries during the training will be discharged and have to refund all expenditure spent on him by the government of India.
What is the minimum NDA qualification required to apply for the exam?
NDA consists of three sections- Army, Air Force, and Navy. Each of them requires different specialities as each of them differs from the other. The minimum NDA qualification required for aspirants are as follows:
For Army: The aspirant must be appearing or passed Class 12 or equivalent examination from a recognized board.
For Air Force & Navy: Candidate must be appearing or cleared his class 12 or equivalent examination with Maths and Physics.
What are the visual standards as per NDA eligibility?
A selected candidate for NDA represents our country and bears plenty of duties and responsibilities. For an applicant, it is crucially important to stand fit medical requirements as per the NDA eligibility. One of the major important requirements is a medical test for vision standards. NDA eligibility criteria for the vision are:
The distance vision (corrected) should be 6/6 in the better eye and 6/9 in the worse eye
Myopia should not be more than 2.5 D
Hypermetropia should not be more than 3.5 D including astigmatism
Candidates who are applying for Air Force must not wear spectacles
Are female candidates eligible according to the NDA eligibility criteria?
Yes, female candidates can apply for the exam. They are eligible if they fulfil all the prescribed NDA eligibility criteria. But only unmarried females can apply for the exam. It is prohibited to marry until they complete their training in the Academy. Also, the candidate who marries during the training will be discharged and have to refund all expenditures spent on her by the government of India.
Is NDA eligibility for height and weight same for everyone?
NDA eligibility for weight and height is almost the same for everyone. There are a few exceptions based on where a candidate is from. It is a must for a candidate to go through these details meticulously to serve any unprecedented flaw in your application.
What are the parameters to qualify India Army Physical test based on NDA eligibility?
To qualify for the exam and join the Indian Army, the candidate will be screened based on various parameters such as height, weight, and medical examination of diseases or deformities. Weight, height, and visual standards are important in NDA eligibility criteria.
What is the minimum height required according to the NDA eligibility criteria?
NDA height and weight parameters are primary requirements of the candidates that are necessary to be fulfilled and no relaxation will be given on this ground except the ones mentioned by them. 157 cms is the minimum height required for male candidates (Note- for the Indian Air Force it is 162.5 cms) to apply for the exam.
What is the NDA eligibility for Female candidates?
Female candidates can now apply for the exam, and they are eligible if they fulfil all the prescribed criteria. The NDA Eligibility for Female candidates is as follows:
Only unmarried females can apply for the exam.
All the other NDA eligibility for female candidates remain the same as for male candidates.
Can 10th pass apply for NDA exam?
The minimum qualification required to apply for the recruitment exam is 12th (higher secondary) as per the NDA eligibility criteria prescribed by the officials. Therefore, 10th-passed candidates cannot apply for the exam unless they clear Grade 12.
NDAAir ForceIndian NavySSC OfficerCoast GuardOther ExamsMock TestPracticeOnline Course
NDA Eligibility Criteria
NDA Question Papers
NDA Books
NDA Study Material
NDA Quiz
NDA Practice Set
NDA SSB Interview
UPSC NDA Result
Indian Air Force Group X Y
Air Force Group X Y Online Form
Air Force Group X Y Syllabus
Air Force Group X Y Question Papers PDF
Indian Air Force Cut Off
Air Force Group X Y Salary
Air Force X Y Group Study Material
Indian Navy SSR MR AA
Indian Navy SSR AA Admit Card 2021
Indian Navy SSR AA Salary
Indian Navy SSR AA Selection Process
Indian Navy SSR AA Syllabus
Indian Navy SSR AA Preparation Tips
Indian Navy SSR AA Test Series
Indian Navy SSR MR AA Practice Set
Indian Navy SSR Officer Exam
Indian Coast Guard Recruitment
Indian Coast Guard Eligibility
Indian Coast Guard Apply Cut Off
Indian Coast Guard Question Papers
Indian Coast Guard Books
TGC Indian Army Recruitment
Assam Rifles Recruitment
Assam Rifles Apply Online
Assam Rifles Syllabus
Assam Rifles Exam Pattern
Indian Army TES
Indian Army Application Form
Indian Army Eligibility Criteria
Indian Army Cut Off
NDA & NA Mock Tests
NDA Mock Test
Indian Navy Mock Test
Indian Air Force Mock Test
NDA NA Quizzes
NDA NA Practice Set
NDA Online Coaching
Indian Air Force Online Coaching
NDA SSB Interview Online Coaching
Indian Navy SSR AA Online Coaching


text:
Pierre Janet
AKA Pierre Marie Félix Janet
Born: 30-May-1859
Birthplace: Paris, France
Died: 24-Feb-1947
Location of death: Paris, France
Cause of death: unspecified
Occupation: Psychologist, Philosopher
Executive summary: Founder of Automatic Psychology
French neurologist, psychiatrist, and philosopher who, independently of Sigmund Freud, originated the idea that hysterical symptoms were the result of subconscious beliefs that have been isolated and forgotten, thus disassociated from the mainstream of the individual's consciousness. According to Janet these beliefs and feelings were typically the result of painful experiences. He is also credited with introducing the terms "subconscious" and "disassociation" into psychology, and his work was an important influence on the early theories of Carl Jung. In addition his extensive work on hypnotism and hysteria, which influenced both Freud and his mentor Josef Breuer, led Janet to assert that both hysteria and the susceptibility toward hypnosis arose from inherited tendencies that led toward imbalances of psychic (i.e. "mental") energy and tension. Janet was a tremendous advocate for uniting the efforts and knowledge of academic psychology with the clinical work of physicians in treating mental patients, and he helped bring about such considerable cooperation between these groups in both the U.S. and France.
At the invitation of Jean-Martin Charcot, Janet became director of the Salpêtrière, Paris' largest mental hospital, serving there from 1890 to 1898. He taught at the Sorbonne and was also professor of experimental and comparative psychology at the Collège de France from 1902. In 1904 he founded the Journal de Psychologie Normal et Pathologique with friend Georges Dumas. Janet's best known works include L'Automatisme psychologique (1889), which laid the groundwork for Automatic Psychology, and his Les Obsessions et la psychasthénie (1903), which contributed the first description of psychasthenia.
L'Automatisme psychologique (1889, psychology)
L'état mental des hystériques (1892, psychology)
Les Obsessions et la psychasthénie (1903, psychology)
Principles of Psychotherapy (1924, psychology)


The spokesman for Attorney General Kathleen Kane is acknowledging that some agency employees are disappointed their boss wasn't ordered removed from office by the state Senate.
Press secretary Chuck Ardo on Thursday attributed the disappointment largely to all the attention brought by the criminal case against Kane, the suspension of her law license and lawmakers' efforts to remove her from office.
He says those employees want the spotlight on their agency to end.
A resolution to oust Kane failed Wednesday in the Senate.
Some of Kane's top deputies have publicly challenged her.
Some gave testimony that helped produce charges against her that she illegally leaked grand jury information and then lied about her actions. Some also question whether agency cases could be undermined by last year's suspension of Kane's law license. Kane has denied any wrongdoing.


Space Adventures royalty free music track is a track with airy guitars, drums, bass and pads that will make eagles fly gently into the evening sky! It is perfect for commercials, presentations, documentaries and other projects! With its positive tone and mood, this track will definitely transform your projects into something new and refreshing!


Born on May 22, 1969 in Gary, IN.
Ordained on May 29, 1999.
Parochial Vicar at Sacred Heart in Moline (1999). Parochial Vicar at Holy Trinity in Bloomington (2001).
Administrator at St. Dominic in Camp Grove and St. Patrick in Wyoming (2001). Pastor at St. Dominic in Wyoming and St. Patrick in Camp Grove and Administrator at St. John in Bradford (2002). While remaining pastor at St. Dominic in Camp Grove and St. Patrick in Wyoming, to Pastor at St. John in Bradford (2003). Pastor at Resurrection, St. Hyacinth, St. Patrick and Queen of the Holy Rosary in LaSalle and Sacred Heart in Dimmick (2006). Pastor of St. Hyacinth and St. Patrick, rector for Queen of the Holy Rosary Shrine all in LaSalle (2007). Pastor of St. Thomase More in Dalzell and St. Benedict in Ladd (2009). Pastor of St. Matthew in Champaign and St. Boniface in Seymour (2012). While remaining Pastor at St. Matthew in Champaign and St. Boniface in Seymour to Vicar Forane of the Champaign Vicariate (2016).


text:
10 Wednesday AM Reads
September 15, 2021 6:50am by Barry Ritholtz
My mid-week morning train WFH reads:
• How the Pandemic Helped Us Recover From the Great Recession For certain industries, 2021 will be their biggest revenue numbers ever, whether it's in motorcycles or boats or certain other industries. They never recovered from 2006, their prior peak revenue. And in 2021, some of these companies think they're going to have even higher revenue than that. And so in some weird way, the pandemic has actually led to the recovery from the last recession. (Slate)
• Lessons From the Rise and Fall of the Pedestrian Mall Car-free shopping streets swept many U.S. cities in the 1960s and '70s, but few examples survived. Those that did could be models for today's "open streets." (CityLab)
• Peter Thiel Gamed Silicon Valley, Donald Trump, and Democracy to Make Billions, Tax-Free In an exclusive excerpt from The Contrarian, a new biography, the disruption-preaching power broker is revealed as just another rich guy desperate to keep his fortune from the IRS. (Businessweek)
• Why most gas stations don't make money from selling gas With gas prices climbing up, you may think station owners are getting greedy. But the economics behind the pump tell a different story. (The Hustle)
• Inflation Is Popping From Sydney to San Francisco. It May Be a Good Sign. Inflation has surged across advanced economies. The shared experience underlines that price gains come from temporary drivers — for now. (New York Times) see also Inflation panic? Don't tell the bond market 5 and 10-year breakevens are circa 2.5 per cent — far below where year-on-year inflation has hit for the majority of the year — suggesting the bond market is very much in the Team Transitory camp at the moment. (FT)
• ETF Taxation In The Crosshairs ETF tax efficiency comes into play because when the Authorized Participants goes to ETF issuers with a block of ETF shares, the ETF issuer gets to decide exactly which shares of stock they hand back to the AP as part of the redemption. As a result, every time there's a redemption in-kind of an ETF, the tax basis of all the positions in the fund ratchets up a bit. Do this enough times, and essentially the ETF ends up with no overhanging implied capital gains to speak of. (ETF Trends)
• Ebooks Are an Abomination If you hate them, it's not your fault. If you hate ebooks like I do, that loathing might attach to their dim screens, their wonky typography, their weird pagination, their unnerving ephemerality, or the prison house of a proprietary ecosystem. If you love ebooks, it might be because they are portable, and legible enough, and capable of delivering streams of words, fiction and nonfiction, into your eyes and brain with relative ease. (The Atlantic)
• We've been radically underestimating the true cost of our carbon footprint The Biden administration needs to factor in climate change's cost in human lives — and what we owe to future generations. (Vox)
• So You've Been Canceled: A Yom Kippur Atonement Guide What I sometimes wonder — both in my role as a rabbi myself and as a denizen of our broader culture of accountability — is how my friend, or any one of us, can find a path back from shame to acceptance.(New York Times)
• How to Grieve for a Very Good Dog When my yellow Lab died last spring, I was flattened by an overwhelming sadness that's with me still. And that's normal, experts say, because losing a pet is often one of the hardest yet least acknowledged traumas we'll ever face. (Outside)
Be sure to check out our Masters in Business interview this weekend with Campbell R. Harvey, professor of finance at Duke University's Fuqua School of Business. He is a partner and senior advisor at Research Affiliates. He is best known for his work on Yield Curve Inversion & Recessions. His most recent publication is a new book on crypto and blockchain: "DeFi and the Future of Finance."
Should we rename High Yield?
Source: Jim Reid, Deutsche Bank
Sign up for our reads-only mailing list here.
December 21, 2011 Home sales bounce but off lower levels
August 23, 2016 Inflation (if you look in the right places)
February 8, 2010 Employment Chart Roundup
Full Disclosure: "The Barry Trade"
Venture Capital Money Everywhere


Want more abundance in your life? Here are 6 powerful oriental tips to tap into unlimited wealth.
Blast your home with good 'yang' energy. Start by being a host. Invite a group of caring friends who are positive and whose auras glow with happiness to come together to celebrate a happy event. Such 'AUSPICIOUS AURA PARTIES' will dissolve any stale or negative energy that has become stuck in your home.
This is perfect for boosting your house aura – and imagine your own personal aura expanding with ABUNDANCE when your guests bless you with their appreciation after the party!
For the 15 days of the waxing moon, draw the dollar sign on the palm of your hand with a red crayon or non-toxic marking pen. Focus and align your thoughts with a strong power of intent and positive energy as you draw the symbol of wealth and money.
While sea salt is a powerful cleanser, it is also a powerful agent to attract wealth! Find a small pouch and fill it with a few grains of sea salt and put it in your wallet. The wallet instantly becomes a cash magnet and ensures money does not leave unnecessarily.
Activate your career luck by creating an aquarium at the north corner of your home. It is important that you maintain it well. The graceful movements of the fish help provide a constant flow of 'yang' energy to boost your career.
Invite a beautiful citrine crystal into your life. It is a stone of abundance that never needs cleansing. It's a tonic for the solar plexus which governs your personal power. Power up your energy with this gift from Mother Nature.
Stop worrying and stop poverty programming. Develop a wealthy mindset; be relaxed rather than uptight about money. Be mindful of your attitudes and motivation towards money. The more positive and relaxed you are about money, the more likely you are able to make your finances grow.
Selina is a certified US Aura Master, principal aromatherapist at Singapore's Auramatherapy Shop, aura training consultant, tarot consultant, energy healer, relationship counsellor, channel practitioner and medium.


text:
Home Entertainment Instagram announces 'take a break' feature to help users 'reflect and reset'
Instagram announces 'take a break' feature to help users 'reflect and reset'
Gonzo TECH
Instagram recently announced a new feature that will apparently help users manage their time spent in-app. As the chief of the major social media network said in a new blog, parents will be able to see how much time their kids spend in-app and set time limits, while teenagers will get reminders to take a break.
For those of you who don't know, Instagram has been under increasing pressure over the use of the platform, especially by teens, over the past couple of months. Some research showed that teens blamed the network for increased anxiety.
Instagram's chief Adam Mosseri is due to meet with US Senate politicians this week, and critics say that the timing of this new feature is "interesting," considering the pressure the social network has had over the past few months.
Messari tweeted about the feature last month and now announced the "take-a-break" officially. He said that it will be launched this week in the Canada, United States, the UK, Ireland, Australia, and New Zealand.
"If someone has been scrolling for a certain amount of time, we'll ask them to take a break from Instagram and suggest that they set reminders to take more breaks in the future," Mosseri wrote in the blog.
There will also be notifications to turn on break reminders, and a new tool for parents (coming in March 2022), which Mossari announced, so that parents can see how much time their children are spending on Instagram, and set time limits accordingly.
Time limits already exist on Instagram – if you want to set a time limit per day and receive a notification when you have scrolled too much and the limit is hit, you can go to Settings > Your Activity > Time > Set Daily Limit.
"While available to everyone, I think this tool is particularly important for teens to more fully understand what information they've shared on Instagram, what is visible to others, and to have an easier way to manage their digital footprint," the Instagram continued, finalizing the blog by saying that he welcomes productive collaboration with lawmakers and policymakers in the shared goal of "creating an online world that benefits and protects many generations to come."
Previous articleDave Chapelle announces special New Year's Day show at Rogers Arena
Next articleGetting into the holiday spirit with 2021 Angel submissions


You are Downloading Seekers Notes® Mod v1.33.0 (Mod Money) Latest APK 1.33.0. The Size of Seekers Notes® Mod v1.33.0 (Mod Money) is 62.03MB. Over 54 users rating a average 4.5 of 5 about Seekers Notes® Mod v1.33.0 (Mod Money).
You are Downloading Seekers Notes® Mod v1.33.0 (Mod Money) Latest APK 1.33.0. Last Updated: Apr 26, 2019.
Seekers Notes® Mod v1.33.0 (Mod Money) developed by Seekers Notes® Mod v1.33.0 (Mod Money) is listed under category G 4.5/5 average rating on Google Play by 54 users).
Seekers Notes® Mod v1.33.0 (Mod Money)'s main feature is Seekers Notes® Mod v1.33.0 (Mod Money) provides you unlimited resource. With this mod you will become the best one. And if you are annoyed by ads you can easily disable them..