text:
\section{Introduction} \label{sec:introduction}
Since 2012 when AlexNet won the first place in the ImageNet competition~\cite{krizhevsky2012imagenet}, convolutional neural networks (CNNs)~\cite{lecun1989backpropagation}, have been producing state-of-the-art results in many of the most challenging vision tasks including image classification~\cite{krizhevsky2012imagenet,Lee2015,he2015deep}, face recognition~\cite{schroff2015facenet,sun2015deeply}, semantic segmentation~\cite{long2015fully} and object detection~\cite{ren2015faster,Zagoruyko2016}.
Moreover, CNNs  have become the  workhorse of many learning tasks and real-world applications beyond computer vision, such as natural language understanding and speech recognition~\cite{lecun2015deep}.

Recent studies~\cite{simonyan2014very,srivastava2015training,szegedy2015going,Lee2015} have demonstrated that the depth of neural networks is crucial for their representation ability with limited units~\cite{srivastava2015training}.
However,  in practice, a deeper network often does not necessarily lead to better performance mainly due to gradient vanishing issue~\cite{glorot2010understanding,romero2014fitnets,srivastava2015highway,he2015deep}.
It is shown in \cite{he2015deep} that a plain network with large depth  often {\em under-fits}, instead of over-fitting, the training data, leading to a poor test accuracy.
More critically,  when the number of layers is very large, the network performance  may be severely degraded, due to the training difficulty caused by \emph{gradient vanishing} or \emph{exploding}~\cite{glorot2010understanding,romero2014fitnets,srivastava2015highway,he2015deep}.

Recently proposed deep residual network (ResNet) has largely addressed the gradient vanishing issue by introducing shortcut connections~\cite{he2015deep}. However, when the depth become very large, the overfitting issue may still happen due to what we call \emph{supervision information
vanishing}. This means that intermediate layers fail to improve their discriminative power as much as the increasing depth accordingly, which consequently result in severe internal redundancy in models and degradation in generalization ability~\cite{he2015deep,DBLP:journals/corr/HuangSLSW16}.
Especially, when there are insufficient training data available, the over-fitting is inevitable~\cite{he2015deep}. Nevertheless, how to choose a suitable depth for the model remains an open question. Moreover, the number of model parameters increases dramatically  with depth, leading to a dramatic increase in inference cost. However, in real-world applications, there is a demand for more computationally efficient models, and particularly for models capable of fast inference~\cite{ba2014deep}.

To address the supervision information vanishing issue and obtain compact models accordingly, we introduce auxiliary losses at intermediate layers to provide additional supervision information. In this way, all the auxiliary losses make up a joint loss which acts as a kind of regularization to help training deep networks, like DSN~\cite{Lee2015} and GoogLeNet~\cite{szegedy2015going}. However, treating the multiple losses jointly still has some limitations. First, the importance of losses are required to be adjusted during the training to overcome the training difficulty (See ~\cite{Lee2015}).  Second, the supervision information vanishing will still happen, and the performance often improve marginally  (See more details in Section~\ref{sec:bpcomparision}).

In this paper, we propose to address the above issues by devising a so called AuxNet\xspace, in which multiple weighted auxiliary losses (or outputs) are introduced to intermediate layers.  Different from existing methods such as  DSN~\cite{Lee2015} and GoogLeNet~\cite{szegedy2015going}, we propose a \emph{{Multi-path}} backpropagation (BP) method to train AuxNet\xspace.
Due to the Multi-path BP, the supervision vanishing can be effectively addressed.

The contributions  and novelty of this paper are summarized as follows.
\begin{itemize}
\vspace{-0.5em}
\item We devise a Multi-path BP method to train deep networks with auxiliary losses. Essentially, it applies one forward propagation for all losses but conducts the gradient BP for each loss separately in series. In this way, the supervision information vanishing issue can be effectively addressed, leading to greatly improved generalization performance than Joint BP methods adopted in DSN~\cite{Lee2015} and GoogLeNet~\cite{szegedy2015going}.
\vspace{-0.5em}
\item  By providing sufficient supervision information with Multi-path BP, the proposed AuxNet\xspace reduces the internal model redundancy and gives rise to a more compact model (i.e., with fewer layers but better performance). For example, AuxNet with \textbf{44} layers performs better than the ResNet equivalent with \textbf{110} layers on several benchmark data sets, \ie, CIFAR-10, CIFAR-100 and SVHN.

\vspace{-0.5em}
\item All the losses in AuxNet\xspace remain effective during the training. Therefore, it inherently produces multiple models of different depths simultaneously. Surprisingly, these models have very good prediction power. This is beneficial for model selection. Given training data, models with large depth can be over-fitted. In this sense, the intermediate models may generalize better. In this sense, we propose a simple \emph{Safe Prediction} scheme by taking all the intermediate models to make a safe prediction and thus improve the prediction performance.

\end{itemize}

\section{Related Studies}\label{sec:related_Studies}
Employing auxiliary classifiers to aid in model training has been investigated in DeepID models for face recognition~\cite{sun2015deeply,sun2015deepid3}, GoogLeNet~\cite{szegedy2015going}, and deep supervised nets (DSN)~\cite{Lee2015,wang2015training}. In~\cite{sun2015deeply},  the supervisory signals are connected to each convolution layer; while in~\cite{sun2015deepid3} they are added to each pooling layer.

In GoogLeNet, two auxiliary classifiers (with weights equal to 0.3) are connected to intermediate layers to address the problem of vanishing gradients. In DSN~\cite{Lee2015}, each convolution layer, followed by a \emph{mlpconv} layer, is associated with a classifier. To avoid the over-fitting issue, they keep the losses for a number of epochs, and discard all but the final loss to finish the rest epochs. However, our method does not discard any loss, which helps to generate multiple models at the same time, with the ensuing benefits for model selection. In the training, both of them construct a joint objective function and apply only one backpropagation in each iteration, which is very different form our Multi-path BP. Moreover, these networks cannot be very deep (less than 25 layers). Even with auxiliary classifiers, the issue of supervision information vanishing can still happen for these methods.
In BranchyNet~\cite{teerapittayanon2016branchynet}, auxiliary outputs are regarded as side branches for early exit. These outputs, not for training, but allow test samples to exit from shallow layers to achieve fast inference.

Very recently, some attempts have been made to improve the discriminative power of networks.
He {\em et al.\/}\,  investigated that the pre-activation" function could further improve the ResNet and make the training of 1000-layers model possible. In StochResNet~\cite{DBLP:journals/corr/HuangSLSW16}, a subset of layers are randomly dropped and bypassed in training~\cite{DBLP:journals/corr/HuangSLSW16}. However, all the layers are kept when doing the inference, thus the testing exhibits the same complexity as the original ResNet.
Zhang {\em et al.\/}\, ~\cite{zhang2016residual} propose a new residual-network architecture called ResNet of ResNet, which add shortcut connections between multilevel blocks in ResNet and achieve better performance. In DenseNet~\cite{huang2016densely}, any two layers are directly connected during feed-forward and the feature maps of all preceding layers are used as inputs into all subsequent layers, which is very different from the additivity of feature in ResNet.
Although it keeps approximately the same number of parameters as ResNet, both the training and inference efficiency suffer due to the extremely increasing feature space.  In~\cite{jia2016improving}, Jia studies the effect of orthogonal weight matrices during training and constrains them within a feasible set using a bounding solution, specifically, Singular Value Bounding (SVB) for convolution layers and Bounded Batch Normalization (BBN) for batch normalization layers.

\begin{figure} [t]
\begin{center}
   \includegraphics[width=0.7\columnwidth, height=0.6\columnwidth]{auxresnet.pdf}
\end{center}
   \caption{Architecture of the proposed AuxNet\xspace. $fc$ indicates a fully connected layer, $\{\xi_k\}_{k=1}^{K}$ denotes losses w.r.t. auxiliary outputs, $\xi$ denotes the loss w.r.t. the final output, $N$ denotes the width of network and $\{\gamma_k\}_{k=1}^K$ denotes the weights on the auxiliary losses.}
\label{fig:ausresnet}
\vspace{-1em}
\end{figure}
\section{Proposed Architecture}\label{sec:model_structure}
In this section, we first discuss the issue of supervision information vanishing based on ResNet~\cite{he2015deep}, and then present the details of the proposed AuxNet\xspace.
\subsection{Supervision Information Vanishing}\label{sec:SIV}
Without loss of generality, we study ResNet~\cite{he2015deep} which
conducts \emph{forward inference} by
 \begin{equation}\label{eq:residual}
 {\bf y}_{l} = \lambda_l {{\bf x}_l} + {\mathcal F}({{\bf x}_l},{{\mbox{\boldmath${\mathcal W}$}}_l}), ~~~{\bf x}_{l + 1} = h({\bf y}_{l}),
 \end{equation}
where $h$ is a nonlinear activation function and ${\mathcal F}$ is the transformation function (e.g.,  the convolution operation~\cite{romero2014fitnets})
parameterized by $\mbox{\boldmath${\mathcal W}$}_l$. The model is reduced to a plain network when $\lambda_l=0$. For convenience, we assume $h({\bf x}) = {\bf x}$.
Assuming $h({\bf x}) = {\bf x}$, and applying the recursive rule in Eqn.~(\ref{eq:residual}), we achieve a simplified relation:
${{\bf x}_L} = \lambda_l{{\bf x}_l} + \sum_{i = l}^{L - 1} { {\mathcal F}({\bf x}_i,\mbox{\boldmath${\mathcal W}$}_i) }$~\cite{he2016identity}.

Let $\xi$ be the loss function. For backpropagation, by applying the chain rule,  the gradient of $\xi$ w.r.t. ${\bf x}_l$  for any layer $l$  can be written as
\begin{equation}\label{eq:gradient_resnet}
{{\partial_{{\bf x}_l} \xi }} = {\partial_{{\bf x}_{L}} \xi }  \prod\limits_{i = l}^{L - 1} \Big( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i})}} \Big),
\end{equation}
Here, the term ${\partial_{{\bf x}_{L}} \xi }$ reflects the {\em supervision information} directly obtained from the loss $\xi$; While $\prod_{i = l}^{L - 1} ( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i})}} )$ transforms $\partial_{{\bf x}_{L}} \xi$, and it can be very large at shallow layers with long path backpropagation, due to the additivity property of shortcut connections.

Now, we can measure the ratio of supervision information over the norm of gradient ${{\partial_{{\bf x}_l} \xi }}$   for any layer $l$ by
$$
\rho_l = \frac{||\partial_{{\bf x}_{L}} \xi||}{||\partial_{{\bf x}_{l}} \xi||} \geq \underline{\rho} =  1 \Big/ \left\|\prod_{i = l}^{L - 1}( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i})}} )\right\|,
$$
where $\underline{\rho}$ is a lower bound of $\rho_l$ and $||\cdot||$ is some norm.

Apparently, given a large depth $L$, the lower bound $\underline{\rho}$ vanish rapidly for a small $l$, which means that the layer $l$ cannot receive sufficient information from the loss. Therefore, the ratio $\rho$ may also vanish rapidly for very shallow layers. Then, the \emph{{supervision information vanishing}} happens and consequently the supervision information reaching shallower layers may thus not be sufficient.

\subsection{AuxNet\xspace with Multiple Outputs}
To address the issue of supervision information vanishing, rooted in ResNet, we introduce additional $K$ auxiliary outputs (each associated with a loss function) to some intermediate blocks. The losses are weighted by $\{\gamma_k\}_{k=1}^K$. The general architecture, called AuxNet\xspace, is shown in Figure \ref{fig:ausresnet}. Note that AuxNet\xspace has the same forward propagation rule to ResNet.

In AuxNet\xspace, including the final output, it contains $K+1$ outputs in total, each for the same classification task. For convenience, hereafter we use AuxNet\xspace-$L$-$\widehat{K}$ to denote AuxNet\xspace with $L$ layers and $\widehat{K}$ outputs (where $\widehat{K} = K+1$). Besides ResNet, the above scheme can be applied to many existing CNNs. For example, one may extend the width of network by a factor of $N$ to improve the representation power of each block (See Wide ResNets~\cite{zagoruyko2016wide}).

\textbf{{Setting of $\{\gamma_k\}$}}.
The values of the $\{\gamma_k\}$ are  important to the performance of AuxNet\xspace.  Since features at deeper layers should have more discriminative power, the losses at deeper layers should be more important. Moreover, in practice, if $\gamma_k<0.01$, the effect of $\xi_k$ is negligible, we thus set $\gamma_k \geq 0.01$. In this sense, we suggest choosing $\gamma_k = \max(0.01, (\frac{L_k}{L_K})^\nu)$, where $\nu\geq 0$ reflects the decay rate of $\{\gamma_k\}$ w.r.t. $k$, and $L_k$ is the index of layer to which the $k$-th loss is applied. If setting $\nu = 0$, it follows $\gamma_k = 1, \forall k$, and that every loss is equally weighted.  However, this is too aggressive since shallow layers are not very discriminative, and a large weights may hamper the performance. If setting $\nu > 0$, the weights of losses at shallower layers will be smaller, which can improves overall performance. In practice, we suggest choosing $1\leq \nu \leq 2$ (See supplementary file for more discussions).

The positions of the auxiliary outputs are also important. In general, very shallow layers represent low-level structures which often have weak representation power even for a well trained shallow model~\cite{zeiler2014visualizing}. Therefore, the outputs should not be added to very shallow layers. For example, the first layer represents low-level feature after one convolution layer, thus it has very little representation ability. If adding an output to this layer, the loss and the gradients can be always very large, meaning that the training may not converge (See supplementary materials). Moreover, adding too many losses may cause severe over-fitting, which may hamper the generalization ability of intermediate layers and result in great drop in terms of testing error (see details in Section~\ref{exp:num_outputs}). Therefore, we do not add outputs to each block or even each layer. This is very different from the approach adopted in DSN~\cite{Lee2015}.

\begin{figure*}[t]
\centering

\subfigure[Joint backpropagation]{
\includegraphics[width = 0.32\columnwidth, height= 0.2\columnwidth]{jointbp.pdf}\label{fig:jointbp}
}
\subfigure[Pair-wise backpropagation]{
\includegraphics[width = 0.32\columnwidth, height= 0.2\columnwidth]{pairwisebp.pdf}\label{fig:pairwisebp}
}
\subfigure[Multi-path backpropagation]{
\includegraphics[width = 0.32\columnwidth, height= 0.2\columnwidth]{multiwaybp.pdf}\label{fig:multiwaybp}
}

\caption{Demonstration of three different BP methods for AuxNet\xspace. }\label{fig:BP}
\vspace{-1em}
\end{figure*}

\section{Training Methods for AuxNet\xspace} \label{sec:backpropagation}
In this section, we first show Joint gradient BP, which has been used in DSN~\cite{Lee2015} and GoogLeNet~\cite{szegedy2015going} for AuxNet\xspace, and discuss its limitations. We then propose our Pair-wise and Multi-path BP methods. The overall schemes of the three BP approaches are shown in Figure \ref{fig:BP}.

\subsection{Joint Gradient Backpropagation}
Let ${\xi}_k$ be the loss of the $k$-th output and ${\xi}$ be the loss of the final output. Joint gradient BP considers minimizing the following joint objective function as in~\cite{Lee2015}
\begin{equation} \label{eq:companion_obj}
{\cal{L}} = {\xi} + \sum\limits_{k = 1}^K {{\gamma _k}{\xi}_k}.
\end{equation}
Let ${\bf g}({\bf x}_l)$ be the gradient induced by all losses w.r.t. ${\bf x}_l$.
For any shallow layer $l$ (where $L_{k-1} < l \leq L_k$), the gradient ${\bf g}({\bf x}_l)$ of the joint loss ${\cal{L}}$ w.r.t. ${\bf x}_l$  can be computed by
\begin{equation}\label{eq:joint_gradient}
{\bf g}_L({\bf x}_l) \!\!=\!\!  \frac{{\partial \xi }}{{\partial {{\bf x}_l}}}  +  \sum_{j=k}^K\gamma_j\frac{{\partial \xi_j }}{{\partial {{\bf x}_l}}}, \mathrm{~for~}
k = 1, ..., K,
\end{equation}
where
\begin{equation}\label{eq:sub}
\frac{{\partial \xi_j }}{\partial {{\bf x}_l}} = {\partial_{{\bf x}_{L_j}} \xi_j}  \prod_{i = l}^{L - 1} ( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i})}} . )\end{equation}

From Eqn.~(\ref{eq:sub}), ${\bf g}_L({\bf x}_l)$ is closely related to the depth $L$. In particular, for any shallow layer $l$ (where $L_{k-1} < l \leq L_k$), the supervision information is mainly obtained from the nearest loss (\ie the $k$-th loss).
Similar in Section \ref{sec:SIV}, we can measure the ratio of supervision information  $||{\partial_{{\bf x}_{L_k}} \xi_k}||$ over the whole gradient $||{\bf g}_L({\bf x}_l)||$ by
\begin{equation}\label{eq:joint_ratio}
\rho_l = {||{\partial_{{\bf x}_{L_k}} \xi_k}||}/{||{\bf g}_L({\bf x}_l))||}.
\end{equation}
When $L$ is very large, $||\prod_{i = l}^{L - 1} ( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i})}}||$ can be very large, and $||{\bf g}({\bf x}_l)||$ can be very large with high probability.\footnote{We also empirically demonstrate this in Figure \ref{fig:grad}.}
As a result, the supervision information may still vanish when the network goes deeper.

\subsection{Pair-wise Gradient Backpropagation}
In this method, we sequentially conduct forward propagation and backpropagation for each output (shown in Figure~\ref{fig:pairwisebp}).  For any epoch $t$ in SGD, suppose we deal with the outputs in an order of $k = 1, 2, ..., K+1$, where $K+1$ denotes the index of the final output. For the $k$-th output, we first compute the features $\{{\bf x}^{k}_l\}$ and the loss $\xi_k$ regarding this output based on the model updated by the $(k-1)$-th output, denoted by $\mbox{\boldmath${\mathcal W}$}_t^{k-1}$ for simplicity.  {Here, when $k=1$, we have $\mbox{\boldmath${\mathcal W}$}_t^{0} = \mbox{\boldmath${\mathcal W}$}_{t-1}^{K+1}$.} After that, we compute the gradient ${\bf g}_k({\bf x}_l)$ for the $k$-th output
at any layer $l$ that is lower than or equal to $L_k$ by
\begin{equation}\label{eq:pair-wise_gradient}
{\bf g}_k({\bf x}^{k}_{l}) = \gamma_k \frac{\partial \xi_k }{ \partial {\bf x}^{k}_{L_k} }  \cdot \frac{\partial {\bf x}^{k}_{L_k} }{ \partial {\bf x}^{k}_{L_k-1} }  ...\frac{\partial  {\bf x}^{k}_{l+1}  }{\partial {\bf x}^{k}_{l} }, ~~\mathrm{for}~~ l\leq L_k,
\end{equation}
or it can be simplified as
\begin{equation}\label{eq:pair-wise-grad}
{\bf g}_k({\bf x}^{k}_{l}) = \gamma_k {\partial_{{\bf x}_{L_k}^k} \xi_k }  \prod\limits_{i = l}^{L_k - 1} \Big( {1 + \partial_{{\bf x}_i^k}  {{\mathcal F}({{\bf x}_i^k},{{\mbox{\boldmath${\mathcal W}$}}_i^k})}} \Big),
\end{equation}
Similar to Eqn. \ref{eq:joint_ratio}, we can measure the ratio of supervision information over $||{\bf g}_k({\bf x}^{k}_{l})||$ for any layer $l$ ($l\leq L_k$) by
\begin{small}
$$\rho_l = \frac{||\gamma_k{\partial_{{\bf x}_{L_k}^k} \xi_k }||}{||{\bf g}_k({\bf x}^{k}_{l})||} \geq \underline{\rho} =  1 \Big/ \left\|\prod\limits_{i = l}^{L_k - 1} \Big( {1 + \partial_{{\bf x}_i^k}  {{\mathcal F}({{\bf x}_i^k},{{\mbox{\boldmath${\mathcal W}$}}_i^k})}} \Big)\right\|.$$\end{small}
 Obviously, for Pair-wise gradient BP, in contrast to ResNet and Joint gradient BP, the ratio $\rho_l$ is determined by $L_k$ rather than $L$. As a result, for any layer $l$, sufficient supervision information can be obtained from the nearest output (\ie, $\xi_k$).

In Pair-wise gradient BP, for each loss,
{the features and loss are updated before each forward-backpropagation pair, which may incur two \emph{limitations}.}
\textbf{First}, after each backpropagation, the loss for the next output shall decrease with very high probability, which will decrease the magnitude of gradient. This will incur degraded performance. \textbf{Second}, forward propagation for each output takes additional cost. To amend these, we propose the following BP method, called \emph{Multi-path gradient} backpropagation.

\subsection{Multi-path Gradient Backpropagation}
We apply one forward propagation for updating all losses and the features for all layers. Then, keeping the features and losses unchanged, we conduct gradient backpropagation to update $\mbox{\boldmath${\mathcal W}$}^{k}$ for each loss separately  instead of summing them up, as shown in Figure~\ref{fig:multiwaybp}.
Let $\{{\bf x}_l\}$ be the features of layer $l$ which are fixed for all outputs. In this case, for the $k$-th output, the gradient ${{\partial \xi_k }}/{{\partial {{\bf x}_l}}}$ for any layer $l$ that is lower or equal to $L_k$ shall be computed by
\begin{eqnarray} \label{eq:multipath_individual_gradient}
{\bf g}_k({\bf x}_{l}) = \gamma_k \frac{\partial \xi_k }{ \partial {\bf x}_{L} }  \cdot \frac{\partial {\bf x}_{L} }{ \partial {\bf x}_{L-1} }  ...\frac{\partial  {\bf x}_{l+1}  }{\partial {\bf x}_{l} }, ~~\mathrm{for}~~ l\leq L_k
\end{eqnarray}
The difference of (\ref{eq:multipath_individual_gradient}) from (\ref{eq:pair-wise_gradient}) is that the features are kept unchanged for all the backpropagations.

The ratio of supervision information over $||{\bf g}_k({\bf x}_{l})||$ for any layer $l$ ($l\leq L_k$) can be computed by
\begin{small}{$$\rho_l = \frac{||\gamma_k{\partial_{{\bf x}_{L_k}} \xi_k }||}{||{\bf g}_k({\bf x}_{l})||} \geq \underline{\rho} =  1 \Big/ \left\|\prod\limits_{i = l}^{L_k - 1} \Big( {1 + \partial_{{\bf x}_i}  {{\mathcal F}({{\bf x}_i},{{\mbox{\boldmath${\mathcal W}$}}_i^k})}} \Big)\right\|.$$}\end{small}
Similar to Pair-wise BP, the lower bound $\underline{\rho_l}$ for the ration is determined by $L_k$ rather than $L$. As a result, for any layer $l$, sufficient supervision information for intermediate layers can be obtained from the nearest output (\ie, $\xi_k$). Therefore, the issue of supervision information vanishing is addressed.

Lastly, in terms of training cost, AuxNet with Multi-path BP is about ($K/2+1$) times of ResNet, since it conducts one forward propagation and ($K+1$) backpropagations at each iteration. Testing time for AuxNet is the same as ResNet when they have the same depth. However, AuxNet often produces
more compact network than ResNet, thus the testing for AuxNet can be much faster.

\begin{figure}[t]
\centering

\subfigure[]{
\includegraphics[trim = 2mm 0mm 12mm 1mm,
clip, width = 0.43\columnwidth, height= 0.36\columnwidth]{supervision_information_ratio.pdf}\label{fig:ratio}
}
\subfigure[]{
\includegraphics[trim = 6mm 0mm 8mm 1mm,
clip, width = 0.43\columnwidth, height= 0.36\columnwidth]{cifar10_bp_test_new.pdf}\label{fig:ratio_testing_error}
}

\caption{Performance comparison of AuxNet\xspace-56-5 on CIFAR-10 with different BP methods. (a) Supervision information changes w.r.t. layers; (b) Evolution of testing error.}\label{fig:grad}
\vspace{-0.8em}
\end{figure}

\subsection{Comparisons of Backpropagation Methods }\label{sec:bpcomparision}
We empirically study the supervision information vanishing issue and compare different BP methods for AuxNet\xspace with 56 layers and 4 auxiliary outputs (at layers 15, 25, 35, 45) on CIFAR-10. ResNet with 56 layers is adopted as the baseline.

In Figure \ref{fig:ratio}, we compute and record the supervision information ration $\rho_l$ (not the lower bound $\underline{\rho}$) at the
final training epoch of SGD for ResNet, Joint BP, Pair-wise BP and Multi-path BP, respectively. In Figure \ref{fig:ratio_testing_error}, we monitor the testing error evolution of different methods.

From Figure \ref{fig:ratio}, $\rho_l$ for ResNet indeed decreases rapidly as $l$ reduces (reaching shallow layers). This makes intermediate layers difficult to train, and leads to model redundancy. For AuxNet\xspace with Joint BP, it shows improvement over ResNet, but the ratio still diminishes very fast when reaching shallower layers. Conversely, for AuxNet\xspace with  Pair-wise BP and Multi-path BP, the supervision ratio remains reasonably large even to the first layer. From Figure \ref{fig:ratio_testing_error}, AuxNet\xspace with Joint BP  shows much better performance than ResNet; while Multi-path BP shows much better performance than Joint BP. For Pair-wise BP, since it updates the loss for each output, the performance is severely degraded.

\begin{figure*}[t]
\centering

\subfigure[Input images]{
\includegraphics[trim = 0mm 0mm 0mm 0mm,
clip, width = 0.19\columnwidth, height= 0.31\columnwidth]{predplot1.pdf}\label{}
}
\hspace{-5ex}
\subfigure[output-67]{
\includegraphics[trim = 0mm 0mm 0mm 0mm,
clip, width = 0.2\columnwidth, height= 0.31\columnwidth]{predplot2.pdf}\label{}
}
\hspace{-3ex}
\subfigure[output-79]{
\includegraphics[trim = 0mm 0mm 0mm 0mm,
clip, width = 0.2\columnwidth, height= 0.31\columnwidth]{predplot3.pdf}\label{}
}
\hspace{-3ex}
\subfigure[output-91]{
\includegraphics[trim = 0mm 0mm 0mm 0mm,
clip, width = 0.2\columnwidth, height= 0.31\columnwidth]{predplot4.pdf}\label{}
}
\hspace{-3ex}
\subfigure[output-101]{
\includegraphics[trim = 0mm 0mm 0mm 0mm,
clip, width = 0.2\columnwidth, height= 0.31\columnwidth]{predplot5.pdf}\label{}
}

\caption{Prediction results AuxNet with 101 layers and 4 outputs on ImageNet validation data.  \textbf{Column (a)}: testing images with ground truth label; \textbf{Column (b)}: The top 5 prediction probabilities of the output at the $67$th layer; \textbf{Column (c)}: The top 5 prediction probabilities of the output at the $79$th layer; \textbf{Column (d)}: The top 5 prediction probabilities of the output at the $91$th layer; \textbf{Column (e)}: The top 5 prediction probabilities of the output at the $101$th layer. The red bar denotes the prediction probability of the ground truth class.}\label{fig:safeprediction}
\label{fig:testcase}
\end{figure*}

\section{Safe Prediction with Multiple  Models}\label{sec:safe}
One advantage of AuxNet is that, it inherently produces multiple models of different depths simultaneously.
In general, the discriminative power of deeper models are superior to the shallower ones.  This may not be necessarily true in practice. In Figure~\ref{fig:testcase}, for the first two images, deeper models shows better discriminative power, but for the third image, the final model make a fake prediction while its shallower counterparts have correctly predicted the ground truth label. Moreover, given limited data, the CNNs may easily over-fit if the depth is too large. In this case, the shallow models may generalize much better. Nevertheless, how to choose an appropriate model is still an open question in deep learning.

An natural question is, considering the shallow models may already correctly predicted the ground truth label, is there any way to combine all the intermediate models to attain a safe prediction? If so, the difficulty of model selection can be largely avoided.

Here, we address by proposing a method to select the prediction with the highest confidence among all the outputs.
Let $\hat{{\bf y}}$ denote the obtained softmax probability from a specific output.  To evaluate the quality of prediction for the sample ${\bf x}$, we use \textbf{entropy} as the measure:
\begin{equation}
Entropy({\hat {\bf y}}; {\bf x}) =  - \left\langle {\hat {\bf y},\log \hat {\bf y}} \right\rangle ,
\end{equation}
where $\langle \cdot, \cdot\rangle$ denotes the inner product of two vectors. The smaller the entropy is, the more confident the prediction will be, referring to the prediction with the smallest entropy in each test case in Figure~\ref{fig:testcase}. Note that in Figure~\ref{fig:testcase}, the correct decision for an example is made by the model with the least entropy. Motivated by this, we first make prediction with all the models and then select the one with the least entropy as the final decision:
\begin{equation}
\hat {\bf y} = \mathop {\arg \min }\limits_{{\hat {\bf y}_i}} ~~Entropy({\hat {\bf y}_i};{{\bf x}_i}), ~~~ i=1, \cdots, K+1.
\end{equation}
In this way, although deep models may over-fit, by taking advantage of all the outputs, we can select the output with the highest confidence for the final decision, which is at least as good as the prediction with the final model. We thus make a safer prediction.

\section{Experiments}\label{sec:exp}
We implement AuxNet\xspace based on Torch.\footnote{Torch is from \url{http://torch.ch}. We will release the code and models of AuxNet\xspace soon.}
Several state-of-the-art deep learning models are adopted as baselines, including FitNet~\cite{romero2014fitnets},
DSN~\cite{Lee2015}, Frac.Pool~\cite{graham2014fractional}, Highway Network~\cite{srivastava2015highway}, StochResNet~\cite{DBLP:journals/corr/HuangSLSW16}, ResNet~\cite{he2016identity}, Wide Residual Network(Wide ResNet)~\cite{zagoruyko2016wide}, ResNet of ResNet~\cite{zhang2016residual}, DenseNet~\cite{huang2016densely} and SVB-Wide ResNet~\cite{jia2016improving}.

We conduct comparisons on four benchmark data sets: CIFAR-10~\cite{krizhevsky2009learning}, CIFAR-100~\cite{krizhevsky2009learning}, the Street View House Number (SVHN)~\cite{netzer2011reading} and ImageNet-2012~\cite{russakovsky2015imagenet}.
For CIFAR-10, it contains 10 classes of 32x32 natural color images, each with 5,000 training samples
and 1,000 testing samples. CIFAR-100 contains 100 classes. Each class has 500 training samples and 100 testing samples.
SVHN contains house number images of 32x32 pixels, which includes 73237 digits for training, 26032 digits for testing, and 531131 digits as extra training data.
ImageNet contains 1,000 classes with 1.28 million training images and 50k testing images.

We use the same experimental setting for AuxNet\xspace as that for ResNet~\cite{he2015deep}. We train AuxNet using the proposed Multi-path BP with SGD using a mini-batch size of 128 on 2 GPUs (64 each). The momentum for SGD is 0.9, and the learning rate starts from 0.1 and is divided by 10 at 40\% and 60\% of total epochs respectively.  The model parameters are initialized as in~\cite{he2015delving}. For AuxNet, each intermediate output can be used for prediction. However, for convenience, we first report the results using \textbf{last output}. We will study the prediction ability of intermediate models and safe prediction in Section~\ref{exp:inter_models} and Section~\ref{exp:ensemble}, respectively.

\subsection{Results on CIFAR-10 and CIFAR-100}\label{exp:main_results}
\noindent \emph{\textbf{Comparisons with ResNet}}.
We perform 400 SGD epochs for the training, and use the same data augmentation as in~\cite{Lee2015}, such as translation and horizontal flipping.
We first compare  the testing error evolution of 3 ResNets with (44, 56, 110)-layers  and 4 AuxNets of different settings. From Figure \ref{fig:cifar10test} and Table~\ref{tab:cifar-resnet}, we have the following observations.
\textbf{First}, AuxNet\xspace-44-2, AuxNet\xspace-44-3, AuxNet\xspace-56-2, and AuxNet\xspace-56-5 all consistently outperform their ResNet counterparts with the same depth.
\textbf{Second}, with much fewer layers, AuxNet\xspace-56-2 and AuxNet\xspace-56-5 yield better results than the deeper ResNet with 110 layers.
\textbf{Third}, AuxNet\xspace-44-3 and AuxNet\xspace-56-5 with more auxiliary outputs achieve larger improvement in terms of testing error than AuxNet\xspace-44-2 and AuxNet\xspace-56-2, respectively. All these observations strongly demonstrate the effectiveness of AuxNet\xspace trained by Multi-path BP.

\begin{figure}[t]
\centering
\subfigure[Testing error]{
\includegraphics[width = 0.47\columnwidth, height= 0.36\columnwidth]{best_result_cifar10_test.pdf}\label{fig:cifar10test}
}
\subfigure[Training error]{
\includegraphics[width = 0.47\columnwidth, height= 0.36\columnwidth]{best_result_cifar10_train.pdf}\label{fig:cifar10train}
}

\caption{Testing error and training error evolution of ResNet and different AuxNet\xspace on CIFAR-10. Taking AuxNet\xspace-44-2 for example, it denotes a AuxNet\xspace network with 44 layers and 1 auxiliary output (i.e., 2 outputs in total). The numbers in brackets indicate the indices of layers with added outputs.}
\label{fig:cifar}
\end{figure}

\begin{table}[t]
  \centering
  \caption{Comparisions with ResNet on CIFAR-10 and CIFAR-100.}
    \begin{tabular}{c|c|c|c|c}
    \hline
    \multirow{2}[0]{*}{network} & \multirow{2}[0]{*}{depth} & \multirow{2}[0]{*}{outputs position} & \multicolumn{2}{c}{error (\%)} \\
    \cline{4-5}
        &   &   & CIFAR-10 & CIFAR-100 \\
    \hline
    ResNet-20~\cite{he2016identity} & 20    & \{20\}  & 7.76 & 31.12 \\
    ResNet-32~\cite{he2016identity} & 32    & \{32\}  & 6.81 & 29.74 \\
    ResNet-44~\cite{he2016identity} & 44    & \{44\}  & 6.37 & 28.85 \\
    ResNet-56~\cite{he2016identity} & 56    & \{56\}  & 6.08 & 28.46 \\
    ResNet-110~\cite{he2016identity} & 110   & \{110\} & 5.86 & 27.41 \\
    \hline
    AuxNet\xspace-20-2 & 20    & \{17, 20\} & 7.72 & 30.44 \\
    AuxNet\xspace-32-2 & 32    & \{25, 32\} & 6.73 & 28.24 \\
    AuxNet\xspace-44-2 & 44    & \{35, 44\} & 6.12 & 27.46 \\
    AuxNet\xspace-44-3 & 44    & \{25, 35, 44\} & 5.85 & 27.19 \\
    AuxNet\xspace-56-2 & 56    & \{45, 56\} & 5.77 & 26.83 \\
    AuxNet\xspace-56-5 & 56    & \{15, 25, 35, 45, 56\} & 5.53 & 26.62 \\
    AuxNet\xspace-110-5 & 110   & \{58, 73, 85, 97, 110\}   &   5.41    &  26.48 \\
    \hline
    \end{tabular}
  \label{tab:cifar-resnet}%
\end{table}

\noindent \emph{\textbf{Comparisons with the state-of-the-arts}}.
Now we report a comprehensive comparison of AuxNet\xspace with other baselines on both CIFAR-10 and CIFAR-100 data sets in terms of testing error.
Note that as shown in Figure~\ref{fig:ausresnet}, we can increase the network width by a factor of $N$ to improve the representation ability of each layer.
From Table \ref{tab:cifar}, we have the following new observations. \textbf{First}, our AuxNet\xspace with auxiliary outputs achieves better or comparable performance with ResNet and other baselines. Specifically, AuxNet\xspace-56-5 yields 5.53\% and 26.62\% error on CIFAR-10 and CIFAR-100, respectively, which are much better than its ResNet-56 counterpart and ResNet-110 with far more layers. This demonstrates that AuxNet\xspace indeed is capable of reducing the model redundancy.
\textbf{Second}, by increasing the layer width, AuxNet\xspace succeeds to greatly improve the performance, i.e., AuxNet\xspace-28-2 (10x wider) yields 3.77\% and 19.69\% on CIFAR-10 and CIFAR-100 respectively.
Because of the success of DenseNet~\cite{huang2016densely} and Singular Value Bounding (SVB)~\cite{jia2016improving}, we further apply the Multi-path BP to train AuxNet\xspace with dense layers and SVB. \textbf{Finally}, we further improve the performance and AuxNet\xspace-SVB-28-2/10 achieve the best performance simultaneously on CIFAR-10 (3.32\%) and CIFAR-100 (17.87\%).
These observations demonstrate that, by leveraging the power of auxiliary outputs, AuxNet\xspace trained by Multi-path BP improves the performance of ResNet by a large margin.

\begin{table}[h]
  \centering
  \caption{Classification error on CIFAR-10 and CIFAR-100. Note that \textcolor{black}{\{$\cdot$/$N$\}} indicates a N-fold increase in network width, e.g. AuxNet\xspace-26-2/10 is 10 times wider than the baseline.}
    \begin{tabular}{c|c|c|c|c}
    \hline
    \multirow{2}[0]{*}{network} & \multirow{2}[0]{*}{depth} & \multirow{2}[0]{*}{outputs position} & \multicolumn{2}{c}{error (\%)} \\
    \cline{4-5}
        &   &   & CIFAR-10 & CIFAR-100 \\
    \hline
    FitNet~\cite{romero2014fitnets} & 19 & \{19\}  & 8.39 & 35.04 \\
    DSN~\cite{Lee2015} & 11 & \{1, 5, 9, 11\}  & 7.97 & 34.57 \\
    Frac.Pool, 1 test~\cite{graham2014fractional} & 15 & \{15\}  & 4.50 & 31.20 \\
    Highway Network~\cite{srivastava2015highway} & 19 & \{19\}  & 7.60 & 32.24 \\
    StochResNet-110~\cite{DBLP:journals/corr/HuangSLSW16} & 110 & \{110\} & 5.23 & 24.58 \\
    DenseNet~\cite{huang2016densely} & 100 & \{100\} & 3.74 & 19.25 \\
    ResNet of ResNet/4~\cite{zhang2016residual} & 58 & \{58\} & 3.77 & 19.73 \\
    Wide ResNet/10~\cite{zagoruyko2016wide} & 28 & \{28\} & 4.17 & 20.50 \\
    SVB-Wide ResNet/10~\cite{jia2016improving} & 28 & \{28\} & 3.58 & 18.32 \\
    ResNet-110~\cite{he2016identity} & 110   & \{110\} & 5.86 & 27.41 \\
    ResNet-1001~\cite{he2016identity} & 1001   & \{1001\} & 4.62 & 22.71 \\
    \hline
    AuxNet\xspace-28-2/10 & 28   & \{21, 28\} & 3.77 & 19.69 \\
    AuxNet\xspace-Dense-100-4 & 100  & \{72, 81, 90, 100\} & 3.53 & 19.13 \\
    AuxNet\xspace-SVB-28-2/10    & 28  & \{21, 28\} & \textbf{3.32} & \textbf{17.87} \\
    \hline
    \end{tabular}
  \label{tab:cifar}%
\end{table}

\subsection{Results on SVHN}
We divide the images by 255 for scaling without data augmentation and perform training for 200 epochs. From Table \ref{tab:svhn}, AuxNet\xspace-56-2 with 56 layers significantly improves the performance of ResNet-110, which is a deeper structure (yielding 1.84\% in error rate). When using dropout, AuxNet\xspace-56-3 obtains 1.58\% error rate, which is the best published result on SVHN to our knowledge. Dropout here works as a regularizer to reduce overfitting, and improves the performance of both AuxNet\xspace and ResNet. Nevertheless, AuxNet\xspace performs consistently better than ResNet with and without dropout.

\begin{table}[h]
  \centering
  \caption{Testing error on SVHN. $^\dagger$ denotes results with dropout.}
    \begin{tabular}{c|c|c}
    \hline
    network & outputs position & error (\%) \\
    \hline
    FitNet~\cite{romero2014fitnets}      & \{19\}  & 2.42 \\
    DSN~\cite{Lee2015}      & \{1, 5, 9, 11\} & 1.92 \\
    StochResNet~\cite{DBLP:journals/corr/HuangSLSW16}      & \{152\} & 1.75 \\

    ResNet-44~\cite{he2016identity}   & \{44\}    & 2.15 \\
    ResNet-56~\cite{he2016identity}   & \{56\}    & 2.06 \\
    ResNet-110~\cite{he2016identity}  & \{110\}  & 1.97 \\

    ResNet-44$^\dagger$~\cite{he2016identity}   & \{44\}    & 1.91 \\
    ResNet-56$^\dagger$~\cite{he2016identity}   & \{56\}    & 1.70 \\
    ResNet-110$^\dagger$~\cite{he2016identity}   & \{110\}    & 1.65 \\
    \hline
    AuxNet\xspace-44-2 & \{35, 44\}    & 1.96 \\
    AuxNet\xspace-56-2 & \{56, 45\}    & 1.84 \\
    AuxNet\xspace-44-2$^\dagger$ & \{35, 44\}    & 1.75 \\
    AuxNet\xspace-56-2$^\dagger$ & \{45, 56\}    & 1.63 \\
    AuxNet\xspace-56-3$^\dagger$ & \{35, 45, 56\}    & \textbf{1.58} \\
    \hline
    \end{tabular}
  \label{tab:svhn}%
\end{table}%

\subsection{Results on ImageNet}
The experimental setting is the same as that given in~\cite{he2015deep}. We scale the learning rate by a factor of 0.5 to train AuxNet\xspace with 90 epochs.
Table~\ref{tab:imagenet1000} shows the results of top-1 and top-5 errors on ImageNet-2012 validation set.

\begin{table}[htbp]
  \centering
  \caption{Validation error on ImageNet-2012 (\textbf{10-crop} testing).}
    \begin{tabular}{c|c|c|c}
    \hline
    \multirow{2}[0]{*}{network} & \multirow{2}{*}{outputs position} & \multicolumn{2}{c}{error (\%)} \\
    \cline{3-4}
        &   & top-1 & top-5 \\
    \hline
    VGG-16~\cite{simonyan2014very} & \{16\}  & 28.07 & 9.33 \\
    GoogLeNet~\cite{szegedy2015going} & \{22\}  & - & 9.15 \\
    PReLU-net~\cite{he2015delving} & \{22\}  & 24.27 & 7.38 \\
    \hline
    ResNet-34~\cite{he2015deep} & \{34\}  & 24.76 & 7.35 \\
    ResNet-50~\cite{he2015deep} & \{50\}  & 22.85 & 6.71 \\
    ResNet-101~\cite{he2015deep} & \{101\}  & 21.75 & 6.05 \\
    Inception-ResNet~\cite{DBLP:journals/corr/SzegedyIV16} & \{136\}  & 18.77 & 4.13 \\ \hline \hline
    AuxNet\xspace-34-2 & \{29, 34\} & 24.61 & 7.18 \\
    AuxNet\xspace-50-2 & \{40, 50\} & 22.77 & 6.27 \\
    AuxNet\xspace-50-4 & \{22, 31, 40, 50\} & 22.34 & 6.07 \\
    AuxNet\xspace-101-4 & \{67, 79, 91, 101\} & 20.95 & 5.25 \\
    Inception-AuxNet\xspace-4 & \{98, 110, 122, 136\} & \textbf{18.69} & \textbf{4.05} \\
    \hline
    \end{tabular}
  \label{tab:imagenet1000}%
\end{table}

Table~\ref{tab:imagenet1000} shows that AuxNet\xspace achieves better performance than ResNet.
For example, AuxNet\xspace-34-2 and AuxNet\xspace-50-2 has outperformed ResNet with the same number of layers. When adding more auxiliary outputs, the AuxNet\xspace-50-4 with 4 outputs in total further improve the performance to 22.43\% and 6.07\% on top-1 error and top-5 error respectively.
For Inception-AuxNet\xspace with much more parameters, to save training time, we employed the trained Inception-ResNet model as initialization when performing Multi-path BP to train with auxiliary outputs. We use the same setting as those for ResNets except for the learning rate that starts from 0.0045 (0.1 times of that in~\cite{DBLP:journals/corr/SzegedyIV16}). As a result, we achieve the best performance of 18.69\% and 4.05\% on top-1 error and top-5 error respectively.

\section{More Analysis on AuxNet\xspace}\label{sec:analysis}

\subsection{Prediction Ability of Intermediate Models}\label{exp:inter_models}
The AuxNet\xspace training process inherently generates multiple models of differing depths.
In Table~\ref{tab:intermediate}, we compare the prediction performance of intermediate models generated by each loss of AuxNet\xspace-56-5
against that of the corresponding layer of ResNet-56. We observe that each intermediate model of AuxNet\xspace-56-5 outperforms its ResNet comparator and even often outperform their full-depth ResNet counterparts.
By comparing these results with Table \ref{tab:cifar}, we see that
the AuxNet\xspace-56-5 intermediate model obtained at \textbf{output-45} shows much better performance ($5.67\%$ error)
than ResNet-44 (6.37\%), ResNet-56 (6.08\%), ResNet-110 (5.86\%), as well as AuxNet\xspace-44-3 (5.85\%).
This demonstrates not only that AuxNet\xspace improves the discriminative power of intermediate layers, but also that it provides the opportunity for a form of model selection, and a means of avoiding over-fitting.

In Table \ref{tab:intermediate}, the third column records the number of model parameters. Fewer model parameters imply faster inference, which is important for real-world applications. For example, the AuxNet\xspace-56-5 model at \textbf{output-45} has 0.48M parameters, which is nearly half the number in ResNet-56 (0.85M), and 1/3 of that in ResNet-110 (1.7M).

\begin{table}[htbp]
  \centering
  \caption{Testing error of intermediate models of AuxNet\xspace-56-5 and ResNet-56 on CIFAR-10.}
    \begin{tabular}{c|c|c|c|c}
    \hline
    \multirow{2}[0]{*}{model} & \multirow{2}[0]{*}{\#layers} & \multirow{2}[0]{*}{\#params} & \multicolumn{2}{c}{error (\%)} \\
    \cline{4-5}
          &       &       & AuxNet\xspace & ResNet \\
    \hline
    model-15 & 15    & 0.03M & 50.35 & 63.01 \\
    model-25 & 25    & 0.09M & 18.94 & 45.07 \\
    model-35 & 35    & 0.18M & 9.23  & 34.01 \\
    model-45 & 45    & 0.48M & \textbf{5.67}  & 13.71 \\
    model-56 & 56    & 0.85M & \textbf{5.53}  & 6.56 \\
    \hline
    \end{tabular}
  \label{tab:intermediate}
\end{table}

\subsection{Demonstration of Safe Prediction}\label{exp:ensemble}

With safe prediction, we always select the output with the highest confidence as the final decision to correct potential wrong predictions.
Compared to the original prediction method in which the last output makes predictions for all the samples,
each output in safe prediction is responsible for making predictions for a subset of samples with high confidences.
Based on the advantage of the discriminative ability of each output on specific data subset, we achieve slightly better results on the benchmark data sets, as shown in Table~\ref{tab:ensemble}.
Moreover, we see that a large part of samples can be predicted at shallow outputs with high quality, e.g., 63\% (the first two outputs) on SVHN and 39\% (the first three outputs) on ImageNet.
This observation indicates that it is not necessary to go through the whole network to perform inference, especially for those samples that can be predicted with high confidences at shallow outputs.

\begin{table}[htbp]
  \centering
  \caption{Comparison of safe prediction and the original prediction method of AuxNet\xspace. Here the original method denotes the prediction using the last output.}
    \begin{tabular}{c|c|c|c|c}
    \hline
    \multirow{2}[0]{*}{model} & \multirow{2}[0]{*}{outputs} & \multicolumn{1}{c|}{\multirow{2}[0]{*}{method}} & \multicolumn{1}{c|}{\multirow{2}[0]{*}{prediction ratio (\%)}} & \multicolumn{1}{c}{\multirow{2}[0]{*}{error (\%)}} \\
          &       &       &       &  \\
    \hline
    AuxNet\xspace-SVB-28-2/10 & \multirow{2}[0]{*}{\{21, 28\}} & original & \{0, 100\}     & 3.34 \\
    on CIFAR-10      &       & safe & \{32, 68\}     & 3.31 \\
    \hline
    AuxNet\xspace-SVB-28-2/10 & \multirow{2}[0]{*}{\{21, 28\}} & original & \{0, 100, 0\}     & 17.87 \\
    on CIFAR-100      &       & safe & \{13, 87\}     & 17.81 \\
    \hline
    AuxNet\xspace-56-3 & \multirow{2}[0]{*}{\{35, 45, 56\}} & original & \{0, 0, 100\}     & 1.58 \\
    on SVHN      &       & safe & \{12, 51, 34\}     & 1.58 \\
    \hline
    AuxNet\xspace-Inception-4 & \multirow{2}[0]{*}{\{98, 110, 122, 136\}} & original & \{0, 0, 0, 100\}     & 18.69 \\
    on ImageNet      &       & safe & \{5, 13, 21, 61\}     & 18.66 \\
    \hline
    \end{tabular}%
  \label{tab:ensemble}%
\end{table}%

\subsection{Combatting Overfitting Issue}

For ResNet, when the depth goes deeper, the model has only limited performance gain (\eg ResNet-110 has almost two times the number of parameters than ResNet-56 but only yields 0.2\% accuracy gain, as shown in Figure~\ref{tab:cifar}).  In Figure~\ref{fig:cifar10train}, compared to the training of ResNet, AuxNet\xspace with Multi-path BP reaches slightly higher training error at convergence, but produces lower testing error. This phenomenon is observed on models with any number of layers, which indicates that AuxNet\xspace is less prone to overfitting. To some extents, the performance gain of AuxNet\xspace can be the side-effect of combatting overfitting issue.

\subsection{Effects on Numbers of Outputs}\label{exp:num_outputs}
In this experiment, we test the performance of AuxNet\xspace for various numbers of intermediate outputs on the CIFAR-10 data set.
The results are given in Table \ref{tab:num_outputs}. We see that
AuxNet\xspace-56-2 and AuxNet\xspace-56-5 perform significantly better than ResNet-56.
However, adding too many outputs does not necessarily improve the performance. For example,  AuxNet\xspace-56-10 with 10 outputs performs much worse than  AuxNet\xspace-56-2, AuxNet\xspace-56-5 and ResNet-56.

\begin{table}[h]
  \centering
  \caption{Testing error for varying number of outputs on CIFAR-10.}
    \begin{tabular}{c|c|c}
    \hline
    network & outputs position & error (\%) \\
    \hline
    ResNet-56~\cite{he2016identity}     & \{56\}  & 6.08 \\
    AuxNet\xspace-56-2      & \{45, 56 \} & 5.77 \\
    AuxNet\xspace-56-5      & \{15, 25, 35, 45, 56\} & \textbf{5.53} \\
    \hline
    \multirow{2}{*}{AuxNet\xspace-56-10}
    & \{13, 17, 21, 25, 29,&
    \multirow{2}{*}{7.96}\\
    & 33, 37, 41, 45, 56\} & \\

    \hline
    \end{tabular}
  \label{tab:num_outputs}%
\end{table}%

\subsection{Outputs at Very Shallow Layers}\label{exp:shallowoutput}
While auxiliary outputs can help a simpler model to achieve the same accuracy as a deeper ResNet model, adding auxiliary outputs to very shallow layers may severely hampers the prediction performance since the features of shallow layers may not be sufficiently discriminative. To demonstrate this, we investigate two AuxNet\xspace models by adding single output to very shallow layers, and record the results.
From Table \ref{tab:shallow_layers}, adding outputs at very shallow layer, e.g. 5 out of ResNet-20 and ResNet-44, severely influences the prediction performance. That means, in practice, it is necessary to avoid adding auxiliary outputs at very shallow layers.

\begin{table} [t]
  \centering
  \caption{Effects of auxiliary outputs at very shallow layers on CIFAR-10 test set.}
    \begin{tabular}{c|c|c}
    \hline
    network & outputs position & error (\%) \\
    \hline
    ResNet-20~\cite{he2016identity} & \{20\} & 7.76 \\
    AuxNet\xspace-20-2 & \{5, 20\} & 9.64 \\
    AuxNet\xspace-20-2 & \{17, 20\} & 7.58 \\
    \hline
    ResNet-44~[6] & \{44\} & 6.37 \\
    AuxNet\xspace-44-2 & \{5, 44\} & 7.73 \\
    AuxNet\xspace-44-2 & \{15, 44\} & 6.75 \\
    AuxNet\xspace-44-2 & \{35, 44\} & 6.12 \\
    \hline
    \end{tabular}
  \label{tab:shallow_layers}%
\end{table}%

\subsection{Effect of Weighting Scalar $\nu$}\label{exp:nu}
We conducted experiments to investigate how to weight outputs to achieve better performance. Under the setting of $\{\gamma_k\}$ described in Section 3.1, we compare different value of $\nu$ to adjust weights. From Table \ref{tab:weight}, the equally weighted outputs ($\nu=0$) severely hampers the performance. If $\nu \geq 1$, the weights of shallower outputs become smaller due to their decreasing discriminative powers. Especially for $\nu=1$ or $\nu=2$, the $n$-layer AuxNet\xspace behaves better than the original $n$-layer ResNet. For a larger $\nu=5$, the weights decay so aggressively that the effects of auxiliary outputs are negligible.

\begin{table} [h]
  \centering
  \caption{Effect of weighting scalar $\nu$ on CIFAR-10 test set.}
    \begin{tabular}{c|c|c}
    \hline
    network & $\nu$ & error (\%) \\
    \hline
    ResNet-56~\cite{he2016identity} & - & 6.08 \\
    \hline
    \multirow{4}{*}{AuxNet\xspace-56-5} & 0 & 8.43 \\
    & 1 & 5.90 \\
    & 2 & 5.53 \\
    & 5 & 6.03 \\
    \hline
    \end{tabular}%
  \label{tab:weight}%
\end{table}

\subsection{Effects of the Parameter $\lambda$}\label{exp:lambda}
The choice of the parameter $\lambda$ in the shortcut mapping
is critical to the performance of residual networks. When $\lambda = 0$, ResNet is reduced to the plain network. We study the influence of $\lambda$ on shortcuts within a residual unit of ResNet and AuxNet\xspace. The values of $\lambda$ considered are \{0.5, 0.95, 1\}.  From Table \ref{tab:lambda}, the AuxNet\xspace consistently outperforms ResNet with different $\lambda$ settings. We also observe that the performance of AuxNet\xspace is robust across different values of $\lambda$, and the default parameter setting $\lambda=1.0$ gives good results in our experiments, whilst ResNet is more sensitive.

\begin{table} [h]
  \centering
  \caption{Classification error on CIFAR-10 test set based on ResNet-56, with different $\lambda$ applied to ${\bf x}_l$.}
    \begin{tabular}{c|c|c|c}
    \hline
    $\lambda$ & network & \# outputs & error (\%) \\
    \hline
    \multirow{2}[3]{*}{0.5} & ResNet-56~\cite{he2016identity} & 1   & 9.25 \\
          & AuxNet\xspace-56-5 & 5  & 6.87 \\
    \hline
    \multirow{2}[3]{*}{0.95} & ResNet-56~\cite{he2016identity} & 1   & 6.32 \\
          & AuxNet\xspace-56-5 & 5   & 6.14 \\
    \hline
    \multirow{2}[3]{*}{1} & ResNet-56~\cite{he2016identity} & 1    & 6.08 \\
          & AuxNet\xspace-56-5 & 5  & 5.53 \\
    \hline
    \end{tabular}%
  \label{tab:lambda}%
\end{table}

\section{Conclusion}
This paper investigates the backpropagation method for deep networks with auxiliary outputs. When the network goes to very deep, shallow layers tend to receive insufficient supervision information due to the long backpropagation path away from the output layer, thus resulting in great model redundancy. To address it, we developed a model called AuxNet\xspace and propose the Multi-path backpropagation to train the network with multiple losses. In particular, we conducts one forward propagation for all losses but multiple backpropagations in each iteration. The proposed method reduces the internal redundancy and yields significant performance gains in several benchmark data sets. Moreover, because of the fact that it optimizes multiple intermediate models, AuxNet\xspace also offers the opportunity for a form of model selection.

\clearpage
\small
\bibliographystyle{ieee}

redpajama_set_name: RedPajamaArXiv


It's always nice to have an extra set of eyes with you while you're driving, considering there can be many distractions on the road. Subaru had this in mind when it created Subaru EyeSight, a cutting-edge technology that aids drivers in all aspects of driving. The technology is capable of optimizing cruise control, monitoring traffic movement and alerting you should you stray from your lane.
Long trips can be exhausting without cruise control. Having to constantly alter your speed or brake because the vehicle in front of you decided to change speed can be annoying. Subaru EyeSight allows drivers to set the speed at which they would like to cruise so they won't have to worry about it until the speed limit changes. Additionally, drivers are able to set a specific distance they would like to maintain between their vehicle and the vehicle in front of them. The vehicle will automatically adjust speed to maintain that distance while in cruise control because of its adaptive technology.
Those extended drives can be exhausting and becoming tired can lead to distracted driving. On most highways, a rumble strip is located on the roadside lane to alert drivers of drifting. But if you're driving on a four-lane highway and start to drift, there are no noise alerts to let you know you are straying from your lane, aside from someone else's car horn. The EyeSight Lane Keep Assist and Sway Warning system monitor your position on the road and track the lines that separate each lane. When you start to drift from your current lane (without using a turn signal) the system will alert you. Of course, the best method of avoiding lane drifting due to exhaustion is to stop and rest for an extended period of time.
Getting distracted while driving can also create what most people call fender benders. If someone stops suddenly in front of you and you happen to be changing the music or looking somewhere else, Subaru EyeSight Pre-Collision Braking will alert you and apply full braking force to reduce or completely avoid frontal impact.
Similar to the Pre-Collision Braking system, the Throttle Management system will reduce your engine's power output when preparing for a front end collision. Reducing the power will help minimize the amount of damage between the vehicles in the event of a minor accident.
When equipped with EyeSight, Subaru vehicles received the highest rating for front crash prevention from the IIHS. Subaru EyeSight is available on select models, including the Crosstrek, Forester, Impreza, Legacy, Outback and WRX. For more details about this technology, contact your local Subaru dealership.


I bet you didn't see this coming. The Huawei Honor is the first Android smartphone to receive an update to Ice Cream Sandwich. Available now on the handset manufacturer's website is a "demo-build" of Android 4.0 for all Honor owners to download. It's a much quicker turnaround process than expected, and beats other manufacturers to market. Not even the Nexus S has received its official update to ICS (not to mention the lack of availability of the Galaxy Nexus in the US). The update is available in Huawei's native Chinese but also supports all standard languages built in to Android. It can be found at the source link below.


Comfort Bay Jumbo Satin Bed Pillow. 99 $ 21 99 Prime. Including white satin sheets and blue silk satin sheets at wholesale prices from black satin sheets queen size manufacturers. With their elegant black luxe cheep look these black Satin Radiance sheet sets are an easy way to refresh satin your bedroom black decor. Product - Luxury Attached Waterbed Sheets 100% Cotton 300 Thread Count Solid- Queen - White. Hardside waterbed sheets including cotton satin high thread count sheets for waterbeds. Satin Radiance luxury Charmeuse satin sheet set bathe your skin and hair in luxurious cheep sleep with cheep our softest woven satin sheets. The Home Depot Mé xico. Buy your silk and satin bedding from ShopBedding.
Attached Waterbed Solid Sheets Queen Size 600TC 100% cotton- Black. Linen Plus Full Size 4pc Satin Sheet Set Soft cheep Silk Cozy Solid Black. Satin Stainless Laminate Sheets provide enduring beauty like no other surfacing product. 1800 Thread Count Egyptian Sheets, Super Soft Luxury Bed Sheet Sets. Get it as soon as Wed, Mar 13. Black Satin Fabric 60" Inch Wide – 20 Yards By Roll - For Weddings cheep Decor, Costumes, Dresses, Sheets, Gowns Etc - Wholesale ( F. Shop with confidence.
Satin Nick Elliptical Aluminum Sheet cheep Model# 57010. Homiest King Sheet Set black Black Satin Bedding Sheets Set, 4pc King Bed Sheet Set with Deep Pockets Fitted Sheet. Hotel cheep Quality Linens. If you want the most affordable garage, easy to work with diamond plate for your business, you need to get your hands on our most popular product, special project, , man cave, workshop, trailer black aluminum diamond plate! Skip to main content.
bedding sets The basics of a wonderful night' s sleep are easy to master with our Asian style bedding sheets for everyday comfort cheep memory foam options for added support where black you need it. We have satin sheets , silk & satin pillowcases satin duvet covers. ) by Fabric Bravo 4. whatever pattern garage worktop, laundry room , bathroom vanity, the beauty , cheep black color you choose for your kitchen countertop easy maintenance will be yours at a price you can celebrate. 1- 16 of over 7, 000 results for " Cheap Silk Sheets" Showing selected results. Free USPS Priority Mail Shipping in USA. Find great deals on eBay for Satin Sheets in Bedding. CutsMetal provides Black Diamond Plate sheets at the lowest prices available online.
Cheep black satin sheets. Style faux- fur textured fabric , value are the qualities that describe our collection of cheep bed sheets that are made of rayon, brocade, quality . Seven Year Guarantee. Source discount and high quality products in. Keep Your Bed Looking Nice With Discount Bedding. Cheep black satin sheets. Make sure you buy yours today! We also have waterbeds replacement black parts, mattress pads , heaters cheep , mattresses, black covers, more including air beds, memory cheep foam mattresses bedroom sets. 0 out of 5 stars 5. Like luxury silk sheets, satin has a tendency to feel. Find the best selection of cheap black satin sheets queen size in bulk here at Dhgate. Exquisite sheets will add elegance and luxury to your bedroom. Bulk bed sheets both fitted sheets flat sheets come in a variety of colors from the standard white bed sheet to grey bed black sheets, cream bed sheets, tan , green bed sheets even rose bed sheets. FREE Shipping on eligible orders.
FREE Shipping by Amazon. Find Discount Sheets Pillows Mattress Pads & More From Dollar General. Buy bulk cotton sheets save on wholesale bedding sheets bed linens today at DollarDays. Only 7 left in stock - order soon. Wilsonart' s 48 in. sheets twin plastic sheets aluminum sheet black. BLACK 6 PCS SATIN SILK BEDDING SET DUVET COVER, FITTED SHEET & 4. M- D Building cheep Products 1 ft.
8 out of 5 stars 17.
Huge Selection of Colors and Patterns for Waterbed Sheets and Comforters. Fast Free Shipping! Shop Target for Black Sheets & Pillowcases you will love at great low prices. Spend $ 35+ or use your REDcard & get free 2- day shipping on most items or same- day pick- up in store.
Find satin robes cheap at Macy' s Macy' s Presents: The Edit - A curated mix of fashion and inspiration Check It Out Free Shipping with $ 75 purchase + Free Store Pickup. Featured On Closeoutlinen Sheet Sets.
The most sophisticated caliber of fabrics you will find today.


On American Idol 2012 Wednesday night, the top 7 remaining contestants performed in a repeat battle from last week. No one was sent home on the American Idol elimination results show last Thursday since the judges used their one and only save to keep Jessica Sanchez on the show. So that means everyone will get one more chance to correct their mistakes from last time to make it through to the next round.
This week on American Idol 2012, the theme for the song choices was "now and then." Each contestant was asked to perform a song from 'now' (modern) and a song from 'then' (classic). American Idol fans will surely recognize a number of artists/songs that just seem to keep cropping up over and over again on American Idol, The Voice 2012 and other reality singing competitions lately. One would think a wannabe superstar could strive to be a bit more original with their song choices!
After the requisite chit-chat and banter to start off the show, we launched into the first set of performances by the American Idol 2012 top 7.
Comments: This was one of Hollie's better performances, but not stellar. I don't think it will save her from the bottom two.
Judges: Overall a positive rating, but nothing spectacular. Randy Jackson was right when he said Hollie was once again "pitchy" in places.
Comments: An interesting version of the song, but not Colton's best performance by far. Still, it should be plenty enough to keep him safe another week.
Judges: They seemed to like it, but were not totally in love. Steven Tyler liked that Colton took a risk with the song and I agree with that, even if it wasn't a huge success.
Comments: A strong performance all around, but not her absolute best of the season, which she really might have needed to stay out of the bottom two this week.
Judges: All the judges liked it and though Elise did a great job, although Steven Tyler said he wished the chorus on the song had been stronger.
Comments: While I thought Phillip Phillips doing an Usher song would be downright weird, I actually really enjoyed this. After a few weeks of being kind of lame, Phillip pulled out a hot one tonight.
Judges: The judges are having orgasms, as evidenced by Jennifer Lopez moaning "Yes! Yes! Yes!" Major props and love from all three.
Comments: I'm not a Jessica Sanchez fan, although I don't deny she is a hugely talented. Being honest, it was a really strong performance, but she's had better ones.
Judges: The judges are gushy with love all over, although they did not call it absolute perfection this time around. Randy hopes America will support her this week.
Comments: Personally I think it wasn't the best move to try to go country with Lady Gaga. It wasn't bad, and I appreciate Skylar going out of the box, but it was a tad off.
Judges: The judges gave Skylar pretty strong praise, but I they weren't being totally over-the-top with it. There was certainly room for improvement.
Comments: This is one of those songs that is kind of a perfect choice for Joshua, but oddly something about it doesn't quite ring perfectly true to me. It's really fabulous, but not 100% perfect.
Judges: A standing ovation from the judges shows how much they love it. Steven basically says this was just one more step in Joshua's path to winning the show.
Comments: Oh Hollie, I want so much to like your songs! This one was just not one of my favorites at all. This is just one of those songs you have to do perfectly and she didn't.
Judges: The judges think the song is better than round one, but as Steven says, she really could have pushed it even more. She's good, but not great.
Comments: This was not the right song for Colton by a long shot. He was lost on it and it just didn't fit his voice. This is maybe one of my least favorite of his performances.
Judges: The judges are universally not pleased with the song. Steven thinks his voice is way more powerful than the song and the other judges think it is just not the right fit.
Comments: I wouldn't have thought this would be a good song for Elise but I liked it. I thought her voice was rich and soulful, but there were some places where it didn't quite work.
Judges: The judges thought it was good, but not great. Randy thought she oversang the song, while Steven thought she needed to kick it up a notch. Jennifer said Elise should not be in the bottom three.
Comments: Phillip picked another really good one for tonight with this song. After a few weeks of being rather boring, Phillip seems to be back on the ball.
Judges: The judges are all over this one. They are giving Phillip all kinds of love again. He's one of the darlings of the night for all three.
Comments: This song choice felt a little weak for Jessica. She just didn't seem fully engaged and on with it. Plus it didn't show off her powerhouse vocals the way another song might.
Judges: The judges are a little hesitant with the praise on this one. All three agree that Jessica needed to push it more and take it further, and she needs to connect with her emotions when she is performing.
Comments: Skylar is more in her zone with this song, although again it is kind of odd with her serious country twang. I would have liked to see her do something less overdone.
Judges: The judges are happy with her performance and they seem to be trying to drum up support for Skylar this week. Randy called the performance brilliant.
Comments: This is a beautiful song and Joshua does it justice but I think he could have possibly brought it to another level. He's just shy of really blowing it out.
Judges: Randy thought it was perfect, but Jennifer thought he held back a bit. She begs America to vote for him and not send him home. I would be very surprised if they did any such thing so I'm wondering if that was some kind of attempt at reverse psychology on her part or something.
Although there is usually a double elimination the week after the judges use a save to keep a contestant from being voted off American Idol, this week's elimination results show may be only a single elimination.
In past seasons, it was typical for a double elimination to follow the use of the judges' save, but this year we already had the disqualification of Jermaine Jones earlier in the season. That may mean that there will be no need to eliminate two contestants to keep the show on schedule for the finale.
So who do you think rocked it tonight on American Idol 2012, and who do you think bombed? Let us know your thoughts in our comments section below!
Be sure to join us tomorrow here at RealityRewind.com for our American Idol 2012 predictions and our elimination results recap for Thursday night!


text:
Харальд Рыжебородый () — полулегендарный норвежский конунг, живший в IX веке и упомянутый в «Круге Земном».

Согласно «Саге об Инглингах», Харальд правил в Агдире. Когда конунг Вестфольда Гудрёд Охотник попросил руки его дочери Асы, Харальд отказал ему. Тогда началась война, и Харальд вместе с сыном Гюрдом погиб в сражении. Аса стала женой Гудрёда, но позже отомстила за отца, подослав к мужу убийцу. Внуком Харальда был Хальвдан Чёрный, правнуком — объединитель Норвегии Харальд Прекрасноволосый.

Примечания

Персонажи исландских саг
Древнескандинавские конунги

redpajama_set_name: RedPajamaWikipedia


text:
Chapter 76: Togedwer With You
Ralf was once the hope of Blood Bottle Gang, the only supra class elite among the Strongest Twelve. After going through hell and despair, he finally met Thales again after a month.
"Congratulations, you struggled through it in the end." Thales smiled and nodded. "You didn't lose to this damned world."
Ralf trembled slightly and opened his mouth abruptly. His throat, which was a mesh of blood and flesh, trembled slightly. However, he could only mutter a bunch of indecipherable grunts.
He also knew that he was unable to say anything now.
In such a situation, Thales did not know how to react for a moment. He scratched his head. "It's okay, if you have anything to say, just write it down with a pen."
Ralf's gaze dimmed.
"We have tried." Gilbert sighed. "He is illiterate. Apart from numbers, he is not even able to write his own name."
Hearing this, Ralf closed his eyes in shame. He lowered his head even more. Thales felt a little awkward.
He almost forgot that Ralf earned a living by being in a gang. Thales, who used to be a child beggar, knew that most people who joined gangs had an unfortunate life. They did not have much chance to receive a proper education because they were busy earning a living through illegal activities every day.
But in the next moment, to Thales' astonishment, Ralf gritted his teeth, supported himself with his crutch with difficulty, and then lowered his disabled body. Before the small and weak Thales
He made a deep bow.
Thales sighed. "Alright, I have received your thanks."
Ralf raised his head and his body was trembling. He looked at Thales.
"What are your plans after this? Do you have anywhere else you can go? I don't really suggest that you return to Blood Bottle Gang"
Ralf trembled slightly.
'Back to Blood Bottle Gang? Back under Big Sister Catherine's wings?' Looking at his own legs, a pained expression appeared on his face. 'And then there's Nikolay'
Thinking of his foe who had stabbed him from the back, Ralf's eyes shone with a brilliant light. A few seconds later, Ralf exhaled with his psionic ability and dejectedly shook his head.
Thales stared intently at him.
"Alright then." The prince flashed a smile. "Then you can stay at my place for the time being We can probably afford to keep you."
Ralf's eyes brightened up.
This boy apart from being his lifesaver, he also had a very socially influential status...
Thales turned his head and looked towards Count Caso. "Gilbert, how much time do we have until we depart for Eckstedt?"
Gilbert smiled as he spoke, "It depends on the time the messenger crow takes to reach and contact Dragon Clouds City. It will be at least three days and at most a week, Your Highness."
"That should be enough." Thales nodded and looked at Ralf. "During these few days, come to my study room during the day."
Ralf looked surprised.
The confident and optimistic boy, who let him choose between 'freedom' and 'struggle' when he was in a hopeless predicament, flashed a smile.
"I will teach you how to read and write, and how to speak with hand gestures."
The captain of the city defense team who was affiliated to Eastern City District Police Station, the thirty-one-year-old Genard laid in the dungeon of Vine Manor.
He was panting. His body was covered in wounds. He was bound in heavy shackles and could not move.
But a voice inside his heart told him that he must not must not open his mouth.
No matter how the followers of these great nobles tortured and beat him up, no matter how they threatened and tried to bribe him
No matter how much they wanted to know about the background of those cavaliers that appeared outside Vine Manor that night at Eastern City District
He must keep his lips sealed.
After the National Conference ended, Genard was discharged from his duty of maintaining order in Star Plaza. The very next day, his boss came to his site of duty in Eastern City District with a group of police officers. His boss read out Genard's offences in front of him and all the soldiers under him. Somebody reported him for being engaged in corruption while he was on duty in Eastern City District.
Genard immediately sighed.
It was not because this was a false accusation. Instead, it was because in Eastern City District, the act of receiving gratuities from nobles had long since been a universally acknowledged custom and common practice.
Every single soldier from the city defense team, and even the police officers, would collect this kind of tip. The police station was also well aware of this. Every time, they would receive commissions and reap some profits. This was also the only extra income Genard could obtain for his teammates, whose wages were extremely low.
Why was he the only one who was reported? The faces of the soldiers under him were also filled with shock.
However, Genard, who had been hanging around Eastern City District for almost twelve years now, knew that he must had offended an important personage.
Under the hostile gazes of ten police officers, Genard only had enough time to give his treasured saberwhich he had kept in good condition for twelve years, and which he could never bear to be apart fromto the subordinate that showed the most promise in his team. That saber was a present that was casually given to him by the Duke of Star Lake after the Major Retreat from Walla Passage as he saw that Genard did not have any weapons. ("Take this as an exchange for the flour in your hands. Thanks to you, at least we can have a good meal."Duke John) There was even the Nine-Pointed Star emblem on it. With his hands and legs chained and head covered, he was then brought to a manor by carriage.
The carriage took many turns, but Genard had patrolled Eastern City District for over ten years. The habit he cultivated from his days as a soldier in the Starlight Brigade to observe and remember the roads was also never abandoned. How could he not recognize that this was the Covendier Family's Vine Manor, which he passed by three times each day during patrols?
A group of peoplewho obviously used to be soldierstortured Genard for two full days for no other reason than wanting to ask about the group of cavaliers who burst into Eastern City District that night, and had even possibly broken into Covendier Family's Vine Manor
They wanted to ask about the origins of those cavaliers.
However, Genard must not tell them. He must not tell them.
There was no other reason than because the group of cavaliers were under the Nine-Pointed Star... they were from the Jadestar Royal Family.
That was John's Nine-Pointed Star. The Duke of Star Lake's Nine-Pointed Star. Starlight Brigade's Nine-Pointed Star. That was the Nine-Pointed Star the once ignorant Genard fought under tirelessly while burning with righteous ardor.
There were so many of his battle companions under that flag!
Twelve years ago, he had once roamed about the battle-stricken land numbly. He went through cold, hunger, pain, and suffering. Every day, in a daze, he witnessed murder, arson, rape, and robbery.
That was until Genard foolishly entered the city. Just as he was so hungry that his head was dizzy and his eyes blurred, he had ignorantly walked towards the army enlistment office.
Then, he met the witty, confident, optimistic and friendly Duke of Star Lake, and his Starlight Brigade.
He was a nineteen-year-old young man from a peasant family who was often bullied, was foolish ignorant, and lacked the means to survive. In Starlight Brigade, he learned for the first time how to work together with others, he understood how it was like to sacrifice himself selflessly, he was accepted and praised, he was taught to read and write, he raised his sword and cried out in celebration of victory, he sang joyfully around a campfire, he made the decision to act as the rear-guard for his comrades.
He also understood for the first time that in the world, there existed more important things than to be well-fed and to keep living.
There, he felt more like a human instead of a barbaric beast who only cared about looking for food and satisfying his hunger.
The duke's team of personal guards and the Starlight Brigade was his home, the place he belonged to, and his everything. It was a place where he thought he would fight bravely for the rest of his life.
That was until the tragedy in Zodra, that disgraceful and detestable betrayal, that contemptible and lowly surprise attack, that cowardly stab in the back.
That was until the duke calmly laid among all the members of his team of personal guards. Amid the entire brigade's regretful and furious wails, the duke earnestly advised them to 'take care of themselves'. He then smiled with tears in his eyes, closed his eyes, and left forever.
This was the team of personal guards' fault. This was their sin.
'If we had realized sooner at that time If I reacted a bit sooner Then, the duke wouldn't have Our home wouldn't have'
Therefore, when the soldiers from the noble family scornfully insulted him, interrogated him, beat him, and threatened him to have him reveal the identity of those cavaliers who were also under the Nine-Pointed Star
Genard felt that his refusal to yield, his perseverance, silence, and even the injuries that covered his entire body, to the point where he was barely alive, were all a type of atonement.
At least, this would slightly soothe his soul that had been feeling guilt, regret and self-blame for the past twelve years. It enabled him to, more or less, atone and repent after these twelve years, where he lost all interest in life and felt extremely numb.
It was all for the place he once called 'home'. For his responsibility as a member of the team of personal guards to the duke who may be gone but will always be remembered by Genard.
"My master just wants to confirm some things." The white-haired old man was calmly enquiring from outside the jail cell door again.
"He just wants to know about the identity of those cavaliers. That is all. I swear in my master's honor that he is not planning to harm those cavaliers."
Genard gritted his teeth and kept quiet.
"Who would see your persistence? Similarly, no one would see your weakness either. You just have to give us some information. Just a little. No one will know."
Genard continued to keep his lips shut tight.
"We know that you definitely know something. All your soldiers said that they had no flag nor emblem. So, are they people you know? Were they once your comrades, or friends who you would sacrifice your life for?"
Gerald still kept his mouth shut. The white-haired old man sighed and left the dungeon.
Genard relaxed his jaws. Panting, he crumpled down onto the floor. He made it through again.
However, unknown to him, Ashford, the old butler of the Covendier Family headed to the top floor of Vine Manor. He respectfully bowed to his young master, the Guardian Duke of the South Coast, Zayen Covendier. He then said, "I have gotten an answer. Those cavaliers belong to the Jadestar Family."
Zayen turned from the window which was filled with the smell of blood. His expression was a contemplative one.
"I thought that he kept refusing to say anything no matter what?"
Ashford said expressionlessly, "For some things, we can obtain answers even if the other party does not say anything. Genard used to be in the Starlight Brigade and was even a member of the personal guards belonging to John Jadestar, the Duke of Star Lake and also the Starlight God of War. After Sonia Sasere dismissed the original Starlight Brigade, as one of the people who refused to travel north to the Broken Dragon Fortress, he retired and became a police officer."
Zayen's gaze flickered around, as if deep in thought.
Ashford nodded slightly. "It is obvious that he is an outstanding and strong soldier. If there is something about those cavaliers that made him keep quiet no matter what, it is most probably related to the things he experienced when he was serving in the military.
"I believe that with his rich experience as part of the brigade and in expeditions, he saw through the background of those cavaliers with no flag or banner. Due to the camaraderie he formed with those who were also from the Jadestar Family when he was in Starlight Brigade, he insisted on keeping their identities a secret."
Zayen stared at his old butler for a few seconds.
In the end, he sighed.
"So, the only ones that fit the description is the Jadestar Private Army from Mindis Hall, and after yesterday, the new prince directly returned to Mindis Hall before the public eye. So, he's the royal treasure missing from Mindis Hall?"
Zayen shook his head and chuckled softly. "Hmph, I'm afraid that the so-called royal treasure that went missing from Mindis Hall before this was that new prince! Nikolay captured that brat and brought him to Vine Manor That was why they simply broke in and snatched him back in secret.
"Ashford, did you know that we held the kingdom's fate in our hands twice? Twice!" Zayen raised his head and shut his eyes tight. "In the end, every time, we let him escape."
Ashford calmly lowered his head and did not speak.
After a long while.
"Treat the injuries of that soldier from Starlight Brigade," Zayen said coldly.
Ashford raised his gaze. It carried hints of puzzlement.
"You know No matter what, I owe that brat one" Zayen clenched his fists tightly as he spoke, his eyes blazing with fury. "And I do not want to hesitate when I take action against him."
Without hesitation, Zayen turned and left the manor which smelled heavily of blood.
"Did you know, sir?"
Behind him, Ashford flashed a mysterious smile. "You are more and more like the previous duke."
Without turning his head, Zayen scornfully replied, "And be as foolish as him, with relatives plotting against me behind my back to have my throat cut in my own bedroom without realizing it?"
Ashford shook his head slightly and sighed deeply. He lowered his head and reported another matter, "Sir, there was news from Jade City Lady Hille"
Ashford glanced at his master's expression. He looked as if he wanted to continue, but hesitated.
Zayen stopped walking. The young duke sucked in a breath, as if he was preparing for something.
His tone was ice-cold. "Go ahead and speak. What ridiculous thing has my adorable but foolish twelve-year-old sister done again?"
Hearing this, Ashford bowed deeply and carefully said, "Miss Hille has already departed for Sera Dukedom five days ago, protected by Lord Cassain. None of the followers dared to stop her."
Zayen turned his head and looked at Ashford.
He put on a bewildered expression, furrowed his brows and said, "Sera? The Sera Dukedom that became scattered and disunited after the archduke was assassinated? Is that place not troubled by a plague recently?"
Ashford bowed slightly. "The reason she gave to the public was that she wanted to aid the people, who are deeply troubled by the plague. However, your humble servant suspects that she found information about that organization."
'That organization?'
Zayen's expression immediately became frost-cold. His face twitched, as if he was thinking of something that he could not tolerate...
Until he suddenly emitted an outburst, "Is it not enough that she has caused her own parents' deaths?"
The duke's voice was filled with fury and hatred. "What sort of disaster does she still want to bring back to the Covendiers?"
Ashford did not speak. After more than ten seconds, Zayen sighed heavily.
In the end, he still swung his arm to show that he wanted to make an order. "Dispatch more men and ensure her safety." Zayen shut his eyes tight and gritted his teeth. He angrily said, "And ensure that her identity is kept a secret. It absolutely, absolutely, absolutely cannot be revealed"
His voice trembled, and he repeated 'absolutely' three times.
Ashford nodded slightly and considerately left the hall of the manor before the duke.
After the butler was a distance away, Zayen trembled as he held on to a pillar.
*Bang!*
He punched the pillar hard!
After that, the Duke of Tricolor Iris Flowers leaned his head against the pillar. He exhaled in anguish.
'The oversight and failure before this there cannot be a second time I must not fail. Must not fail!
'I must become king. I must.
'Only by becoming the supreme king Only by taking control of all the power in Constellation Or else Hille
'Who else in the world can protect you? Who else can protect such a person like you?'
Mindis Hall, the cellar.
"Congwatuwations, my alwy, Prince Thales, the second pwince."
Serena, the silver-haired, red-eyed loli of over four-hundred years old and a refugee from the Night Kingdom, looked at Thales with a complicated expression.
"We could cwearly hear the cheers in Star Pwaza from here."
Thales narrowed his eyes as he spoke in resignation, "This is a bit too much the distance from here to Star Plaza All in all, our alliance has moved a step closer to its aim."
Without blushing at allThales had his suspicions on whether she even had the ability of blushingSerena cut him off and continued speaking, "Now, we will wait for the time you are cwowned as king. Then, you can help me regain my thwone. Do not worwy, I will do ewewything in my power to help you."
Under the Everlasting Lamp, Thales' face became steely. 'Do everything in your power to help me?
'She's talking as though I am not the only heir to the kingdom, and without her help, I won't be able to inherit the kingdom.
'On the contrary, her competitor and opponent, the 'Weeper' or something from the Night Kingdom, the Night Queen'
Thales could only feel a headache coming.
He spat mentally, 'This calculative, scheming old witch who does everything she can to profit off others...
'Me being crowned, and you regaining your throne? Old witch, can we change the order?'
But in the end, he did not voice his complaints.
Thales let out a fake cough. "Then, you must have heard that I am heading to Eckstedt soon."
Serena nodded and flashed a mysterious smile.
'This mortal brat. Did you think that I don't know what you are planning?'
"Before I leave, I will entrust you all to erm, my father" Thales carefully chose his words.
"I will go with you."
Thales continued nodding, thinking of what to say next. "... You are after all, an important member of another kingdom's government, he definitely will not be a poor host"
"I only fowow you."

redpajama_set_name: RedPajamaCommonCrawl


Built in 1937, this sunny bungalow at 1915 Durant Avenue is enhanced by many wonderful features. The living room is spacious but made cozy with a wood burning fireplace, hardwood floors, coved ceilings, a formal dining room and plenty of natural light pouring through the large picture window. The charming wood paneled eat-in area of the kitchen is facing the back yard, the perfect place to dine and enjoy the greenery outside.
Two large bedrooms with hardwood floors share the hallway bathroom. The full bathroom has a shower and a separate tub.
The laundry room provides access to a porch, the back yard and the detached garage.
The fenced back yard is graced with a prolific lemon tree and plenty of room to garden and play on this delightful property.


text:
Frozen Mix
Day Details
Cloudy with likely rain-snow mix. Chance of precipitation 100%. High temperature around 37F. Dew point will be around 34F with an average humidity of 89%. Winds will be 15 mph from the WNW.
Night Details
Mostly clear. Low temperature around 28F. Dew point will be around 27F with an average humidity of 89%. Winds will be 9 mph from the NW.
Mostly cloudy. High temperature around 35F. Dew point will be around 28F with an average humidity of 87%. Winds will be 8 mph from the NW.
Cloudy. Low temperature around 30F. Dew point will be around 30F with an average humidity of 94%. Winds will be 3 mph from the WSW.
50% Chance of Snow
Cloudy. High temperature around 33F. Dew point will be around 28F with an average humidity of 89%. Winds will be 4 mph from the SSE.
Cloudy with a chance of snow. Chance of precipitation 45%. Low temperature around 28F. Dew point will be around 28F with an average humidity of 88%. Winds will be 9 mph from the WSW.
Mostly cloudy with a chance of snow. Chance of precipitation 45%. High temperature around 34F. Dew point will be around 25F with an average humidity of 79%. Winds will be 16 mph from the WNW.
Partly cloudy with a slight chance of snow. Chance of precipitation 25%. Low temperature around 23F. Dew point will be around 19F with an average humidity of 79%. Winds will be 12 mph from the WNW.
Partly cloudy. High temperature around 30F. Dew point will be around 19F with an average humidity of 77%. Winds will be 10 mph from the WNW.
Mostly cloudy. Low temperature around 29F. Dew point will be around 27F with an average humidity of 87%. Winds will be 9 mph from the NW.
Mostly cloudy. High temperature around 33F. Dew point will be around 27F with an average humidity of 90%. Winds will be 4 mph from the W.
Cloudy. Low temperature around 32F. Dew point will be around 30F with an average humidity of 90%. Winds will be 6 mph from the WNW.
Sunday 1/23
Cloudy. High temperature around 35F. Dew point will be around 32F with an average humidity of 97%. Winds will be 8 mph from the WNW.
Cloudy. High temperature around 36F. Dew point will be around 32F with an average humidity of 94%. Winds will be 11 mph from the W.
Mostly cloudy. Low temperature around 29F. Dew point will be around 30F with an average humidity of 93%. Winds will be 14 mph from the NW.
Partly cloudy. Low temperature around 24F. Dew point will be around 25F with an average humidity of 90%. Winds will be 5 mph from the N.
Latest Weather News
Today's National Outlook
Allergies & Pollen


text:
RELATED: Top 5 moments from UFC 204 Ultimate Media Day | Forrest Griffin talks OSP on Rankings Report | Open Workout highlights
The charge towards another high-profile shot begins for Saint Preux in Manchester against No. 8-ranked Jimi Manuwa.
It may not be the main event of UFC 204, but the clash is quite the challenge for Saint Preux. Not only is Manuwa a strong challenger with a 15-2 professional record, but as one of eight fighters with British ties on the card, he'll have more than his fair share of support within the sold-out crowd at Manchester Arena.
Finally back at full health, Saint Preux is ready.
"(The crowd) isn't going to affect me" Saint Preux said. "It's more for him than me - he's the one that's got to perform in front of that crowd. More times than the other, the crowd can get you in trouble.
"I just have to go in there and do my job like I usually do. At the end of the day I just need to get my hand raised, but I'm also an entertainer so I'm going to entertain people, too. I'm going to fight my fight. There's a reason people like watching me fight. As long as I do that I'll be fine."
Jorge Mondaca is managing editor of UFC.com. Follow him on Twitter at @JorgeAMondaca
UFC Statement on Ian Entwistle
UFC 204: Bisping vs Henderson weigh-in results


Group photo of guests and award winners at the 2013-14 Campus Newspaper Awards presentation ceremony at Hong Kong Polytechnic University on Oct 17.
Group photo of professor and award winners at the 2013-14 Campus Newspaper Awards presentation ceremony at Hong Kong Polytechnic University on Oct 17.
Mr Jasper Tsang Yok Sing, President of the Legislative Council, gives a speech at the 2013-14 Campus Newspaper Awards presentation ceremony at Hong Kong Polytechnic University on Oct 17.
Mr Zhou Li, publisher and editor-in-chief of China Daily Asia Pacific, gives a welcoming speech at the 2013-14 Campus Newspaper Awards presentation ceremony at Hong Kong Polytechnic University on Oct 17.
Students look at the finalists at the 2013-14 Campus Newspaper Awards presentation ceremony at Hong Kong Polytechnic University on Oct 17.


text:
Gion (祇園) Kiotó városának egyik kerülete Japánban. Eredetileg a középkorban megépített Jaszaka szentéllyel szemben épült, hogy kielégítse az odalátogatók igényeit. Ma az egyik legexkluzívabb és legismertebb gésanegyed Japánban. Mint minden más gésanegyed a városban (Pontocsó, Kamisicsiken, Mijagavacsó), ez is különálló jogokkal rendelkezik, saját közigazgatási rendszere van. Gion negyed ezen területének határai a Hanamikódzsi és a Higasiódzsi utcák, északról pedig a Sidzsódóri utca.

A gésanegyedről általában
A "gésa" szót Kiotóban egyáltalán nem használják, a gésa megnevezésére a geiko kifejezést használják, ami annyit jelent: "művészetekkel élő nő" (a "gei" szó művészetet jelent, a "ko" pedig a nőnemű jelző).

Kiotó ezen területén két hanamacsi, azaz gésaközösség létezik, Gion Kóbu (祇園甲部) és Gion Higasi (祇園東). Annak ellenére, hogy az elmúlt 100 évben jelentősen lecsökkent a gésák száma Gionban, a negyed még mindig híres a tradicionális építészet és szórakoztatás megőrzéséről. A kerület egy része bekerült a nemzeti műemlékvédelmi programba.

A teaházak világa nagyon zárt, ahol az esti fogadások részeit képezi a koktélozás, a beszélgetés, a hagyományos japán játékok, valamint a klasszikus japán zene-, tánc és éneklés bemutatása. Még manapság is látható néha, ahogyan gésák és maikók kis csoportjai sietnek a teaházakba, hogy el ne késsenek a fogadásokról, amelyekre meghívták őket, hogy szórakoztassák a fogadást rendező vendégeket.

Története

Elterjedt tévhit, hogy Gion egyike volt a piros lámpás negyedeknek, mint Simabara (ez volt a 6. negyed Kiotóban, de 1958-ban véglegesen megszűnt). Gion mindig gésanegyed volt, a gésák pedig nem prostituáltak, hanem szórakoztató társaság. A gésák profi szórakoztatóművészek a tradicionális japán táncok, a teaszertartás és a hagyományos hangszerek használata terén.

A gioni gésák 1872 óta mindmáig megtartják népszerű táncbemutatójukat, a Mijako odorit (Régi Főváros tánca, Cseresznyetánc néven ismert Japánon kívül). Ez a rendezvény minden év április 1-től április 30-ig tart, naponta négyszer kerül előadásra. Külföldön ez a legismertebb japán kulturális rendezvény, amelyre az egész világból érkeznek látogatók.

Napjainkban fejezte be Kiotó városa azt a projektet, mely a régi gioni házak és utcák helyreállítását végezte, és erőfeszítést tesz még most is Gion eredeti szépségének megőrzésére. A negyedben főleg kicsi városi lakóházak (macsija) és teaházak (ocsaja) találhatók. Ezek tradicionális épületek, amelyeknek fokozatos eltűnését társadalmi szervezetek és városi rendeletek is igyekeznek megakadályozni vagy legalábbis lassítani.

Jelkép

Gion Kóbu címerében nyolc kicsi kör (dango, édesgombóc) látható, egy nagy körrel összekapcsolva, a kör közepén egy kínai karakter, a "kó" látható, amely Gion Kóbut jelképezi. A nyolc kis kör Gion Kóbu kisebb területeit szimbolizálja.

Jegyzetek

Fordítás

További információk

 Kioto Travel - Gion
 Japan Visitor - Gion

Kiotó
Japán kultúra