text:
\section{Introduction}

\input{intro}
\section{Related work}
\input{related}

\section{Quantifying the role of context}
\input{model}

\section{Experiments and Results}
\input{results}

\section{Conclusions}
\input{conclusions}

{\small
\bibliographystyle{ieee}

\subsection{Object removal}
\label{sec:object_removal_model}
To create edited images with context objects removed, we need a fully automatic object removal model. For this, we utilize ground-truth object masks to remove the desired object and use an in-painting network to fill in the removed region.
We base our in-painting network on the model proposed in~\cite{shetty2018adversarial}, since this inpainter is directly optimized for removal, and can better handle irregular masks~\cite{shetty2018adversarial}, regularly encountered in object removal. More details about the network architecture can be found in the supplementary material.
The above removal method works well for medium sized objects, but struggles for large objects since then the in-painter needs to synthesize most of the image. Hence, we impose size restrictions on the objects we choose to remove to be less than 30\% of the image.
In the classification scenario on the COCO dataset, we consider all 80 object categories for removal.
In the segmentation setting on the ADE20k dataset, we consider only the non-stuff categories (90 categories) for removal and measure the effects of removing these objects on the segmentation of all 140 categories.
The stuff categories include objects like road, sky and field which are typically very large and hard to inpaint and hence are excluded from removal.
An important point to note here is that the in-painter model is not aware of the downstream models and is not optimized to fool or change their decisions. The effects of the in-painter are local and only affects the region the object is removed from.
Qualitative examples in Figures \ref{fig:classInconsist} and \ref{fig:InconsistentSeg} show that the in-painting works reasonably well in the object removal setting.

\subsection{Measuring context dependency}
\label{sec:robust_metrics}
To understand the effect contextual cues have on image-classification and segmentation models, we test them models on edited images where a context object has been removed.
Precisely, given an original image $I$ containing a set of objects $C=\{c_1, c_2 \cdots c_n\}$,  we first create a set of edited images $\mathcal{I}_e = \{I-c_i | c_i \in C\text{ and }\text{removable}(c_i)\}$. Next we test classification and segmentation models on $I$ and $\mathcal{I}_e$ and the check their outputs for consistency with the performed removal as described below for each task. %

\myparagraph{Image-level classification}
Given a trained classifier $S_{c_i}$ for class $c_i$, we will now characterize how robust it is to changes in context of $c_i$. We first obtain classifier scores for the original image $I$, edited image $I-c_i$ with object $c_i$ removed and for the edited set $\mathcal{I}_{\text{owc}} = \{I-c_j: c_j\in I , j\neq i\}$, all of which contain the object $c_i$ but have one context object removed. Ideally, if the classifier $S_{c_i}$ is robust to context changes it should score all the images in $\mathcal{I}_{\text{owc}}$ higher than the image $I-c_i$,
since $I-c_i$ does not contain the object $c_i$ and the images in $\mathcal{I}_{\text{owc}}$ do.
Precisely, a classifier robust to context should satisfy the below in-equality:
\begin{align}
 S_{c_i}(I_{\text{owc}})\geq S_{c_i}(I-c_i), \forall I_{\text{owc}}\in\mathcal{I}_{\text{owc}}
\label{eq:contconst}
\end{align}
\noindent We can count the number of times this condition is violated to quantitatively measure the robustness of the classifier.
{
\thickmuskip=0.4\thickmuskip
\begin{align}
    %
    V^{\text{min}}(c_i) &= \frac{\sum_{I}{\mathbbm{1}\left[\left(\min_{I_{\text{owc}}} {S_{c_i}(I_{\text{owc}})}\right) < S_{c_i}(I-c_i)\right]}}{\sum_{I}{\mathbbm{1}[c_i \in I]}}\\
    V^{\text{mean}}(c_i) &= \frac{\sum_{I}{\mathbbm{1}\left[\mathbb{E}_{I_{\text{owc}}}{\left[S_{c_i}(I_{\text{owc}})\right]} < S_{c_i}(I-c_i)\right]}}{\sum_{I}{\mathbbm{1}[c_i \in I]}}
\end{align}
}
\noindent where $\mathbbm{1}$ is the indicator variable. $V^{\text{min}}(c_i)$ is a strict metric counting instances classifier scores $I-c_i$ higher than any of the edited images, whereas $V^{\text{min}}(c_i)$ is a softer metric counting instances where $I-c_i$ is scored higher than the average score assigned to the edited images.

\myparagraph{Semantic segmentation}
To understand the role context plays in this pixel-level labeling task, we analyze the behaviour of a trained segmentation model by removing one object at a time from the original image. Specifically, we measure how the segmentation correctness of the rest of the image changes~(as compared to segmentation of the original image) when we remove an object from the original image. Given a segmentation model $P$, we compute the intersection-over-union~(IoU) for a class $c_i$ (w.r.t. ground-truth) on the original image $I$ and edited image $I-c_j$. If the IoU value changes more than threshold $\alpha$, we consider the segmentation prediction for class $c_i$ to be affected by removal of $c_j$. Counting these violations we get,
\begin{align}
    AR(c_i,c_j) &= \frac{\sum_I{\mathbbm{1}\left[\left|\Delta\text{IoU}_{c_ic_j}\right|\geq \alpha\right]}}{\sum_I{\mathbbm{1}\left[c_i,c_j \in I\right]}}
    \label{eq:crmetrticSeg}
\end{align}
\noindent where $\Delta\text{IoU}_{c_ic_j}$ is the change in IoU of class $c_i$ with removal of object $c_j$ and $\alpha$ is the change threshold. The matrix $AR(c_i, c_j)$, represents the fraction of images where removing the object $c_j$, affects the segmentation of the object $c_i$ with high values of $AR(c_i, c_j)$ indicating that the segmentation model depends heavily on the presence of the context object $c_j$ to segment $c_i$.
\begin{figure}
    \setlength{\tabcolsep}{0.1mm}
    \noindent\makebox[\linewidth]{
    \scriptsize
    \begin{tabular}{x{1.8cm}cx{5mm}c}
        & \bf\small \textcolor{blue}{Object} without \textcolor{magenta}{Context} && \bf\small \textcolor{magenta}{Context} without \textcolor{blue}{Object}\\[0.5em]
    %
         \includegraphics[valign =c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000158384_keyboard/real} Original
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000158384_keyboard/f_sec_rtv}&
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000158384_keyboard/f_prim_rkeyboard}\\
          %
         \bf Regular       & $S(\text{\it keyboard})=\textcolor{red}{1.99}$\textcolor{red}{\Lightning}         &\multirow{2}{*}{\large$\boldmath\geq$ }&$S(\text{\it keyboard)} = \textcolor{red}{4.67}$\textcolor{red}{\Lightning}\\
         \bf Ours & $S(\text{\it keyboard})=\textcolor{PineGreen}{3.40}$    &&$S(\text{\it keyboard}) = \textcolor{PineGreen}{1.39}$\\[0.5em]

         \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000136908_skateboard/real} Original
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000136908_skateboard/f_sec_rperson}&
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000136908_skateboard/f_prim_rskateboard}\\

          %
         \bf Regular       & $S(\text{\it skate}) = \textcolor{red}{0.39}$\textcolor{red}{\Lightning}       &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it skate}) = \textcolor{red}{2.97}$\textcolor{red}{\Lightning}         \\
         \bf Ours & $S(\text{\it skate}) = \textcolor{PineGreen}{2.33}$ && $S(\text{\it skate}) = \textcolor{PineGreen}{-0.13}$ \\

         \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000219841_frisbee/real} Original
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000219841_frisbee/f_sec_rperson}&
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/correlErrors/COCO_val2014_000000219841_frisbee/f_prim_rfrisbee}\\
         \bf Regular       & $S(\text{\it frisbee})=\textcolor{red}{0.39}$\textcolor{red}{\Lightning}    &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it frisbee}) = \textcolor{red}{2.06}$\textcolor{red}{\Lightning}\\
         \bf Ours & $S(\text{\it frisbee})=\textcolor{PineGreen}{3.32}$    && $S(\text{\it frisbee}) = \textcolor{PineGreen}{0.23}$ \\[0.5em]

         \includegraphics[valign = c, width=0.9\linewidth]{classification/difficultCases/COCO_val2014_000000003865_person/real} Original
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/difficultCases/COCO_val2014_000000003865_person/f_sec_rkite}&
        & \includegraphics[valign = c, width=0.34\linewidth]{classification/difficultCases/COCO_val2014_000000003865_person/f_prim_rperson}\\
        %

         \bf Regular       & $S(\text{\it person}) = \textcolor{red}{2.15}$\textcolor{red}{\Lightning}      &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it person}) = \textcolor{red}{2.79}$\textcolor{red}{\Lightning}      \\
         \bf Ours & $S(\text{\it person}) = \textcolor{PineGreen}{2.83}$&& $S(\text{\it person}) = \textcolor{PineGreen}{-2.20}$\\[0.5em]

         %
        %
        %
        %
        %
        %
        %
        %
    \end{tabular}
    }
    \vspace*{-2mm}
    \caption{Context violations by image-level classifier. The primary object is marked with blue box and the context object is marked with magenta. The first column shows the original image, middle shows the image with only object and the third with only the context. We see that the baseline classifier depends heavily on the context and always scores the context only images~(last column) higher than the image with only the primary object (middle column). The data augmented model does better and gets the ordering right. }
    \vspace*{-1em}
    \label{fig:classInconsist}
\end{figure}
\begin{figure*}
    \setlength{\tabcolsep}{1pt}
    \noindent\makebox[\textwidth]{
    \scriptsize
    \begin{tabular}{cccccc}
    %
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_sec}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/pred_sec}\\
          original $(\mathcal{I})$ & Upernet   & Ours & original $(\mathcal{I})$ & Upernet & Ours\\
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001992_sidewalk/pred_sec_edited}\\
          $\mathcal{I} - sign $ & Upernet & Ours & $\mathcal{I} - car$ & Upernet & Ours\\[0.3em]
        \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/pred_sec}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/pred_sec}\\
          original $(\mathcal{I})$ & Upernet  & Ours & original $(\mathcal{I})$ & Upernet & Ours\\
        \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth,  width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001388_grass/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000851_sidewalk/pred_sec_edited}\\
          $\mathcal{I} - tree $ & Upernet  & Ours & $\mathcal{I} - tree$ & Upernet & Ours\\
    \end{tabular}
        }
    \vspace*{-0.5em}
    \caption{Examples of segmentation failures due to removal of a single context object. We see the segmentation of road, sidewalk and grass affected significantly when context objects like signboard, car and tree is removed~(comparing odd and even rows). Model trained with proposed data-augmentation is more robust to these changes.}
    \vspace*{-1em}
    \label{fig:InconsistentSeg}
\end{figure*}
\subsection{Data augmentation with object removal}
\label{sec:da_strategies}
We now present our data augmentation solution to reduce the sensitivity of classification and segmentation models to context distribution. The main idea is to expose these models to training images of object-without-context and context-without-object scenarios. This will help the models deal with the lack of contextual information and hence become more robust to context changes. For this we perform object removal to create edited images with some objects removed and add these edited images to the training batch. Specific details of how to pick objects for removal and how to use them in training for the two tasks are discussed below.

\myparagraph{Classification} For classification we experiment with two strategies to use the edited images in training. In the first approach, we refer to as \emph{Data-aug-rand}, we randomly select one object to remove with uniform probability. Edited image is assigned the same labels as the same as the original image excluding the removed object class. Now we train the classifier with original and data augmented images using simple binary cross-entropy loss. In the second approach referred to as \emph{Data-aug-const} we explicitly optimize for robustness by including the in-equality in \eqref{eq:contconst} in the loss function. To do this, for a randomly selected images in the training batch, we create the full edited image set $\{I-c_i: c_i\in I\}$. Then we can incorporate the robustness constraint as a hinge loss with final loss being weighted sum of cross-entropy loss and the hinge loss $L_h$.
\vspace*{-2mm}
\begin{align*}
    \mathcal{L}_h(I) &= \sum_{c_i\in I}\max{\left[0, S_{c_i}(I-c_i)-\min_{c_j, j\neq i}{S_{c_i}(I-c_j)}\right]} \ \  (5)
\end{align*}\vspace*{-2mm}

\myparagraph{Segmentation} We also perform data augmentation on the segmentation task by creating edited images by selectively removing objects. The edited images can be used in training the segmentation model in two ways. First we can simply ignore the removed pixels and train the model to predict the original ground-truth labels on the rest of the image~(\emph{Ignore}). This helps the model learn that the labeling of a pixel should not be affected by the removal of a context object. Alternatively, we can explicitly tell the model that the removed object is not present by minimizing the likelihood assigned to the removed class at the edited pixel locations~(\emph{Negative loss}). %
The next question is how to sample the objects to remove. We explore three different strategies to select these objects. The first strategy, \emph{Random},  selects one random object to remove from the objects present in the image with uniform probability. However, sometimes the \emph{Random} strategy can select very large object for removal, which can harm the quality of the edited image. To address this we use the \emph{Sizebased} strategy, which select objects based on their relative sizes in the image, giving higher probability to selecting smaller objects. The probability for picking an object is computed as
    $p(c_i,I) \propto \left[\frac{\sum_{c_i \in I}{a(I,c_i)}}{a(I,c_i)}\right]$
where $a(I,c_i)$ is the area of the class $c_i$ in image $I$.%
We also explore a hard negative mining based strategy, where we create harder training examples for the segmentation model by removing easy classes.  This allows the model to focus on segmenting the harder classes while also becoming robust to context. Concretely, in this \emph{HardNegative} strategy we monitor the average cross-entropy segmentation loss $l_{\text{avg}}(c_i)$ for an object class $c_i$ and calculate the probability of removal of $c_i$  as inversely proportional to $l_{\text{avg}}(c_i)$.%

\subsection{Image level classification}
\subsubsection{Experimental setup for classification}
\vspace*{-1mm}
\myparagraph{Training data} We run our classification experiments on the COCO dataset~\cite{lin2014microsoft}, which contains 80 labeled object classes in their natural contexts. The dataset also has bounding box and segmentation annotation for each object. We use image-level labels to train the classifiers and use the object segmentation masks to test them with object removal.

\myparagraph{Out-of-context testing} Apart from testing the classifier models on regular COCO data we conduct additional experiments to quantify the performance in out-of-context scenarios with natural images. We divide the COCO images into two splits: the first split \emph{Co-occur} with images having at least two objects in them and the second split \emph{Single} with images containing a single object. The \emph{Full} split is all images combining \emph{Co-occur} and \emph{Single}. The idea behind this splitting of the dataset is to separate out images where objects occur in their context~(\emph{Co-occur}) and images where object occur alone without the usual co-occurring context objects~\emph{Single}. Now we can train our models on the \emph{Co-occur} split and test it on the \emph{Single} split to measure, using only real images, how a classifier trained with only co-occurring objects performs when objects appear without the context seen in training.
Additionally we also test our COCO trained models on the \emph{UnRel} dataset~\cite{Peyre17} which contains natural images with objects occurring in unusual contexts and relationships. We keep the classes which map to one of the 80 object classes in COCO, leaving 29 classes and 1071 images in the UnRel dataset.

\myparagraph{Baseline classifier} The image-level classification model we test is based on the architecture proposed in~\cite{oquab2015object}. It consists of a Imagenet~\cite{imagenet_cvpr09} pre-trained VGG-19 network for feature extraction network followed by two convolution layers, global max-pooling layer and a linear classification layer with sigmoid activations. The model is trained with binary cross-entropy loss. We train and test the model at single scale at 256x256 resolution, to simplify the analysis. Our classifier achieves similar mAP on real coco data as reported in \cite{oquab2015object}, with our mAP slightly lower (0.600 vs 0.628 in \cite{oquab2015object}) due to single scale training and testing. %

\begin{figure*}[t]
    \adjustbox{valign=t}{
    \begin{minipage}[t]{.58\linewidth}
        \centering
      \begin{minipage}[b]{1.0\linewidth}
        \newcommand*{\mrowt}[1]{\multirow{2}{*}{#1}}
        \footnotesize
        \setlength{\tabcolsep}{0.1mm}
        \begin{tabular}{x{2cm}c ccc cc c}
        \toprule
        \multirow{2}{*}{Model} & \multirow{2}{*}{\parbox{1.8cm}{\centering Training Data}}& \multicolumn{3}{c}{COCO test set}&  \multicolumn{2}{c}{Robustness Metrics}& UnRel\\\cmidrule(lr){3-5}\cmidrule(lr){6-7}
                               & & Full $\uparrow$& Co-occur $\uparrow$& Single $\uparrow$& $V^{\text{min}} \downarrow$& $V^{\text{mean}} \downarrow$ &dataset $\uparrow$ \\\cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-5}\cmidrule(lr){6-7}\cmidrule(lr){8-8}
            Baseline        & Full~(39k) & 0.60    & 0.57    & 0.62    & 34\% & 24\%        & 0.50    \\
            Data-aug-rand   & Full~(39k) & \bf0.61 &\bf0.58  & \bf0.65 & 32\% & 22\%        & \bf0.54 \\
            Data-aug-const  & Full~(39k) & 0.60    &\bf0.58  & 0.63    & \bf 25\% & \bf14\% & 0.52     \\\cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-5}\cmidrule(lr){6-7}\cmidrule(lr){8-8}
            Baseline        & Co-occur~(30k)& 0.56    & 0.55    & 0.58    & 34\% & 24\%       & 0.46      \\
            Data-aug-rand   & Co-occur~(30k)& \bf0.58 & \bf0.57 & \bf0.60 & 31\% & 21\%       &0.49       \\
            Data-aug-const  & Co-occur~(30k)& \bf0.58 & \bf0.57 & \bf0.60 & \bf27\% & \bf15\% & \bf0.51   \\
         \bottomrule
        \end{tabular}
        \vspace*{-2mm}
        \captionof{table}{ Effect of data augmentation on classification model}
        \label{tab:da_classification}
    \end{minipage}\\[0.5em]
\begin{minipage}[b]{1.0\linewidth}
        \footnotesize
    \setlength{\tabcolsep}{8pt}
    \begin{tabular}{p{10mm} p{8mm}p{8mm} p{8mm}p{8mm} p{8mm}p{8mm}}
    \toprule
      \multirow{2}{*}{Model} & \multicolumn{2}{c}{all (407 images)}& \multicolumn{2}{c}{with car (258)}& \multicolumn{2}{c}{without car (149)}\\
      & Road & Sidewalk &Road & Sidewalk &Road & Sidewalk \\\cmidrule(lr){1-1}\cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
      Upernet   &  0.81 & 0.59 & \bf0.86 & \bf 0.67 & 0.68 & 0.40 \\
      DataAug  &  \bf0.82 & \bf0.60 & \bf0.86 & 0.65 & \bf0.72 & \bf0.46 \\
      \bottomrule
    \end{tabular}
     \vspace*{-2mm}
    \captionof{table}{Comparing the performance of road and sidewalk\\ segmentation on natural images with and without cars.}
    \label{tab:daseg_road}
\end{minipage}
\end{minipage}
}\hfill
\adjustbox{valign=t}{
    \begin{minipage}{.42\linewidth}
        \includegraphics[width=0.8\linewidth]{images/classification/Effect_of_data_augmentation_const.pdf}
        \captionof{figure}{Comparing the \% of violations in different classes with and without data augmentation. Points below the diagonal line show improvement with data-augmentation and the ones above degrade. The colors denote the average precision.}
\label{fig:PerclassViol_DA_comparison}
\end{minipage}}
\end{figure*}
\subsubsection{Analyzing classifier robustness to context}
To measure the robustness of the trained classifier to context, we test it on real images and edited images and compute the robustness scores $V^{\text{min}}$ and $V^{\text{mean}}$ as described in \secref{sec:robust_metrics}.
\Tableref{tab:da_classification} shows the robustness scores averaged over all classes computed on the COCO test along with the standard performance metric mean average precision (mAP) for the baseline classifier~(first row). We can see that, despite achieving good mAP (0.6), the baseline classifier trained on full data performs poorly in-terms of robustness metrics. In about 34\% of cases the model violates the context consistency requirement of \eqref{eq:contconst}. This means in 34\% cases, the classifier scores images without the target object higher than an image where object is present but a context object has been removed. Comparing the per-class robustness score, $V^{\text{min}}(c_i)$ and the per-class average precision~(AP)~(see supplementary for visualization), we see that good performance in AP does not mean the classifier is robust to context. Many classes like mouse, keyboard, sink, tennis racket etc, which are performing well in AP ($\geq 0.8$), but have poor robustness to changes in context ($V_o^{\text{min}} \geq 50\%$). In extreme case, the \emph{mouse} classifier violates the consistency in more than 90\% of cases, despite having very good AP (0.88). This indicates that the classifiers are relying too much on contextual evidence to detect the objects but perform poorly when tested on images where the context distribution is different from  training.%

We visualize these violations in \figref{fig:classInconsist}. In the first row we see the case where the keyboard classifier scores the image with the keyboard removed higher~(4.67) than the image with the keyboard but with the monitors removed (1.99). Similarly we see the skateboard and the frisbee classifiers relying on person to hallucinate the respective objects. The violations shown in the first three rows of \figref{fig:classInconsist} occur in objects with high co-occurrence dependence with other classes.
However, such context violations can also be seen in classes like \emph{person} which occur in very different contexts as seen in the last row of \figref{fig:classInconsist}. Here, the violation occurs in a difficult image where the \emph{person} is small, but a more distinct class with co-occurrence dependence on person is clearly visible (\emph{kite}). The classifier seems uses the \emph{kite} context class to hallucinate that there is a \emph{person}, even when the \emph{person} has been removed.

\subsubsection{Data augmentation to improve robustness}
We train two variants of the data-augmented image classification models as described in \secref{sec:da_strategies}. The first \emph{Data-aug-rand} learns with standard cross-entropy loss on the edited images with a random object removed and the second \emph{Data-aug-const} which is optimized directly for robustness using a set of edited images and hinge loss.

\myparagraph{Quantitative results} We present the evaluation of the data-augmented and the baseline models in \tableref{tab:da_classification}. On models trained with \emph{Full} training data, the data-augmented model \emph{Data-aug-rand} provides a small improvement in overall mAP on the COCO test set (0.61 vs 0.60). However measuring the performance on the two splits \emph{Co-occur} and \emph{Single} reveals that the improvement is significant on the \emph{Single} split (0.65 vs 0.62), indicating that the data augmentation helps the classifier better deal with out of context objects.
This is also seen when comparing the performance of the two models on the UnRel dataset, where \emph{data-aug-rand} significantly improves over the baseline model (0.54 vs 0.50).
This improved robustness of the data augmented classifier to context changes is also measured by our robustness metrics $V^{\text{min}}$ and $V^{\text{mean}}$.
\emph{Data-aug-rand} classifier makes overall 2\% less violations under both worst-case~($V^{\text{min}}$) and average-case~($V^{\text{mean}}$) context changes.
Directly optimizing the robustness constraints allows the model \emph{Data-aug-const} to significantly improve upon the baseline model in robustness metrics, while still obtaining improvement in the performance metrics. It exhibits much less worst-case~(25\% vs 34\% for baseline) and average-case violations~(14\% vs 24\% for baseline), while improving the performance in the UnRel dataset~(0.52 mAP vs 0.50 for baseline).
The benefit of optimizing for robustness is clearly seen when we constrain the training data to the \emph{Co-occur} set, where the classifier never sees objects alone. Baseline model trained on the \emph{Co-occur} set drops in performance on the \emph{Single}~(0.58 from 0.62 on when trained on \emph{Full}) and the UnRel test sets~(0.46 vs 0.50 with \emph{Full}) . However, with data augmentation and enforcing robustness constraints, we can recover some of this performance. On the \emph{Single} test set \emph{Data-aug-const} model trained on \emph{Co-occur} set gets 0.58 mAP compared to 0.60 by baseline model trained on full data and even surpass it on the UnRel test set with 0.51 mAP. This shows that the data augmented model is able to overcome the contextual bias in the training set and perform well in unseen contexts.

When we compare the per-class robustness metrics between regular and data augmented models~(data-aug-const), as shown in the \figref{fig:PerclassViol_DA_comparison}, we see that data-augmentation significantly reduces the worst case violations~($V^{\text{min}}$) on well-performing classes. For example, $V^{\text{min}}$ drops from 95\% to less 36\% for the mouse class and from 58\% to 28\% for the keyboard class. %
The effect of this increased robustness is seen in qualitative examples in \figref{fig:classInconsist}.
For example, in the first row the baseline keyboard classifier gives too much weight to evidence from \emph{monitor} and scores the image with only \emph{monitor} higher than the only keyboard. However, the data augmented model correctly orders the images.

\subsection{Semantic segmentation}
So far, we have seen that multi-label classification models suffer from sensitivity to context, with classifiers often mixing up contextual and visual evidence. Next we will measure the context sensitivity of models in a more local and strongly supervised task of semantic segmentation.

\subsubsection{Experimental setup for segmentation}
\myparagraph{Training and test data} We conduct our semantic segmentation experiments primarily on the ADE20k dataset~\cite{zhou2017scene} containing 140 categories of labeled objects, in different settings. Some of the 140 classes are typical background classes like \emph{sky}, \emph{sea} and \emph{wall} and are large and difficult to in-paint and are hence excluded from removal.

\myparagraph{Out-of-context testing} Following the process in image-level classification, we also measure the performance of the segmentation models on real out-of-context data. This in done in two ways. First, we train the segmentation model in a restricted setting with only three classes \emph{car}, \emph{road} and \emph{sidewalk}. Now, we can again make two splits of the training and testing images into the \emph{Co-occur} split of images with at-least two objects~(3317 images) and the \emph{single} split with only a single object~(1693 images). Then we train the segmentation models on \emph{co-occur} split and test on \emph{single} split to see how well it can perform  segmentation without context.
Additionally we also test the models trained with ADE20k data on the Pascal-context dataset~\cite{mottaghi2014role} in order to measure the performance under a different context distribution.  This is done by manually mapping the 59 labels in the pascal-context to ADE20k labels and restricting the segmentation model to produce only the mapped labels.

\myparagraph{Baseline segmentation model}
We use the recent UperNet~\cite{xiao2018unified} model, with good results on the ADE20k, as our baseline segmentation model. We train the variant with the Resnet-50 encoder and a Upernet decoder with batch size of 6 images~(maximum that fit in GPU) and with the default hyper-parameters suggested by the authors. This model achieves mean intersection-over-union~(mIoU) of 0.377 and accuracy of 78.19\% with single scale testing. %

\subsubsection{Context in semantic segmentation}
We analyze robustness of the segmentation models to context by removing objects and computing the matrix $AR(c_i, c_j)$ presented in \secref{sec:robust_metrics}, which measures the \% of images where removal of object $c_j$ significantly affects segmentation of object $c_i$.
The matrix $AR(c_i,c_j)$ we obtain for the Upernet model in ADE20k dataset is a sparse matrix with sharp peaks~(see supplementary for a visualization). This indicates that the classes depend on specific context objects and are significantly affected by their removal. The sparsity also indicates that the effects on the segmentation are due the class being removed and not in-painting artifacts~(otherwise the segmentation would be affected by all removal). Some of dependencies we discover in  $AR(c_i,c_j)$ are reasonable and harmless, for example between \emph{pot} and \emph{plant}~($AR=50\%$). Once you remove the \emph{plant}, \emph{pot} looks more like a \emph{trash can} and the segmentation model often flips the label to \emph{trash can}. However other dependencies are spurious and not desirable. For example, we notice that often the segmentation model uses presence of \emph{car} to differentiate between \emph{road} and \emph{sidewalk}. Removing \emph{car} affects the IoU of the \emph{road} and \emph{sidewalk} in 21\% and 22\% of cases respectively. This dependence is undesirable, and can be catastrophic in applications like self-driving cars.

We show qualitative examples where removal affects segmentation of Upernet model in \figref{fig:InconsistentSeg}. The first two rows show the cases where removal of an object negatively impacts the segmentation of other objects. This include cases where removal of \emph{street sign} and \emph{car} severely affects segmentation of \emph{road} and \emph{sidewalk}, and a case where removal of \emph{trees} affects segmentation of \emph{grass}. We can see from these examples that while edit on the image is small and local, the effects of this removal on segmentation prediction is not local. Removal of a small objects can have drastic effects on segmentation in a far-away region.

\subsubsection{Data augmentation for segmentation}
\begin{table}[t]
    \newcommand*{\mrowt}[1]{\multirow{2}{*}{#1}}
    \footnotesize
    \setlength{\tabcolsep}{1mm}
    \centering
    \begin{tabular}{lc cc}
    \toprule
    \multirow{2}{*}{Model} & Removed &  \multicolumn{2}{c}{ADE20k}\\\cmidrule(lr){3-4}
              & Pixels & mIoU & Acc\\\cmidrule(lr){1-2}\cmidrule(lr){3-4}
        Upernet\cite{xiao2018unified}      & - & 0.377 & 78.31 \\\cmidrule(lr){1-2}\cmidrule{3-4}
        %
        %
        DA (random)        & Ignore   & 0.320 & 75.2 \\
        DA (sizebased)     & Ignore   & 0.379 & 78.31 \\
        DA (hard negative) & Ignore   & 0.375 & 77.8 \\\cmidrule(lr){1-2}\cmidrule{3-4}
        DA (sizebased)     & Negative & 0.377 & 78.25 \\
        DA (hard negative) & Negative & \bf0.385& \bf78.47\\
     \bottomrule
    \end{tabular}
    %
    \caption{Data augmentation results on ADE20k dataset}
    %
    \label{tab:da_segmentation}
    %
\end{table}
Next we will look at the results of using data-augmentation for segmentation models. For this purpose we train the Upernet~\cite{xiao2018unified} based data-augmented models on the ADE-20k dataset with on three different strategies for selecting the object to remove as discussed in \secref{sec:da_strategies}.

\myparagraph{Quantitative results} \Tableref{tab:da_segmentation}, shows the results comparing the data-augmented models with the baseline Upernet model. We can see that random sampling strategy, which worked well in image classification, fails here leading to drop in performance. This is because, many object categories in ADE20k dataset are large and difficult to remove like bed, sofa and mountain and random strategy suffers by picking these. Instead when we switch to size-based and hard-negative based sampling, we see that the performance improves and the the size-based sampling model achieves the best mIoU of the three models (0.379). Applying negative likelihood loss on the removed object class gets further improvement when combined with hard negative sampling. This model also improves upon the Upernet baseline~(achieving 0.385 IoU vs 0.377 by Upernet), despite the fact that the removal based data-augmentation is designed to make the model more robust to contextual variations.%

To understand how data-augmentation impacts sensitivity to context, \Figref{fig:PerclassViolSegm_DA_comparison} visualizes the maximum sensitivity of a class to removal of other classes, $\max_{c_j}{AR(c_i,c_j)}$ for different classes with and without data-augmentation. We see that for majority of classes robustness to context improves with data augmentation. For example \emph{pillow} class is only affected 32\% of the time with context changes, compared to 53\% before data augmentaion. Similary, \emph{road} and \emph{sidewalk} classes are only affected 9\% and 14\% of the time respectively, compared to 21\% and 22\% before.
This improved robustness translates into better generalization to real out-of-context data. We can see this in~\tableref{tab:daseg_road} where the performance of the \emph{road} and \emph{sidewalk} segmentation is measured on the validation set on images with and without cars. On the full set and on the split with cars, we see that the performance of the baseline Upernet and our augmented model (DA hard negative with negative loss) is equivalent. However, when we look at only images without car, the Upernet model performs significantly worse in both road ~(0.68 vs 0.72 for ours) and sidewalk~(0.40 vs 0.46 for ours) segmentation. This quantitatively shows that the baseline model struggles to distinguish between \emph{road} and \emph{sidewalk} without \emph{car} in the image, whereas our data augmentation is more robust and performs well even without context~(car).

We also see the benefit of data augmentation in experiments on restricted \emph{Co-occur} training set and on the Pascal-context dataset.
Our data augmented model outperforms the Upernet model~(both trained on the ADE20k dataset) when tested on the Pascal-context dataset in both mIoU and pixel accuracy. While the Upernet model achieves mIoU of 0.284 and pixel accuracy of 61.3\% our data augmented model achieves 0.293 and 62.10\% respectively, indicating that it is able to generalize better when tested on a dataset with different context distribution than one seen during training.
\Tableref{tab:da_seg_rest} presents the experiments with the \emph{Co-occur} training set in the three class setting. First we can see that when we switch from training on \emph{Full} training data to \emph{Co-occur} split (containing only images with atleast two objects), the performance of the Upernet greatly drops on the \emph{Single} test split~(from 0.67 to 0.52). This is indicates that the model overfits to the context it sees, and is not able to segment objects when it seeing them out of context. However, with data-augmentation we generate images of objects without context, and can recover most of this performance loss~(0.646). Surprisingly, data-augmented model trained on smaller \emph{co-occur} data also outperforms the baseline trained with \emph{Full} data when tested on the \emph{co-occur} split. %

Qualitative examples in \figref{fig:InconsistentSeg} also show the effect of increased robustness to context. While the baseline Upernet model is affected by context object removal causing drastic changes in predictions of other regions, our data augmented model is more stable. For example the removal of \emph{signboard},  \emph{car} or \emph{tree} does not effect the segmentation of the \emph{road} or \emph{sidewalk} by our model. %

\begin{figure}
\centering
    \includegraphics[width=0.65\linewidth]{images/segmentation/Effect_of_data_augmentation_withIous.pdf}
    %
    \caption{Comparing the context sensitivity of different classes with and without data augmentation with $\max_{c_j}{AR(c_i, c_j)}$ metric. Points below the diagonal improve with data-augmentation. The color denotes the mIoU. }%
    %
\label{fig:PerclassViolSegm_DA_comparison}
\end{figure}

\begin{table}[t]
    \newcommand*{\mrowt}[1]{\multirow{2}{*}{#1}}
    \footnotesize
    \setlength{\tabcolsep}{1mm}
    \centering
    \begin{tabular}{lc ccc }
    \toprule
    Model & Training Data& Full & Only Cooccur & Only Single \\\cmidrule(lr){1-1}\cmidrule(lr){2-2}\cmidrule(lr){3-5}\cmidrule(lr){3-5}
        Upernet  & Full~(5k) & 0.774 & 0.797 & 0.670 \\
        Data Aug & Full~(5k) & 0.742 & 0.754 & \bf0.675 \\\cmidrule(lr){1-5}
        Upernet  & Co-occur~(3.3k) & 0.680 & 0.713 & 0.520 \\
        Data Aug & Co-occur~(3.3k) & \bf0.82 & \bf0.86 & 0.646 \\
     \bottomrule
    \end{tabular}
    %
\caption{Experiments in three class setting on ADE20k}%
    \label{tab:da_seg_rest}
\end{table}

\section{Object removal model}
\label{sec:remmodel}
To remove objects we use the ground truth segmentation masks and dilate them by a small factor (5 in the coco dataset and 7 in the ADE20k dataset). This dilated mask is multiplied with the input image to remove the target object from the image. Then the masked image and the mask is passed to an in-painting network which fills the masked area with a plausible background texture. We use the in-painting architecture and the training procedure proposed in~\cite{shetty2018adversarial}. \Tableref{tab:removalModel} and \ref{tab:removalModelResidual} present the detailed architecture of the in-painting network. We will make the code and pre-trained in-painter models available after the review process.

\section{Analyzing robustness to context}
\label{sec:robus}
In the main paper, we presented our analysis showing that the classification and segmentation models are sensitive to context and their predictions are significantly affected when presented with edited images with context objects removed. In the following sub-sections we present additional visualizations to support these arguments.

\begin{table}[t]
    \footnotesize
    \setlength{\tabcolsep}{1mm}
    \centering
    \begin{tabular}{c}
    \toprule
    Masked Image + mask\\
    Conv 4x4, 64 filters, stride 1\\
    Conv 4x4, 128 filters, stride 2\\
    Conv 4x4, 256 filters, stride 2\\
    Conv 4x4, 512 filters, stride 2\\
    Residual Block, 256 filters\\
    Residual Block, 256 filters\\
    Residual Block, 256 filters\\
    Residual Block, 256 filters\\
    Residual Block, 256 filters\\
    Residual Block, 256 filters\\
    Upsample + Conv 3x3, 256 filters\\
    Upsample + Conv 3x3, 128 filters\\
    Upsample + Conv 3x3, 64 filters\\
    Conv 7x7, 3 filters, stride 1\\
    \bottomrule
    \end{tabular}
    \caption{In-painting model architecture starting with input in the first row to the output layer in the last. Each convolutional layer  is followed by a Instance Norm layer and a Leaky Relu non-linearity with slope 0.1}
    \label{tab:removalModel}
    %
\end{table}

\begin{table}[t]
    \footnotesize
    \setlength{\tabcolsep}{1mm}
    \centering
    \begin{tabular}{c}
    \toprule
    Conv 3x3, n filters, stride 1\\
    Instance Norm\\
    Leaky Relu (slope 0.1)\\
    Conv 3x3, n filters, stride 1\\
    Instance Norm\\
    Leaky Relu (slope 0.1)\\
    \bottomrule
    \end{tabular}
    \caption{Architecture of the residual block with n filters}
    \label{tab:removalModelResidual}
    %
\end{table}

\subsection{Image-level Classification}
\label{sec:robusClas}
\myparagraph{Co-occurrence of objects} An important factor which causes the image-level classification models to use contextual dependencies is the co-occurrence distribution of objects.Many objects in COCO have a strong co-occurrence relation with other objects.
We quantify this using the normalized co-occurrence counts for each object with others given by $$NC(c_i,c_j) = \frac{\text{Count}(c_i\cap c_j)}{\text{Count}(c_i)}$$. This matrix is visualized in \figref{fig:cococooccur}. $NC(c_i,c_j)$ takes value between 0 and 1 and represents the fraction of images containing object $c_i$, which also contains object $c_j$. We can see that, for classes like skateboard, surfboard, tennis racket and handbag, this ratio is very high ($\geq90\%$) with the person class, since these classes often occur with a person holding or riding them. We also see that the matrix is not symmetric. This is because, while the skateboard might occur always with a person, but person class occurs in various contexts without skateboard. However, for some groups of objects like mouse, keyboard and monitor, and spoon, fork, and cup have symmetric co-occurrence relationship.

For many categories, including the cases discussed above, the co-occurrence ratio is very high ($>$ 60\%). This causes problems for object classifiers of these categories, as we see in the analysis presented in the main paper. When a small or difficult to detect object class like mouse or skateboard, frequently co-occurs with a more easy to detect object class like monitor or person, the classifiers tend to overuse the contextual relationship for making their classification decisions instead of visual evidence for the object of interest. This leads to failures when the context is different or the object occurs without context. %
\begin{figure}
\centering
    \includegraphics[width=1.0\linewidth]{classification/coco_cooccurence}
    \caption{Co-occurrence ratio, $N(c_i, c_j)$ of objects on the COCO dataset}
\label{fig:cococooccur}
\end{figure}

\myparagraph{Relation of performance to robustness}
As discussed in section $4.1.2$ in the main paper, we find that many well-performing object classes in terms of average precision~(AP) perform poorly in terms of robustness. To show this we plot the per-class average precision against the worst-case robustness metric $V^{\text{min}}(c_i)$ in \figref{fig:PerclassViol}. We can see for example that classes like mouse, tennis racket, sports ball, baseball bat and book which have high AP~($geq$ 0.6) have poor robustness ~( $V^{\text{min}}(c_i)$ > 0.5). In all these cases, the classifier seems to predominantly use contextual objects to make their predictions and achieve high average precision. But they fail when presented with object-without-context and context-without-object images, usually scoring the context-without-object images higher. This is also seen in further visual examples presented in \figref{fig:classInconsistapp}. Interestingly visually distinct classes like zebra, elephant, giraffe achieve high AP, while also being robust as seen in \figref{fig:PerclassViol} .

\begin{figure}
\centering
    \includegraphics[width=1.0\linewidth]{classification/Average_precision_vs_robustness_Scores_smaller}
    \caption{Comparing class-wise average precision to the \% of violations in changes to context. Many well-performing categories (high mAP), have high percentage of violations, including mouse, tennis racket, keyboard, book, and sink.}
\label{fig:PerclassViol}
\end{figure}

\begin{figure}
    \setlength{\tabcolsep}{0.1mm}
    \noindent\makebox[\linewidth]{
    \scriptsize
    \begin{tabular}{x{1.8cm}cx{5mm}c}
        & \bf\small \textcolor{blue}{Object} without \textcolor{magenta}{Context} && \bf\small \textcolor{magenta}{Context} without \textcolor{blue}{Object}\\[0.5em]
    %
         \includegraphics[valign =c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000278967_surfboard/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000278967_surfboard/f_sec_rperson}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000278967_surfboard/f_prim_rsurfboard}\\
          %
         \bf Regular       & $S(\text{\it surfboard})=\textcolor{red}{2.60}$\textcolor{red}{\Lightning}         &\multirow{2}{*}{\large$\boldmath\geq$ }&$S(\text{\it surfboard)} = \textcolor{red}{3.78}$\textcolor{red}{\Lightning}\\
         \bf Ours & $S(\text{\it surfboard})=\textcolor{PineGreen}{1.81}$    &&$S(\text{\it surfboard}) = \textcolor{PineGreen}{-1.80}$\\[0.5em]

         \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000161720_sink/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000161720_sink/f_sec_rtoilet}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000161720_sink/f_prim_rsink}\\

          %
         \bf Regular       & $S(\text{\it sink}) = \textcolor{red}{-0.74}$\textcolor{red}{\Lightning}       &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it sink}) = \textcolor{red}{0.60}$\textcolor{red}{\Lightning}         \\
         \bf Ours & $S(\text{\it sink}) = \textcolor{PineGreen}{-0.01}$ && $S(\text{\it sink}) = \textcolor{PineGreen}{-0.24}$ \\

         \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000500062_sportsball/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000500062_sportsball/f_sec_rperson}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000500062_sportsball/f_prim_rsportsball}\\
         \bf Regular       & $S(\text{\it ball})=\textcolor{red}{1.96}$\textcolor{red}{\Lightning}    &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it ball}) = \textcolor{red}{2.50}$\textcolor{red}{\Lightning}\\
         \bf Ours & $S(\text{\it ball})=\textcolor{PineGreen}{4.85}$    && $S(\text{\it ball}) = \textcolor{PineGreen}{-1.24}$ \\[0.5em]

          \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000392048_fork/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000392048_fork/f_sec_rcup}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000392048_fork/f_prim_rfork}\\
        %

         \bf Regular       & $S(\text{\it fork}) = \textcolor{red}{-2.12}$\textcolor{red}{\Lightning}      &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it fork}) = \textcolor{red}{-1.22}$\textcolor{red}{\Lightning}      \\
         \bf Ours & $S(\text{\it fork}) = \textcolor{PineGreen}{-3.98}$&& $S(\text{\it fork}) = \textcolor{PineGreen}{-5.06}$\\[0.5em]

          \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000341118_couch/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000341118_couch/f_sec_rtv}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000341118_couch/f_prim_rcouch}\\
        %

         \bf Regular       & $S(\text{\it couch}) = \textcolor{red}{0.63}$\textcolor{red}{\Lightning}      &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it couch}) = \textcolor{red}{2.09}$\textcolor{red}{\Lightning}      \\
         \bf Ours & $S(\text{\it couch}) = \textcolor{PineGreen}{0.73}$&& $S(\text{\it couch}) = \textcolor{PineGreen}{-0.02}$\\[0.5em]

          \includegraphics[valign = c, width=0.9\linewidth]{classification/correlErrors/COCO_val2014_000000309696_car/real} Original
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000309696_car/f_sec_rfirehydrant}&
        & \includegraphics[valign = c, width=0.32\linewidth]{classification/correlErrors/COCO_val2014_000000309696_car/f_prim_rcar}\\

         \bf Regular       & $S(\text{\it car}) = \textcolor{red}{-1.13}$\textcolor{red}{\Lightning}      &\multirow{2}{*}{\large$\boldmath\geq$ }& $S(\text{\it car}) = \textcolor{red}{0.08}$\textcolor{red}{\Lightning}      \\
         \bf Ours & $S(\text{\it car}) = \textcolor{PineGreen}{-0.96}$&& $S(\text{\it car}) = \textcolor{PineGreen}{-2.46}$\\[0.5em]
         %
        %
        %
        %
        %
        %
        %
        %
    \end{tabular}
    }
    \vspace*{-2mm}
    \caption{Context violations by image-level classifier. The primary object is marked with blue box and the context object is marked with magenta. The first column shows the original image, middle shows the image with only object and the third with only the context. We see that the baseline classifier depends heavily on the context and always scores the context only images~(last column) higher than the image with only the primary object (middle column). The data augmented model does better and gets the ordering right. }
    \vspace*{-1em}
    \label{fig:classInconsistapp}
\end{figure}

\subsection{Semantic Segmentation}
\label{sec:robusSegm}
\myparagraph{Visualizing robustness metrics} We compute the robustness metric $AR(c_i,c_j)$, which measures the ratio of instances when segmentation of class $c_i$ is affected by the removal of class $c_j$, in the ADE20k dataset for the Upernet~\cite{xiao2018unified} model.
This is visualized in \figref{fig:segmAffect}.  The y-axis the affected class and the x-axis is the removed object class. We show the rows and the columns which have atleast one entry $> 0.1$, for readability.
We can see that the $AR(c_i,c_j)$ matrix is very sparse, indicating that the segmentation is not affected by all removal, but of only specific classes. As discussed in section $4.2.2$ of the main paper, we can see that classes like road and sidewalk depend on the class car. The sidewalk is also to an extent affected by removal of trees.

To measure the direction of the effect, that is if removal of context harms or improves the segmentation of a class, we visualize the average change in IoU in \figref{fig:segmAffectValue}.  Surprisingly, we find that not all context removal negatively affects the segmentation. Sometimes removing an object helps the model to resolve ambiguities and fix the segmentation of other objects. We can see in \figref{fig:segmAffectValue} that while majority of change is negative, for a few pairs of objects removal positively affects the IoU. For example removing \emph{lamp} class improves the segmentation of ceiling \emph{light}. Similarly, removing \emph{armchair} improves segmentation of \emph{chair} and \emph{sofa} classes, since the ambiguity is resolved.

\begin{figure}[t]
\centering
    \includegraphics[trim={1cm 0 1cm 0},width=1.0\linewidth]{images/segmentation/Affected_class_vs_Affectors.pdf}
    \caption{Visualizing frequency with which classes are affected by removal of other objects. Y-axis are the affected objects and the x-axis shows the removed objects}
\label{fig:segmAffect}
\end{figure}
\begin{figure}[t]
\centering
    \includegraphics[trim={0.5cm 0 1cm 0},height=1.2\linewidth,width=1.0\linewidth]{images/segmentation/Affected_class_vs_Affectors_value.pdf}
    \caption{Visualizing the mean change in the IoU of object segmentation with context object removal. Y-axis are the affected objects and the x-axis shows the removed objects}
\label{fig:segmAffectValue}
\end{figure}

\myparagraph{Ablation on data augmentation} To understand if removal and in-painting is really needed for data augmentation, we conduct two ablation studies which are presented in \Tableref{tab:da_segmentation_abl}. First we train a version of the baseline model where for each training sample we randomly select an object and set its label to 'ignore'. We use the same sampling strategy as \emph{sizebased} data augmentation model. Hence this mimics exactly the training procedure in DA (sizebased), except without actually removing the object. Comparing the results of this model (No removal (sizebased)) to the data augmented version, we see that removing the object is necessary and simply ignoring the label leads to a severe drop in performance (0.354 vs 0.379). Similarly, in the second experiment we train a model with the object removed but without in-painting. In this case we can see from \tableref{tab:da_segmentation_abl} that, having in-painter during augmentation is slightly better than the model without ~(0.379 vs 0.375).

Further visual examples of the sensitivity of the baseline Upernet~\cite{xiao2018unified} model to contextual changes and the robustness provided by data-augmentation is seen in \figref{fig:InconsistentSegApp}.

\myparagraph{Confirming the source of sensitivity} Finally we conduct an additional experiment to verify that the volatility we see in the output of the segmentation models on edited images are due to removal of context objects and not due to editing artifacts. To test this we observe the predictions of the model on three version of the input image. First is the original image. Second is the image with a context object removed. Finally the last image is the false edited image created by masking and in-painting the input image with the same mask as the context object, except with a horizontal flip. Thus the context object is not removed in the false edited image, but a similar shape and size region is removed and in-painted in a different part of the input image. All three images are fed to the segmentation models and the output is shown in \figref{fig:TestingSource}. We can see that the Upernet model output is virtually identical on the original image and the false edited image(third row). However the segmentation on the edited image with context object removed is significantly different (second row). This indicates that the segmentation models are not affected by the editing artifacts but by the removal of the context objects as claimed in the paper.

\begin{table}[t]
    \newcommand*{\mrowt}[1]{\multirow{2}{*}{#1}}
    \footnotesize
    \setlength{\tabcolsep}{1mm}
    \centering
    \begin{tabular}{lc cc}
    \toprule
    \multirow{2}{*}{Model} & Removed &  \multicolumn{2}{c}{ADE20k}\\\cmidrule(lr){3-4}
              & Pixels & mIoU & Acc\\\cmidrule(lr){1-2}\cmidrule(lr){3-4}
        Upernet\cite{xiao2018unified}      & - & 0.377 & 78.31 \\\cmidrule(lr){1-2}\cmidrule{3-4}
        No removal (sizebased) & Ignore & 0.354 & 77.45 \\
        No inpainter (sizebased) & Ignore & 0.375 & 78.25 \\%\cmidrule(lr){1-2}\cmidrule{3-4}
        %
        DA (sizebased)     & Ignore   & 0.379 & 78.31 \\
        %
        %
        %
     \bottomrule
    \end{tabular}
    \vspace*{-1mm}
    \caption{Data augmentation results on ADE20k dataset}
    \vspace*{-5mm}
    \label{tab:da_segmentation_abl}
    %
\end{table}
\begin{figure*}
    \setlength{\tabcolsep}{1pt}
    \noindent\makebox[\textwidth]{
    \scriptsize
    \begin{tabular}{cccccc}
    %
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/pred_sec}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/pred_sec}\\
          original $(\mathcal{I})$ & Upernet   & Ours & original $(\mathcal{I})$ & Upernet & Ours\\
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001798_sidewalk/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000678_road/pred_sec_edited}\\
          $\mathcal{I} - car$ & Upernet & Ours & $\mathcal{I} - car$ & Upernet & Ours\\[0.3em]
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/pred_sec}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/pred_sec}\\
          original $(\mathcal{I})$ & Upernet  & Ours & original $(\mathcal{I})$ & Upernet & Ours\\
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001286_bench/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000491_house/pred_sec_edited}\\
          $\mathcal{I} - column$ & Upernet  & Ours & $\mathcal{I} - boat$ & Upernet & Ours\\
    \end{tabular}
        }
    \vspace*{-0.5em}
    \caption{Examples of segmentation failures due to removal of a single context object. We see the segmentation of road, bench and house affected significantly when context objects like car, columns and boat is removed~(comparing odd and even rows). Model trained with proposed data-augmentation is more robust to these changes.}
    \vspace*{-1em}
    \label{fig:InconsistentSegApp}
\end{figure*}

\begin{figure*}
    \setlength{\tabcolsep}{1pt}
    \noindent\makebox[\textwidth]{
    \scriptsize
    \begin{tabular}{cccccc}
    %
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/pred_sec}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/original}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_prim}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_sec}\\
          original $(\mathcal{I})$ & Upernet   & Ours & original $(\mathcal{I})$ & Upernet & Ours\\[0.3em]
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00001804_road/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/correlErrors/ADE_val_00000819_sidewalk/pred_sec_edited}\\
          $\mathcal{I} - car$ & Upernet & Ours & $\mathcal{I} - sign$ & Upernet & Ours\\[0.3em]
          \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00001804_sidewalk_car/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00001804_sidewalk_car/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00001804_sidewalk_car/pred_sec_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00000819_sidewalk_signboard/original_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00000819_sidewalk_signboard/pred_prim_edited}
        & \includegraphics[valign = c, height=0.06\linewidth, width=0.16\linewidth]{segmentation/dummyErrors/ADE_val_00000819_sidewalk_signboard/pred_sec_edited}\\
          $\mathcal{I} - car$ (flipped mask) & Upernet & Ours & $\mathcal{I} - sign$ (flipped mask) & Upernet & Ours\\[0.3em]
    \end{tabular}
        }
    \vspace*{-0.5em}
    \caption{Experiment to verify that the volatality of the segmentation output is due to object removal and not due to editing artifacts. First row shows original images and the segmenations produced for them. Second row shows the edited images with an object removed and the segmentation output for them. Here we can see the segmentation output of Upernet significantly affected by the removal of car and sign. Final, row shows the original image edited with the same object mask as the second row, but horizontally flipped. Thus the object is not removed, but a different part of the image is edited with the same mask. We can see here that the segmentation is not affected at all by this edit and is very similar to the segmentation produced by the original image.}
    \vspace*{-1em}
    \label{fig:TestingSource}
\end{figure*}

redpajama_set_name: RedPajamaArXiv


text:
Festival Hub
LA Venues
Best Shows This Week
Can't Miss Concerts
About Pass The Aux
Coachella 2018 set times: What are the biggest conflicts?
April 11, 2018 April 11, 2018 Mark Ortega
As you may have seen, the Coachella 2018 set times dropped Tuesday night and as per usual, are full of difficult conflicts on who to go see. First, take a look at the full set times below, and then proceed to the bottom for some of our advice on who to check out.
FRIDAY CONFLICTS
Kali Uchis at The Novo photo by Thane Fernandes
Kali Uchis (5:00 to 5:45 PM, Outdoor Theatre) vs. Greta Van Fleet (4:45 to 5:35)
Clearly, if you stick to one genre then this isn't much of a choice to you. If you favor pop and R&B you'll head over to see Colombian-born Kali Uchis, if you're a guitars guy then you'll see Greta Van Fleet. But if you like to dabble in a variety of genres, this is a chance to see one of two acts that will likely be much higher up the bill the next time they play Coachella. Both are playing a handful of other fests, but Kali Uchis will be back in LA at FYF Fest in July. So if you're headed to Expo Park this summer, catch her then and go see the band that everyone is comparing to Led Zeppelin. If you're a big fan of surprise guests, Kali Uchis' new album Isolation is excellent and chock full of collaborators, including Coachella 2018 acts Tyler the Creator, Jorja Smith, and Steve Lacy of The Internet (who appeared at her show at The Novo on Tuesday night).
Kygo (8:30 to 9:25 PM, Main) vs. St. Vincent (8:55 to 9:50, Outdoor) vs. DREAMS (8:40 to 9:30, Mojave)
Of these acts, only St. Vincent is playing more than a handful of festival dates. Kygo almost certainly will drop a massive tour after Coachella that includes a major LA date (my money is on the LA Convention Center). DREAMS is one of those boom-or-bust sets. It's the collaboration of Empire of the Sun's Luke Steele and Silverchair's Daniel Johns. They only have one song out. Their only US dates are Coachella, and they have just one Aussie date scheduled otherwise. They only have one (very EOTS-sounding) song out. If that's your bag, I recommend starting out with some of their set and depending on how you like it, drifting over to Kygo or St. Vincent for the second half of one of those sets.
SATURDAY CONFLICTS
Hundred Waters at El Rey Theatre, photo by Liat Halmos
alt-J (10:10 PM to 11:10, Outdoor) vs. Jungle (9:50 to 10:40, Mojave) vs. Hundred Waters (10:05 to 11:05, Sonora) vs. Tom Misch (10:10 to 10:55, Gobi)
This is just an absolute clusterfuck and probably the worst conflict of the entire weekend, no matter your musical inclination. I'm a little bummed that Hundred Waters isn't getting a slot in the Sahara tent considering their awesome production at their El Rey show seems like they could make great use of the improved Sahara's production. They'll be in the rock-driven Sonora Tent which is a slight letdown. Tom Misch just put out one of the hottest albums this year and could draw on a couple collaborations for his set. Both Misch and Hundred Waters were among my 10 predicted breakthrough undercard acts. But Jungle's soulful sound is where the ultimate dance party will take place and 10 PM is about as good of a time as any to move your ass — so that's my recommendation. Good news if you're an alt-J fan though, they close out the Outdoor Theatre and it should be easy to get close as people start getting ready for Beyonce on the main stage.
David Byrne (7:05 to 8:05 PM, Outdoor) vs. Tash Sultana (7:20 to 8:10, Mojave)
David Byrne is obviously an icon, but Tash Sultana is one of the most promising musicians to emerge in recent years and I have the sense that Coachella has the chance to be a massive breakthrough performance for the multi-instrumentalist and singer. If you're an LA local, Byrne will play the Shrine in August. I've seen Tash play The Echo and Fonda Theatre to sold-out crowds the last year-and-change, and she's only going to continue to grow. Based on Setlist.fm info, Byrne has usually front-loaded a lot of Talking Heads material, playing classics like "This Must Be The Place" and "Once in a Lifetime" in the first half-dozen or so songs. I recomment checking out the first half of Byrne's set then sprinting over to catch the last 15 or 20 minutes of Tash.
SUNDAY CONFLICTS
Miguel at KCRW Sound in Focus 2017 photo by Stephanie Varela Rheingold
ODESZA (8:40 to 9:40 PM, Main) vs. Miguel (8:30 to 9:20 PM, Outdoor)
Let's be real — ODESZA might end up with some of the sickest production as one of the few EDM acts playing the main stage at night this year. But Miguel's tunes seem like they are better suited for the wind-down nature of Sunday night after you've run out of your party favors and are starting to feel the pain of everything you did to your body this weekend. Also if you've been courting a honey all weekend, what better time to get grinded on or do some grinding than when Miguel is crooning?
Cardi B (6:00 to 6:35 PM, Main) vs. Jessie Ware (5:40 to 6:30, Outdoor)
Cardi B reportedly spent $300K of her own money on her stage setup. This is interesting considering she is playing just a 35-minute set. Will her show be bells and whistles with no punch as she just revealed on SNL that she's pregnant? I've been playing her new album Invasion of Privacy and really digging it, but early videos of her performing live have been lackluster. Jessie Ware is a proven talented live performer with a bunch of soothing melodies behind those fantastic vocals. The answer to what you should do is this — are you trying to turn up or turn down?
10 Coachella 2018 Undercard Acts Likely To Break Through
Coachella 2018 sees an uptick in number of women on lineup
Read more from Pass The Aux
Best Shows This Week: April 9 to April 15, 2018
Goldenvoice Presents April: Full Coachella 2018 side show list revealed
Austin City Limits 2018 lineup: Paul McCartney, Childish Gambino, Arctic Monkeys
Best Shows This Week: April 16 to April 22, 2018
Features Festivals Mainbar Featured
alt-J Cardi B Coachella Coachella 2018 Coachella 2018 set times Coachella set times David Byrne DREAMS Greta Van Fleet Hundred Waters Jessie Ware Jungle Kali Uchis Kygo Miguel ODESZA set times St. Vincent Tash Sultana Tom Misch
Mark Ortega
Kali Uchis, Cuco justify buzz with sold-out show at The Novo
Soccer Mommy makes LA headline debut at The Echo a memorable one
ALBUM REVIEW: Protomartyr provide stunning listen on 'Ultimate Success Today'
Alfie Templeman poised to be the groovy breakout star 2020 needs
ALBUM REVIEW: The Beths deliver hooks that linger on 'Jump Rope Gazers'
Rumor has it: New Adele album in the works
ALBUM REVIEW: Phoebe Bridgers exceeds expectations, avoids sophomore jinx on 'Punisher'
Pass The Aux Summer '18 Playlist
5 Things To Do Near The Fonda
Let Your Ears Decide
The Scenestar
Ellenwood-EP

redpajama_set_name: RedPajamaCommonCrawl


Join us in August for the Virginia Craft Brewers Fest in Charlottesville, VA. The Fest will be hosted by Three Notch'd Brewing Company and is the only sanctioned event of the Virginia Craft Brewers Guild. Thousands of craft beer lovers from Virginia and surrounding states enjoy chatting with the brewers responsible for the tasty libations.


Sean Wotherspoon x Nike Air Max 97 of the most hottest items in Nike Air Max 97. This new edition is designed in different colorways. Other products also hot sale online.Sean Wotherspoon x Nike Air Max 97 with comfortable high memory polyurethane insole can give you a wonderful experience. More surprise you can get along the way in cheap Nike Air Max 97 store.


Video shows the moment that Eric Arnold opened fire on Sacramento police officers.
Sacramento, CA - The Sacramento Police Department has released video showing a gunfight between officers and a double homicide suspect.
The incident happened September 7 when officers stopped 41-year-old Eric Arnold at 27th Ave and Franklin Boulevard, according to Fox40.
Arnold was wanted for the murder of a mother and daughter a week earlier.
After the officers stopped Arnold, they started giving him commands. That's when Arnold exited his vehicle and opened fire on the officers.
Officers returned fire, filling Arnold with bullet holes until he was no longer a threat. Arnold died on scene.
Officer Victor Wolfe was shot in the leg and Officer Tim Martin was shot in his ballistic vest.
Officer Martin spoke with FOX40 about the memory loss that comes after combat.
"Immediately afterwards, I actually questioned whether or not I had returned fire, even when someone asked me if I had, I said that 'yes I did.' But I could no longer actually remember, for a while, the actual action of returning fire," Officer Martin said.
"You don't want to see people get killed. Very unhappy for all of us. I wish we could stop it before it happened. But unfortunately we're responding to what happens. Can't stop it before it happens," Officer Martin said.
Sacramento police department is now looking into bullet resistant doors to provide officers with more cover in future shootings.
You can see the video of the gunfight here. There are multiple camera angles - the first shows the suspect and the second shows the officers. WARNING: Graphic Content.
Doors and windows should be bullet proof!
I am so thankful that the body armor works to protect the lives of our men and women in blue. Praying the officers will recover quickly. Relieved that the perp is now facing the eternal punishment.
The dead dude was using a .380???
Got exactly what he asked for!
"grey matter" someone got in a head shot. Awesome job officers.


text:
Exact Searches
Manage Your Holds
Displaying 1 of 1
The Cabin at the End of the World
Tremblay, Paul
Ebook, Electronic Resources, Book
Publisher, Date:
[S.I.] : HarperCollins, 2018.
"A tremendous book―thought-provoking and terrifying, with tension that winds up like a chain. The Cabin at the End of the World is Tremblay's personal best. It's that good." — Stephen King The Bram Stoker Award-winning author of A Head Full of Ghosts adds an inventive twist to the home invasion horror story in a heart-palpitating novel of psychological suspense that recalls Stephen King's Misery, Ruth Ware's In a Dark, Dark Wood, and Jack Ketchum's cult hit The Girl Next Door. Seven-year-old Wen and her parents, Eric and Andrew, are vacationing at a remote cabin on a quiet New Hampshire lake. Their closest neighbors are more than two miles in either direction along a rutted dirt road. One afternoon, as Wen catches grasshoppers in the front yard, a stranger unexpectedly appears in the driveway. Leonard is the largest man Wen has ever seen but he is young, friendly, and he wins her over almost instantly. Leonard and Wen talk and play until Leonard abruptly apologizes and tells Wen, "None of what's going to happen is your fault". Three more strangers then arrive at the cabin carrying unidentifiable, menacing objects. As Wen sprints inside to warn her parents, Leonard calls out: "Your dads won't want to let us in, Wen. But they have to. We need your help to save the world." Thus begins an unbearably tense, gripping tale of paranoia, sacrifice, apocalypse, and survival that escalates to a shattering conclusion, one in which the fate of a loving family and quite possibly all of humanity are entwined. The Cabin at the End of the World is a masterpiece of terror and suspense from the fantastically fertile imagination of Paul Tremblay.
Other Author:
Overdrive Inc.
Suggestions and more
Large Cover Image
Publishers Weekly Review
The apocalypse begins with a home invasion in this tripwire-taut horror thriller. Eric and Andrew are vacationing with their seven-year-old daughter, Wen, at remote Gaudet Lake in New Hampshire when their cabin is invaded by a quartet of weapons-wielding strangers, each of whom has been driven there by a shared vision: that the world will end unless one member of this family sacrifices another. That sets the stage for an excruciatingly tense standoff between them and their prisoners as they try to outmaneuver one other. Tremblay (Disappearance at Devil's Rock) skillfully seeds his tale with uncertainties, including news reports of portentous world catastrophes, that suggest the invaders' vision is genuine, and he introduces enough doubt into the beliefs and behaviors of all the parties to keep them and the reader off-balance. His profoundly unsettling novel invites readers to ask themselves whether, when faced with the unbelievable, they would do the unthinkable to prevent it. (June) © Copyright PWxyz, LLC. All rights reserved.

 "A tremendous book―thought-provoking and terrifying, with tension that winds up like a chain. The Cabin at the End of the World is Tremblay's personal best. It's that good." -- Stephen King

The Bram Stoker Award-winning author of A Head Full of Ghosts adds an inventive twist to the home invasion horror story in a heart-palpitating novel of psychological suspense that recalls Stephen King's Misery, Ruth Ware's In a Dark, Dark Wood, and Jack Ketchum's cult hit The Girl Next Door.

Seven-year-old Wen and her parents, Eric and Andrew, are vacationing at a remote cabin on a quiet New Hampshire lake. Their closest neighbors are more than two miles in either direction along a rutted dirt road.

One afternoon, as Wen catches grasshoppers in the front yard, a stranger unexpectedly appears in the driveway. Leonard is the largest man Wen has ever seen but he is young, friendly, and he wins her over almost instantly. Leonard and Wen talk and play until Leonard abruptly apologizes and tells Wen, "None of what's going to happen is your fault". Three more strangers then arrive at the cabin carrying unidentifiable, menacing objects. As Wen sprints inside to warn her parents, Leonard calls out: "Your dads won't want to let us in, Wen. But they have to. We need your help to save the world."

Thus begins an unbearably tense, gripping tale of paranoia, sacrifice, apocalypse, and survival that escalates to a shattering conclusion, one in which the fate of a loving family and quite possibly all of humanity are entwined. The Cabin at the End of the World is a masterpiece of terror and suspense from the fantastically fertile imagination of Paul Tremblay.

"Read Paul Tremblay's new novel, The Cabin at the End of the World, and you might not sleep for a week. Longer. It will shape your nightmares for months - that's pretty much guaranteed." -- NPR

"Gripping, horrifying, and mesmerizing." -- GQ

"A tour-de-force of psychological and religious horror." -- BN.com

"A blinding tale of survival and sacrifice." -- Kirkus Reviews

"Tremblay has a real winner here." -- Tor.com

Librarian's View


text:
Eller's Entrepreneurship Expertise Expands to San Francisco with Impact Hub Partnership
The Bay Area has long been known as the epicenter of innovation and start-up culture in the United States—it is the place where new things are born.
The Eller College of Management at the University of Arizona has taken note.
In response to an increasing demand on aspiring and established entrepreneurs for business acumen to successfully steward their ventures, Eller's McGuire Center for Entrepreneurship—consistently ranked in the top 10 for entrepreneurship programs by U.S. News & World Report—is now providing courses and training at Impact Hub San Francisco.
"Our new partnership with Impact Hub will create more expertly trained entrepreneurs in the Bay Area through the McGuire Center's specially tailored graduate-level courses," says Paulo Goes, Eller College dean and Halle Chair in Leadership. "The courses will be focused around social innovation to speak directly to Impact Hub's mission-driven co-working members and community to support them as change makers with global impact."
To celebrate this pioneering partnership, in March 2018 the Eller College and Impact Hub held a moderated panel of speakers to discuss disruptive innovation. Panelists included:
Tim Freundlich, CEO and Founder, Impact Hub
Paulo Goes, Dean, Eller College of Management
Sara Olsen, Founder and CEO, SVT Group
Matt Van Horn '06 BSBA (Marketing and Entrepreneurship), Co-Founder, Lyft; Co-founder and CEO, June
Michelle Vock '13 BSBA (Marketing and MIS), Team Manager, Andela
"We are pleased to introduce our McGuire curriculum, which offers extremely relevant learning for innovators and entrepreneurs," says Remy Arteaga, executive director of the McGuire Center and panelist moderator. "There is an ever-expanding need for entrepreneurs—even in an environment like the Bay Area—to fully understand the business side of innovation and entrepreneurship, no matter where they are in the start-up cycle."
McGuire | Impact Hub Launch Event
Part of a growing network across the world, Impact Hub San Francisco is a community of entrepreneurs, activists, creatives and professionals taking action to drive positive social, economic and environmental change.
"Our vibrant community of hundreds of impact innovators is thrilled to provide a home and supportive ecosystem to surround this ground breaking educational program, says Tim Nichols, managing director of Impact Hub San Francisco. "The McGuire Center's curriculum will feed into and off of our rich landscape in a unique, high-value way."
For more information, visit the Impact Hub San Francisco.
Photos courtesy of Sumit Kohli.
mcguire center
McGuire Center for Entrepreneurship
paulo goes
remy arteaga


text:
#pragma once
#include "paddle/fluid/framework/op_registry.h"
#include "paddle/fluid/operators/math/math_function.h"

namespace paddle {
namespace operators {

template <typename DeviceContext, typename T>
class FillConstantBatchSizeLikeOpKernel : public framework::OpKernel<T> {
 public:
  void Compute(const framework::ExecutionContext& ctx) const override {
    auto* out = ctx.Output<framework::Tensor>("Out");
    auto* in = ctx.Input<framework::LoDTensor>("Input");
    if (in->lod().size() && ctx.Attr<int>("input_dim_idx") == 0) {
      // set the correct batch size for the LoDTensor.
      auto odims = out->dims();
      int output_dim_idx = ctx.Attr<int>("output_dim_idx");
      odims[output_dim_idx] = static_cast<int>(in->lod().back().size()) - 1;
      out->mutable_data<T>(odims, ctx.GetPlace());
    }
    out->mutable_data<T>(ctx.GetPlace());
    auto value = ctx.Attr<float>("value");

    math::SetConstant<DeviceContext, T> setter;
    setter(ctx.template device_context<DeviceContext>(), out,
           static_cast<T>(value));
  }
};

}  // namespace operators
}  // namespace paddle

redpajama_set_name: RedPajamaGithub


Last Update: 15 October 15; High Resolution: Yes; Compatible OS Versions: Windows XP, Windows Vista, Windows 7, Windows 8 Desktop, Windows 8 Metro; Application Runtime: Native.
Your customers connect with Facebook to unlock your promotion, refer friends and join your mailchimp mailing list with one click!
Collect subscribers INSIDE Facebook newsfeed!
Run a promotion and collect Facebook verified emails that users check daily (no fake or "throw away" emails). Perfect for marketers looking to grow their subscriber base.
Socialize your in-store shopping baskets!
Shoppers scan the QR Code on the basket ad to claim your promotion so you can collect verified emails and page likes. Then send shoppers regular deals to increase store visits.
Turn the public into your referral team!
Users scan the QR code on the billboard and connect with Facebook to unlock your promotion, subscribe and share with friends so you can schedule future offers using Mailchimp.
While people are drinking, they can scan the QR Code to claim your promotion, share with friends and subscribe to your mailing list with one click so you can send deals or events.
Put your promotion directly in their hands!
This provide exposure to not only the coffee drinker but also everyone else those people encounter at the coffee shop, street, work, and everywhere else they drink and carry it.
Attract, engage and convert customers in-store!
Studies show that people love interacting with kiosks, give customers a reason to keep returning to your store by offering regular promotions to drive foot traffic.
WhatsApp Marketing Suite; Twitter Automation - Marketing Suite.
Keywords: coupon, email, facebook, facebook like, facebook share, form, kiosk, mailchimp, mobile page, promotions, qr code, Social Marketing, subscribe, viral marketing, voucher.
Premium CodeCanyon Social Promotion Suite 13236155 WordPress Themes, Website Templates, Mobile Themes, Flash Templates & Scripts from creative professional designers. Full details of CodeCanyon Social Promotion Suite 13236155 for digital design and education. CodeCanyon Social Promotion Suite 13236155 desigen style information or anything related.


The past few days have been the hardest to take in. Yesterday we met with our team in Phoenix and learned that Piper's Restrictive Atrial Septum is now fully Intact. This means that now there is no mixing of blood from the right and left sides of her heart and all of the blood is being pushed back into her lungs. Her lung pressure is very high and is now putting stress on her brain. Originally we thought that her first surgery would be immediately following birth but because of the current state of her heart we will be heading to Texas Children's Hospital tomorrow to be potentially having her first in utero surgery next week. Saying I'm scared is an understatement. If this doesn't work her chances of survival after birth are 10-20%. There's also the chance that she won't survive the surgery. We are putting all of our eggs in God's basket. As hard and uncomfortable as asking for donations is, we have no choice. We still haven't started jobs and our house in Montana has yet to close. If nothing else, please please share and pray.
We are currently on our way to Houston any recommendations for a church?
If you can give please give but if not please share and pray for us above all.


President Barack Obama recently delivered the commencement address at Morehouse College and talked personally about fatherhood and family.
...I sure wish I had had a father who was not only present, but involved. Didn't know my dad. And so my whole life, I've tried to be for Michelle and my girls what my father was not for my mother and me. I want to break that cycle where a father is not at home -- where a father is not helping to raise that son or daughter. I want to be a better father, a better husband, a better man.
It's hard work that demands your constant attention and frequent sacrifice. And I promise you, Michelle will tell you I'm not perfect. She's got a long list of my imperfections. Even now, I'm still practicing, I'm still learning, still getting corrected in terms of how to be a fine husband and a good father. But I will tell you this: Everything else is unfulfilled if we fail at family, if we fail at that responsibility.
Watch President Obama's full address and read President Obama's commencement speech transcript.


A person with knowledge of the decision tells The Associated Press the eight-team spring football league is not folding, but games will not be played this weekend.
According to Rovell, the operations stoppage comes not due financial reasons. But having already lost $70 million, Dundon - who owns the NHL's Carolina Hurricanes - reportedly balked at adding an additional $20 million to keep the AAF going through the end of the season. Alliance CEO and founder Charlie Ebersol used his past knowledge working closely with Vince McMahon's XFL to develop a league built on quality football, an interactive mobile app, and high-profile head coaches and football minds like co-founder Bill Polian to jump-start enthusiasm.
"I sincerely regret that many that believed in this project will see their hopes and efforts unrewarded", Polian said in his statement Tuesday.
"If you're an AAF player and the league does dissolve, the last check you got will be the last one that you get", Manziel continued on Twitter. In the AAF's case, that's more than 400 players and almost 100 coaches - plus front-office staff, medical staff, equipment staff and other support workers.
Previously it was reported that former National Football League quarterback Colin Kaepernick demanded $20 million to play in the AAF, which ultimately passed on the controversial civil rights figure. They received game checks for the eighth week of the season, something Pertile said he appreciated.
The year-'round diehards who can never get enough are more than serviced in the off-season by the endless news flow - on draft prospects, veteran free-agency rumblings, preseason predictions, new coaches, new schemes, new rules, etc.
"That guy may not feel like he's completely lived his dream".
The Alliance was able to set up a ideal schedule which fulfills the void of no football between the Super Bowl and spring practices for college programs but it has struggled to stay afloat financially.
Just last week, Dundon told USA TODAY Sports that the NFLPA would not allow the league to use practice squad players from NFL rosters.
"While Dundon is said to have liked the tech component of the business and owns the entire company, sources say it would not be legal for him to shut down the league in order to strip the assets". "To capture that affinity was really important". There, former Cardinals tight end Gerald Christian scored the first touchdown in franchise history.
The AAF folded eight games into a 10-game regular season, and before their first championship game, scheduled for April 27 in Frisco, Texas.


The "Kinder ins Glück" ("Joy to the Children") campaign was initiated in 2012 by the Rotary Club Passau-Dreiflüssestadt with the objective of offering needy children and young people in the Passau region the support necessary for them to be able to look hopefully towards a happy future once again. The campaign does not focus on a single short-term fate, but rather on a continuing partnership on many levels of social life in our region.
Passavia is supporting the project by sponsoring the website and the flyer.