\subsection{Preliminaries}

Consider an undirected graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$ composed of a node set $\mathcal{V}$ of cardinality $|\mathcal{V}|=N$, and an edge set $\mathcal{E}$ connecting nodes.
\textit{Graph signal} refers to data/features associated with the nodes of $\mathcal{G}$, denoted by $\mathbf{X} \in \mathbb{R}^{N \times C}$ with $i$th row representing the $C$-dimensional graph signal at the $i$th node of $\mathcal{V}$.

To characterize the similarities (and thus the graph structure) among node signals, an \textit{adjacency matrix} $\mathbf{A}$ is defined on $\mathcal{G}$, which is a real-valued symmetric $N \times N$ matrix with $a_{i,j}$ as the weight assigned to the edge $(i,j)$ connecting nodes $i$ and $j$.
Formally, the adjacency matrix is constructed from graph signals as follows,
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    \mathbf{A} = f(\mathbf{X}),
    \label{eq:A_X}
\end{equation}
where $f(\cdot)$ is a linear or non-linear function applied to each pair of nodes to get the pair-wise similarity. For example, a widely adopted function is to nonlinearly construct a $k$-nearest-neighbor ($k$NN) graph from node features \cite{wang2019dynamic,zhang2018graph}.

\begin{figure*}[t]
    \centering
    \subfigure[Original model]{
    \includegraphics[width=0.18\textwidth]{aero.pdf}
    }
    \subfigure[Global+Isotropic]{
    \includegraphics[width=0.18\textwidth]{global_same.pdf}
    }
    \subfigure[Global+Anisotropic]{
    \includegraphics[width=0.18\textwidth]{global_different.pdf}
    }
    \subfigure[Local+Isotropic]{
    \includegraphics[width=0.18\textwidth]{local_same.pdf}
    }
    \subfigure[Local+Anisotropic]{
    \includegraphics[width=0.18\textwidth]{local_different.pdf}
    }
    \caption{\textbf{Demonstration of different sampling (Global or Local) and node-wise \textit{translation} (Isotropic or Anisotropic) methods on 3D point clouds.} Red and blue points represent transformed and non-transformed points, respectively. Note that we adopt the wing as a sampled local point set for clear visualization.}
    \label{fig:sampling_transform}
    \vspace{-0.2in}
\end{figure*}

\vspace{-0.05in}
\subsection{Graph Signal Transformation}

Unlike Euclidean data like images, graph signals are irregularly sampled, whose transformations are thus nontrivial to define.
To this end, we define a graph transformation on the signals $\mathbf{X}$ as node-wise \textit{filtering} on $\mathbf{X}$.

Formally, suppose we sample a graph transformation $\mathbf{t}$ from a transformation distribution $\mathcal{T}_g$, \textit{i.e.}, $\mathbf{t} \sim \mathcal{T}_g$.
Applying the transformation to graph signals $\mathbf{X}$ that are sampled from data distribution $\mathcal{X}_g$, \textit{i.e.}, $\mathbf{X} \sim \mathcal{X}_g$, leads to the filtered graph signal
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    \tilde{\mathbf{X}}=\mathbf{t}(\mathbf{X}).
    \label{eq:graph_signal_t}
\end{equation}

The filter $\mathbf t$ is applied to each node individually, which can be either node-invariant or node-variant. In other words, the transformation of each node signal associated with $\mathbf t$  can be different from each other. For example, for a  translation $\mathbf t$, a distinctive translation can be applied to each node. We will call the graph transformation isotropic (anisotropic) if it is node-invariant (variant).

Consequently, the adjacency matrix of the transformed graph signal $\tilde{\mathbf{X}}$ equivaries according to \eqref{eq:A_X}:
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    \tilde{\mathbf{A}} = f(\tilde{\mathbf{X}}) = f(\mathbf{t}(\mathbf{X})),
\end{equation}
which transforms the \textit{graph structures}, as edge weights are also filtered by $\mathbf{t}(\cdot)$.

Under this definition, there exist a wide spectrum of graph signal transformations.
Examples include affine transformations (translation, rotation and shearing) on the location of nodes (\textit{e.g.}, 3D coordinates in point clouds), and graph filters such as low-pass filtering on graph signals by the adjacency matrix \cite{sandryhaila2013discrete}.

\vspace{-0.05in}
\subsection{Node-wise Graph Signal Transformation}

As aforementioned, in this paper, we focus on \textit{node-wise} graph signal transformation, {\it i.e.}, each node has its own transformation, either isotropically or anisotropically.
We seek to learn graph representations through the node-wise transformations by revealing how different parts of graph structures would change globally and locally.

Specifically, here are two distinct advantages.
\begin{itemize}
    \item The node-wise transformations allow us to use node sampling to study different parts of graphs under various transformations.
    \item By decoding the node-wise transformations, we will be able to learn the representations of individual nodes. Moreover, these node-wise representations will not only capture the local graph structures under these transformations, but also contain global information about the graph when these nodes are sampled into different groups over iterations during training.
\end{itemize}

Next, we discuss the formulation of learning graph transformation equivariant representations by decoding the node-wise transformations via a graph-convolutional encoder and decoder.

\subsection{The Formulation}

Given a pair of graph signal and adjacency matrix $(\mathbf{X}, \mathbf{A})$, and a pair of {\em transformed} graph signal and adjacency matrix $(\tilde{\mathbf{X}},\tilde{\mathbf{A}})$ by a node-wise graph transformation $\mathbf{t}$, a function $E(\cdot)$ is \textit{transformation equivariant} if it satisfies
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    E(\tilde{\mathbf{X}}, \tilde{\mathbf{A}}) = E\left(\mathbf{t}(\mathbf{X}), f\left(\mathbf{t}(\mathbf{X})\right)\right) = \rho(\mathbf{t})\left[E(\mathbf{X}, \mathbf{A})\right],
    \label{eq:ter}
\end{equation}
where $\rho(\mathbf{t})$ is a homomorphism of transformation $\mathbf{t}$ in the representation space.

Our goal is to learn a function $E(\cdot)$, which extracts equivariant representations of graph signals $\mathbf{X}$.
For this purpose, we employ an encoder-decoder network: we learn a graph encoder $E: (\mathbf{X},\mathbf{A}) \mapsto E(\mathbf{X},\mathbf{A})$, which encodes the feature representations of individual nodes from the graph.
To ensure the transformation equivariance of representations, we train a decoder $D: \left(E(\mathbf{X},\mathbf{A}), E(\tilde{\mathbf{X}},\tilde{\mathbf{A}})\right) \mapsto \hat{\mathbf{t}}$ to estimate the node-wise transformation $\hat{\mathbf{t}}$ from the representations of the original and transformed graph signals.
Hence, we cast the learning problem of transformation equivariant representations as the joint training of the representation encoder $E$ and the transformation decoder $D$. It has been proved that the learned representations in this way satisfy the generalized transformation equivariance without relying on a linear representation of graph structures \cite{qi2019avt}.

Further, we sample a subset of nodes $\mathbf S$ following a sampling distribution $\mathcal S_g$ from the original graph signal $\mathbf X$, locally or globally in order to reveal graph structures at various scales.
Node-wise transformations are then performed on the subset $\mathbf S$ isotropically or anisotropically, as demonstrated in Fig.~\ref{fig:sampling_transform}.
In order to predict the node-wise transformation $\mathbf{t}$, we choose a loss function $\ell_\mathbf S(\mathbf{t}, \hat{\mathbf{t}})$ that quantifies the distance between $\mathbf{t}$ and its estimate $\hat{\mathbf{t}}$ in terms of their parameters.
Then the entire network is trained end-to-end by minimizing the loss
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    \min_{E,D} \;
    \underset{\mathbf S \sim \mathcal S_g}{\mathbb E}~\underset{\mathbf X \sim \mathcal X_g}{\underset{\mathbf{t} \sim \mathcal{T}_g}{\mathbb E}}
     ~ \ell_\mathbf S(\mathbf{t}, \hat{\mathbf{t}}),
    \label{eq:loss}
\end{equation}
where the expectation $\mathbb{E}$ is taken over the sampled graph signals and transformations, and the loss is taken over the (locally or globally) sampled subset $\mathbf S$ of nodes in each iteration of training.

In \eqref{eq:loss}, the node-wise transformation $\hat{\mathbf{t}}$ is estimated from the decoder
\begin{equation}
    \setlength{\abovedisplayskip}{3pt}
    \setlength{\belowdisplayskip}{3pt}
    \hat{\mathbf{t}} = D\left(E(\mathbf{X},\mathbf{A}), E(\tilde{\mathbf{X}},\tilde{\mathbf{A}})\right).
\end{equation}
Thus, we update the parameters in encoder $E$ and decoder $D$ iteratively by backward propagation of the loss.

\begin{figure}[t]
    \centering
    \includegraphics[width=8.3cm]{framework.pdf}
    \caption{\textbf{The architecture of the proposed GraphTER.}
    In the unsupervised feature learning stage, the representation encoder and transformation decoder are jointly trained by minimizing (\ref{eq:loss}). In the supervised evaluation stage, the first several blocks of the encoder are fixed with frozen weights and a linear classifier is trained with labeled samples.
    }
    \label{fig:framework}
    \vspace{-0.2in}
\end{figure}

\vspace{-0.05in}
\subsection{The Algorithm}

Given graph signals $\mathbf{X}=\{\mathbf{x}_1,\mathbf{x}_2,...,\mathbf{x}_N\}^{\top}$ over $N$ nodes, in each iteration of training, we \textit{randomly sample} a subset of nodes $\mathbf S$ from the graph, either globally or locally.
Global sampling refers to random sampling over the entire nodes globally, while local sampling is limited to a local set of nodes in the graph.
Node sampling not only enables us to characterize global and local graph structures at various scales, but also reduces the number of node-wise transformation parameters to estimate for computational efficiency.

Then we draw a node-wise transformation $\mathbf{t}_i$ corresponding to each sample $\mathbf{x}_i$ of nodes in $\mathbf S$, either isotropically or anisotropically.
Accordingly, the graph $\tilde{\mathbf{A}}$ associated with the transformed graph also transforms equivariantly from the original $\mathbf{A}$ under $\mathbf t$.
Specifically, as illustrated in Fig.~\ref{fig:node_wise_transform}, we construct a $k$NN graph to make use of the connectivity between the nodes, whose matrix representation in $\mathbf{A}$ changes after applying the sampled node-wise transformations.

To learn the applied node-wise transformations, we design a full graph-convolutional auto-encoder network as illustrated in Fig.~\ref{fig:framework}.
Among various paradigms of GCNNs, we choose EdgeConv \cite{wang2019dynamic} as a basic building block of the auto-encoder network, which efficiently learns node-wise representations by aggregating features along all the edges emanating from each connected node. Below we will explain the representation encoder and the transformation decoder in detail.

\vspace{-0.05in}
\subsubsection{Representation Encoder}

The representation encoder $E$ takes the signals of an original graph $\mathbf{X}$ and the transformed counterparts $\tilde{\mathbf{X}}$ as input, along with their corresponding graphs.
$E$ encodes node-wise features of $\mathbf{X}$ and $\tilde{\mathbf{X}}$ through a Siamese encoder network with shared weights, where EdgeConv layers are used as basic feature extraction blocks.
As shown in Fig.~\ref{fig:node_wise_transform}, given a non-transformed central node $\mathbf{x}_i$ and its transformed neighbors $\mathbf{t}_j(\mathbf{x}_j)$, the input layer of encoded feature of $\mathbf{x}_i$ is
\begin{equation}
\setlength{\abovedisplayskip}{3pt}
\setlength{\belowdisplayskip}{3pt}
\begin{split}
    E_{\text{in}}(\tilde{\mathbf{X}}, \tilde{\mathbf{A}})_i & = \underset{j \in \mathcal{N}(i)}{\max} \; \tilde{a}_{i,j} \\
    & = \underset{j \in \mathcal{N}(i)}{\max} \; \text{ReLU}(\theta(\mathbf{t}_j(\mathbf{x}_j)-\mathbf{x}_i) + \phi \mathbf{x}_i),
\end{split}
\label{eq:encoding}
\end{equation}
where $\tilde{a}_{i,j}$ denotes the edge feature, {\it i.e.}, edge weight in $\tilde{\mathbf{A}}$.
$\theta$ and $\phi$ are two weighting parameters,
and $j \in \mathcal{N}(i)$ denotes node $j$ is in the $k$-nearest neighborhood of node $i$.
Then, multiple layers of regular edge convolutions \cite{wang2019dynamic} are stacked to form the final encoder.

Edge convolution in (\ref{eq:encoding}) over each node essentially aggregates features from neighboring nodes via {\it edge weights} $\tilde{a}_{i,j}$.
Since the edge information of the underlying graph transforms with the transformations of individual nodes as demonstrated in Fig.~\ref{fig:node_wise_transform}, edge convolution is able to extract higher-level features from the original and transformed edge information.
Also, as features of each node are learned via propagation from transformed and non-transformed nodes isotropically or anisotropically by both local or global sampling, the learned representation is able to capture intrinsic graph structures at multiple scales.

\vspace{-0.05in}
\subsubsection{Transformation Decoder}

Node-wise features of the original and transformed graphs are then concatenated at each node, which are then fed into the transformation decoder.
The decoder consists of several EdgeConv blocks to aggregate the representations of both the original and transformed graphs to predict the node-wise transformations $\mathbf{t}$.
Based on the loss in \eqref{eq:loss}, $\mathbf{t}$ is decoded by minimizing the mean squared error (MSE) between the ground truth and estimated transformation parameters at each sampled node. Fig.~\ref{fig:framework} illustrates the architecture of learning the proposed GraphTER in such an auto-encoder structure.

\subsection{Datasets and Experimental Setup}

\textbf{ModelNet40} \cite{wu20153d}. This dataset contains $12,311$ meshed CAD models from $40$ categories, where $9,843$ models are used for training and $2,468$ models are for testing.
For each model, $1,024$ points are sampled from the original mesh.
We train the unsupervised auto-encoder and the classifier under the training set, and evaluate the classifier under the testing set.

\textbf{ShapeNet part} \cite{yi2016scalable}. This dataset contains $16,881$ 3D point clouds from $16$ object categories, annotated with $50$ parts.
Each 3D point cloud contains $2,048$ points, most of which are labeled with fewer than six parts.
We employ $12,137$ models for training the auto-encoder and the classifier, and $2,874$ models for testing.

We treat points in each point cloud as nodes in a graph, and the $(x,y,z)$ coordinates of points as graph signals.
A $k$NN graph is then constructed on the graph signals to guide graph convolution.

Next, we introduce our node-wise graph signal transformation. In experiments, we sample a portion of nodes with a sampling rate $r$ from the entire graph to perform node-wise transformations, including
1) \textbf{Global sampling:} randomly sample $r\%$ of points from all the points in a 3D point cloud;
2) \textbf{Local sampling:} randomly choose a point and search its $k$ nearest neighbors in terms of Euclidean distance, forming a local set of $r\%$ of points.

Then, we apply three types of node-wise transformations to the coordinates of point clouds, including
1) \textbf{Translation:} randomly translate each of three coordinates of a point by three parameters in the range $[-0.2, 0.2]$;
2) \textbf{Rotation:} randomly rotate each point with three rotation parameters all in the range $[\ang{-5},\ang{5}]$;
3) \textbf{Shearing}: randomly shear the $x$-, $y$-, $z$-coordinates of each point with the six parameters of a shearing matrix in the range $[-0.2, 0.2]$.
We consider two strategies to transform the sampled nodes: \textbf{Isotropically} or \textbf{Anisotropically}, which applies transformations with node-invariant or node-variant parameters.

\begin{table}[t]
\centering
\small
\caption{Classification accuracy (\%) on ModelNet40 dataset.}
\label{tab:classification}
\begin{tabular}{rcccc}
\hline
\multicolumn{1}{c}{\textbf{Method}} & \textbf{Year} & \textbf{Unsupervised} & \textbf{Accuracy} \\ \hline
3D ShapeNets \cite{wu20153d} & 2015 & No & 84.7 \\
VoxNet \cite{maturana2015voxnet} & 2015 & No & 85.9 \\
PointNet \cite{qi2017pointnet} & 2017 & No & 89.2 \\
PointNet++ \cite{qi2017pointnet++} & 2017 & No & 90.7 \\
KD-Net \cite{klokov2017escape} & 2017 & No & 90.6 \\
PointCNN \cite{li2018pointcnn} & 2018 & No & 92.2 \\
PCNN \cite{atzmon2018point} & 2018 & No & 92.3 \\
DGCNN \cite{wang2019dynamic} & 2019 & No & 92.9 \\
RS-CNN \cite{liu2019relation} & 2019 & No & 93.6 \\ \hline
T-L Network \cite{girdhar2016learning} & 2016 & Yes & 74.4 \\
VConv-DAE \cite{sharma2016vconv} & 2016 & Yes & 75.5 \\
3D-GAN \cite{wu2016learning} & 2016 & Yes & 83.3 \\
LGAN \cite{achlioptas2018learning} & 2018 & Yes & 85.7 \\
FoldingNet \cite{yang2018foldingnet} & 2018 & Yes & 88.4 \\
MAP-VAE \cite{han2019multi} & 2019 & Yes & 90.2 \\
L2G-AE \cite{liu2019l2g} & 2019 & Yes & 90.6 \\ \hline
GraphTER & & Yes & \textbf{92.0} \\ \hline
\end{tabular}
\vspace{-0.2in}
\end{table}

\vspace{-0.05in}
\subsection{Point Cloud Classification}

First, we evaluate the GraphTER model on the ModelNet40 \cite{wu20153d} dataset for point cloud classification.

\vspace{-0.1in}
\subsubsection{Implementation Details}

In this task, the auto-encoder network is trained via the SGD optimizer with a batch size of $32$.
The momentum and weight decay rate are set to $0.9$ and $10^{-4}$, respectively.
The initial learning rate is $0.1$, and then decayed using a cosine annealing schedule \cite{loshchilov2016sgdr} for $512$ training epochs.
We adopt the cross entropy loss to train the classifier.

We deploy eight EdgeConv layers as the encoder, and the number $k$ of nearest neighbors is set to $20$ for all EdgeConv layers.
Similar to \cite{wang2019dynamic}, we use shortcut connections for the first five layers to extract multi-scale features, where we concatenate features from these layers to acquire a $1,024$-dimensional node-wise feature vector.
After the encoder, we employ three consecutive EdgeConv layers as the decoder -- the output feature representations of the Siamese encoder first go through a channel-wise concatenation, which are then fed into the decoder to estimate node-wise transformations.
The batch normalization layer and LeakyReLU activation function with a negative slope of $0.2$ is employed after each convolutional layer.

During the training procedure of the classifier, the first five EdgeConv layers in the encoder are used to represent input cloud data by node-wise concatenating their output features with the weights frozen. After the five EdgeConv layers, we apply three fully-connected layers node-wise to the aggregated features. Then, global max pooling and average pooling are deployed to acquire the global features, after which three fully-connected layers are used to map the global features to the classification scores. Dropout with a rate of $0.5$ is adopted in the last two fully-connected layers.

\vspace{-0.1in}
\subsubsection{Experimental Results}
Tab.~\ref{tab:classification} shows the results for 3D point cloud classification, where the proposed model applies isotropic node-wise shearing transformation with a global sampling rate of $r=25\%$.
We compare with two classes of methods: unsupervised approaches and supervised approaches.
The GraphTER model achieves 92.0\% of classification accuracy on the ModelNet40 dataset, which outperforms the state-of-the-art unsupervised methods.
In particular, most of the compared unsupervised models combine the ideas of both GAN and AED, and map 3D point clouds to unsupervised representations by auto-encoding data, such as FoldingNet \cite{yang2018foldingnet}, MAP-VAE \cite{han2019multi} and L2G-AE \cite{liu2019l2g}.
Results show that the GraphTER model achieves significant improvement over these methods, showing the superiority of the proposed node-wise AET over both the GAN and AED paradigms.

Moreover, the unsupervised GraphTER model also achieves comparable performance with the state-of-the-art fully supervised results.
This significantly closes the gap between unsupervised approaches and the fully supervised counterparts in literature.

\vspace{-0.1in}
\subsubsection{Ablation Studies}

Further, we conduct ablation studies under various experimental settings of sampling and transformation strategies on the ModelNet40 dataset.

First, we analyze the effectiveness of different node-wise transformations under global or local sampling.
Tab.~\ref{tab:diff_transform} presents the classification accuracy with three types of node-wise transformation methods.
We see that the shearing transformation achieves the best performance,  improving by 1.05\% on average over translation, and 0.52\% over rotation.
This shows that the proposed GraphTER model is able to learn better feature representations under more complex transformations.

Moreover, we see that the proposed model achieves an accuracy of 90.70\% on average under global sampling, which outperforms local sampling by 0.46\%.
This is because global sampling better captures the global structure of graphs, which is crucial in such a graph-level task of classifying 3D point clouds.
Meanwhile, under the two sampling strategies, the classification accuracy from isotropic transformations is higher than that from the anisotropic one.
The reason lies in the intrinsic difficulty of training the transformation decoder with increased complexity of more parameters when applying anisotropic transformations.

\begin{table}[t]
\centering
\small
\caption{Unsupervised classification accuracy (\%) on ModelNet40 dataset with different sampling and transformation strategies.}
\label{tab:diff_transform}
\begin{tabular}{c|cc|cc|c}
\hline
\multirow{2}{*}{} & \multicolumn{2}{c|}{Global Sampling} & \multicolumn{2}{c|}{Local Sampling} & \multirow{2}{*}{Mean} \\ \cline{2-5}
 & Iso. & Aniso. & Iso. & Aniso. & \\ \hline
Translation & 90.15 & 90.15 & 89.91 & 89.55 & 89.94 \\
Rotation & 91.29 & 90.24 & 90.48 & 89.87 & 90.47 \\
Shearing & \textbf{92.02} & \textbf{90.32} & \textbf{91.65} & \textbf{89.99} & \textbf{90.99} \\ \hline
\multirow{2}{*}{Mean} & \textbf{91.15} & 90.24 & \textbf{90.68} & 89.80 & \\ \cline{2-5}
 & \multicolumn{2}{c|}{\textbf{90.70}} & \multicolumn{2}{c|}{90.24} &  \\ \hline
\end{tabular}
\vspace{-0.1in}
\end{table}

Moreover, we evaluate the effectiveness of different sampling rates $r$ under the translations as reported in Tab.~\ref{tab:sample_rate}.
The classification accuracies under various sampling rates are almost the same, and the result under $r=25\%$ is comparable to that under $r=100\%$.
This shows that the performance of the proposed model is insensitive to the variation of sampling rates, {\it i.e.}, applying node-wise transformations to a small number of nodes in the graph is sufficient to learn intrinsic graph structures.

\begin{table}[t]
\centering
\small
\caption{Unsupervised classification accuracy (\%) on ModelNet40 dataset applying translation at different node sampling rates.}
\label{tab:sample_rate}
\begin{tabular}{c|cc|cc|c}
\hline
\multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Sampling\\ Rate\end{tabular}} & \multicolumn{2}{c}{Global Sampling} & \multicolumn{2}{c|}{Local Sampling} & \multirow{2}{*}{Mean} \\ \cline{2-5}
 & Iso. & Aniso. & Iso. & Aniso. & \\ \hline
25\% & 90.15 & 90.15 & 89.91 & 89.55 & 89.94 \\
50\% & 90.03 & 89.63 & 89.95 & 89.47 & 89.77 \\
75\% & 91.00 & 89.67 & 91.41 & 89.75 & 90.46 \\
100\% & 89.67 & 89.99 & 89.67 & 89.99 & 89.83 \\ \hline
\end{tabular}
\vspace{-0.1in}
\end{table}

\begin{table*}[t]
\centering
\scriptsize
\caption{Part segmentation results on ShapeNet part dataset. Metric is mIoU(\%) on points.}
\label{tab:segmentation}
\begin{tabular}{r|p{0.65cm}<{\centering}|p{0.5cm}<{\centering}|p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}p{0.4cm}<{\centering}}
\hline
 & Unsup. & Mean & Aero & Bag & Cap & Car & Chair & \begin{tabular}[c]{@{}c@{}}Ear\\ Phone\end{tabular} & Guitar & Knife & Lamp & Laptop & Motor & Mug & Pistol & Rocket & \begin{tabular}[c]{@{}c@{}}Skate\\ Board\end{tabular} & Table \\ \hline
Samples & & & 2690 & 76 & 55 & 898 & 3758 & 69 & 787 & 392 & 1547 & 451 & 202 & 184 & 283 & 66 & 152 & 5271 \\ \hline
PointNet \cite{qi2017pointnet} & No & 83.7 & 83.4 & 78.7 & 82.5 & 74.9 & 89.6 & 73.0 & 91.5 & 85.9 & 80.8 & 95.3 & 65.2 & 93.0 & 81.2 & 57.9 & 72.8 & 80.6 \\
PointNet++ \cite{qi2017pointnet++} & No & 85.1 & 82.4 & 79.0 & 87.7 & 77.3 & 90.8 & 71.8 & 91.0 & 85.9 & 83.7 & 95.3 & 71.6 & 94.1 & 81.3 & 58.7 & 76.4 & 82.6 \\
KD-Net \cite{klokov2017escape} & No & 82.3 & 80.1 & 74.6 & 74.3 & 70.3 & 88.6 & 73.5 & 90.2 & 87.2 & 81.0 & 94.9 & 57.4 & 86.7 & 78.1 & 51.8 & 69.9 & 80.3 \\
PCNN \cite{atzmon2018point} & No & 85.1 & 82.4 & 80.1 & 85.5 & 79.5 & 90.8 & 73.2 & 91.3 & 86.0 & 85.0 & 95.7 & 73.2 & 94.8 & 83.3 & 51.0 & 75.0 & 81.8 \\
PointCNN \cite{li2018pointcnn} & No & 86.1 & 84.1 & 86.5 & 86.0 & 80.8 & 90.6 & 79.7 & 92.3 & 88.4 & 85.3 & 96.1 & 77.2 & 95.3 & 84.2 & 64.2 & 80.0 & 83.0 \\
DGCNN \cite{wang2019dynamic} & No & 85.2 & 84.0 & 83.4 & 86.7 & 77.8 & 90.6 & 74.7 & 91.2 & 87.5 & 82.8 & 95.7 & 66.3 & 94.9 & 81.1 & 63.5 & 74.5 & 82.6 \\
RS-CNN \cite{liu2019relation} & No & 86.2 & 83.5 & 84.8 & 88.8 & 79.6 & 91.2 & 81.1 & 91.6 & 88.4 & 86.0 & 96.0 & 73.7 & 94.1 & 83.4 & 60.5 & 77.7 & 83.6 \\ \hline
LGAN \cite{achlioptas2018learning} & Yes & 57.0 & 54.1 & 48.7 & 62.6 & 43.2 & 68.4 & 58.3 & 74.3 & 68.4 & 53.4 & 82.6 & 18.6 & 75.1 & 54.7 & 37.2 & 46.7 & 66.4 \\
MAP-VAE \cite{han2019multi} & Yes & 68.0 & 62.7 & 67.1 & 73.0 & 58.5 & 77.1 & 67.3 & 84.8 & 77.1 & 60.9 & 90.8 & 35.8 & 87.7 & 64.2 & 45.0 & 60.4 & 74.8 \\ \hline
GraphTER & Yes & \textbf{81.9} & \textbf{81.7} & \textbf{68.1} & \textbf{83.7} & \textbf{74.6} & \textbf{88.1} & \textbf{68.9} & \textbf{90.6} & \textbf{86.6} & \textbf{80.0} & \textbf{95.6} & \textbf{56.3} & \textbf{90.0} & \textbf{80.8} & \textbf{55.2} & \textbf{70.7} & \textbf{79.1} \\ \hline
\end{tabular}
\vspace{-0.2in}
\end{table*}

\subsection{Point Cloud Segmentation}

We also apply the GraphTER model to 3D point cloud part segmentation on ShapeNet part dataset \cite{yi2016scalable}.

\vspace{-0.2in}
\subsubsection{Implementation Details}

We also use SGD optimizer to train the auto-encoding transformation network.
The hyper-parameters are the same as in 3D point cloud classification except that we train for $256$ epochs.
We adopt the negative log likelihood loss to train the node-wise classifier for segmenting each point in the clouds.

The auto-encoding architecture is similar to that of the classification task, where we employ five EdgeConv layers as the encoder.
However, the first two EdgeConv blocks consist of two MLP layers with the number of neurons \{64, 64\} in each layer.
We use shortcut connections to concatenate features from the first four layers to  a $512$-dimensional node-wise feature vector.

As for the node-wise classifier, we deploy the same architecture as in \cite{wang2019dynamic}. The output features from the encoder are concatenated node-wise with globally max-pooled features, followed by four fully-connected layers to classify each node.
During the training procedure, the weights of the first four EdgeConv blocks in the encoder are kept frozen.

\vspace{-0.2in}
\subsubsection{Experimental Results}

We adopt the Intersection-over-Union (IoU) metric to evaluate the performance.
We follow the same evaluation protocol as in the PointNet \cite{qi2017pointnet}: the IoU of a shape is computed by averaging the IoUs of different parts occurring in that shape, and the IoU of a category is obtained by averaging the IoUs of all the shapes belonging to that category.
The mean IoU (mIoU) is finally calculated by averaging the IoUs of all the test shapes.

We also compare the proposed model with unsupervised approaches and supervised approaches in this task, as listed in Tab.~\ref{tab:segmentation}.
We achieve a mIoU of $81.9\%$, which significantly outperforms the state-of-the-art unsupervised method MAP-VAE \cite{han2019multi} by $13.9\%$.

Moreover, the unsupervised GraphTER model also achieves the comparable performance to the state-of-the-art fully supervised approaches, greatly pushing closer towards the upper bound set by the fully supervised counterparts.

\vspace{-0.2in}
\subsubsection{Visualization Results}

Fig.~\ref{fig:sup_seg_results} visualizes the results of the proposed unsupervised model and two state-of-the-art fully supervised methods: DGCNN \cite{wang2019dynamic} and RS-CNN \cite{liu2019relation}.
The proposed model produces better segmentation on the ``table" model in the first row, and achieves comparable results on the other models.
Further, we qualitatively compare the proposed method with the state-of-the-art unsupervised method MAP-VAE \cite{han2019multi}, as illustrate in Fig.~\ref{fig:unsup_seg_results}.
The proposed model leads to more accurate segmentation results than MAP-VAE, {\it e.g.}, the engines of planes and the legs of chairs.

\begin{figure}[t]
    \centering
    \subfigure[Ground-truth]{
    \includegraphics[width=0.22\columnwidth]{seg_results_gt.pdf}
    }
 	\subfigure[DGCNN]{
 	\includegraphics[width=0.22\columnwidth]{seg_results_dgcnn.pdf}
 	}
 	\subfigure[RS-CNN]{
 	\includegraphics[width=0.22\columnwidth]{seg_results_rscnn.pdf}
 	}
 	\subfigure[GraphTER]{
 	\includegraphics[width=0.22\columnwidth]{seg_results_gter.pdf}
 	}
    \caption{\textbf{Visual comparison of point cloud part segmentation with supervised methods. }
    Our unsupervised GraphTER learning achieves comparable results with the state-of-the art fully supervised approaches.
    }
    \label{fig:sup_seg_results}
    \vspace{-0.1in}
\end{figure}

\begin{figure}[t]
    \centering
 	\subfigure[MAP-VAE]{
 	\includegraphics[width=0.95\columnwidth]{seg_results_unsup_map_vae.pdf}
 	}
 	\subfigure[GraphTER]{
 	\includegraphics[width=0.95\columnwidth]{seg_results_unsup_gter.pdf}
 	}
    \caption{\textbf{Visual comparison of point cloud part segmentation with the state-of-the-art unsupervised method MAP-VAE. }
    We achieve more accurate segmentation even in tiny parts and transition regions.
    }
    \label{fig:unsup_seg_results}
    \vspace{-0.1in}
\end{figure}

\vspace{-0.1in}
\subsubsection{Ablation Studies}

\begin{table}[t]
\centering
\small
\caption{Unsupervised segmentation results on ShapeNet part dataset with different transformation strategies. Metric is mIoU (\%) on points.}
\label{tab:seg_ablation}
\begin{tabular}{c|cc|cc|c}
\hline
\multirow{2}{*}{} & \multicolumn{2}{c}{Global Sampling} & \multicolumn{2}{c|}{Local Sampling} & \multirow{2}{*}{Mean} \\ \cline{2-5}
 & Iso. & Aniso. & Iso. & Aniso. & \\ \hline
Translation & 79.83 & 79.88 & 80.05 & 79.85 & 79.90 \\
Rotation & 80.20 & \textbf{80.29} & 80.87 & 80.02 & 80.35 \\
Shearing & \textbf{81.88} & 80.28 & \textbf{81.89} & \textbf{80.48} & \textbf{81.13} \\ \hline
\multirow{2}{*}{Mean} & \textbf{80.64} & 80.15 & \textbf{80.94} & 80.12 & \multirow{2}{*}{\textbf{}} \\ \cline{2-5}
 & \multicolumn{2}{c|}{80.39} & \multicolumn{2}{c|}{\textbf{80.53}} & \\ \hline
\end{tabular}
\vspace{-0.2in}
\end{table}

Similar to the classification task, we analyze the effectiveness of different node-wise transformations under global or local sampling, as presented in Tab.~\ref{tab:seg_ablation}.
The proposed model achieves the best performance under the shearing transformation, improving by 1.23\% on average over translation, and 0.78\% over rotation, which demonstrates the benefits of GraphTER learning under complex transformations.

Further, the proposed model achieves a mIoU of 80.53\% on average under local sampling, which outperforms global sampling by 0.14\%.
This is because local sampling of nodes captures the local structure of graphs better, which is crucial in node-level 3D point cloud segmentation task.

\section{Introduction}
\vspace{-0.05in}
\label{sec:intro}
\input{01_introduction.tex}

\vspace{-0.1in}
\section{Related Works}
\vspace{-0.05in}
\label{sec:related_works}
\input{02_related_works.tex}

\vspace{-0.1in}
\section{Graph Transformations}
\label{sec:graphT}
\input{03_graph.tex}

\vspace{-0.1in}
\section{GraphTER: The Proposed Approach}
\label{sec:method}
\input{04_method.tex}

\vspace{-0.1in}
\section{Experiments}
\vspace{-0.05in}
\label{sec:experiments}
\input{05_experiments.tex}

\section{Conclusion}
\vspace{-0.05in}
\label{sec:conclusion}
\input{06_conclusion.tex}

\clearpage

{\small
\bibliographystyle{ieee}


\section{Introduction}
\label{sec:Introduction}
Deep learning~\cite{lecun2015deep}, especially deep Convolutional Neural Network (CNN)~\cite{lecun1998gradient}, has led to state-of-the-art results spanning many machine learning fields~\cite{he2016deep,ren2015faster}. Despite the great success in numerous applications, recent studies show that deep CNNs are vulnerable to some well-designed input samples named as Adversarial Examples~\cite{DBLP:journals/corr/SzegedyZSBEGF13,biggio2013evasion}. Take the task of image classification as an example, for almost every commonly used well-performed CNN, attackers are able to construct a small perturbation on an input image, which is almost imperceptible to humans but can make the model give a wrong prediction. The problem is serious as some well-designed adversarial examples can be transferred among different kinds of CNN architectures~\cite{DBLP:journals/corr/PapernotMG16}. As a result, a machine learning system can be easily attacked even if the attacker does not have access to the model parameters.

There is a rapidly growing body of work on how to obtain a robust CNN, mainly based on adversarial training~\cite{DBLP:journals/corr/SzegedyZSBEGF13,madry2017towards,goodfellow2014,huang2015learning}. However, those methods need lots of extra computation to obtain adversarial examples at each time step and tend to overfit the attacking method used in training~\cite{buckman2018thermometer}. In this paper, we tackle the problem in a perspective different from most existing methods. In particular, we explore the possibility of designing new CNN architectures which can be trained using standard optimization methods on standard benchmark datasets and can enjoy robustness by themselves, without appealing to other techniques. Recent studies~\cite{geirhos2017comparing,geirhos2018imagenet,baker2018deep} show that the predictions of standard CNNs mainly depend on the texture of objects. The textural information has a high degree of redundancy and may be easily injected with adversarial noise ~\cite{yang2019me,hosseini2019dropping}. Also, \cite{liu2018analyzing} finds adversarial attack methods may perturb local patches to contain textural features of incorrect classes.
All the literature suggests that the wrong prediction by CNNs for adversarial examples mainly comes from the change in the textural information. The small perturbation of adversarial examples will change the textures and eventually affect the features extracted by the CNNs. Therefore, a natural way to avoid adversarial examples is to let the CNN make predictions relying less on textures but more on other information, such as the shape, which cannot be severely affected by small perturbations.

In real practice, sometimes a camera might have mechanical failures which cause the output image to have many defective pixels (such pixels are always black in all images). Nonetheless, humans can still recognize objects in the image with defective pixels since we are able to classify the objects even in the absence of local textural information. Motivated by this, we introduce the concept of defectiveness into the convolutional neural networks: we call a neuron a defective neuron if its output value is fixed to zero no matter what input signal is received; similary, a convolutional layer is a \emph{defective convolutional layer} if it contains defective neurons. Before training,
we replace the standard convolutional layers with the defective version on a standard CNN and train the network in the standard way. As defective neurons of the defective convolutional layer contain no information and are very different from their spatial neighbors, the textural information cannot be accurately extracted from the bottom defective layers to top layers. Therefore, we destroy local textural information to a certain extent and prompt the neural network to rely more on other information for classification. We call the architecture deployed with defective convolutional layers as \emph{defective convolutional network}.

We find that applying the defective convolutional layers to the bottom\footnote{In this paper, bottom layer means the layer close to the input and top layer means the layer close to the output prediction.} layers of the network and introducing various patterns for defective neurons arrangement across channels are crucial for robustness. In summary, our main contributions are:

\begin{itemize}

    \item We propose the Defective CNNs that, compared to the standard CNNs, relies less on textures and more on shapes of the inputs for making predictions. The adversarial robustness is implanted in the design of Defective CNNs, because most adversarial attacks produce perturbations that leave the underlying shape intact, only to manipulate the local texture of the inputs.

    \item Experimental results show that Defective CNN has superior defense performance than standard CNN against transfer-based attacks, decision-based attacks, additive Gaussian noise, and grey-box attacks.
    \item Using the standard training method, Defective CNN achieves state-of-the-art results against two transfer-based black-box attacks while maintaining high accuracy on clean test data.

\end{itemize}
\vspace{-0.1in}

\vspace{-0.2in}

\section{Related Work}
Various methods have been proposed to defend against adversarial examples. One line of research is to derive a meaningful optimization objective and optimize the model by adversarial training~\cite{DBLP:journals/corr/SzegedyZSBEGF13,madry2017towards,goodfellow2014,huang2015learning}. The high-level idea of these works is that if we can predict the potential attack to the model during optimization, then we can give the attacked sample a correct signal and use it during training. Another line of research is to take an adjustment to the input image before letting it go through the deep neural network~\cite{liao2017defense,song2017pixeldefend,samangouei2018defense,sun2018adversarial,Xie_2019_CVPR}. The basic intuition behind this is that if we can clean the adversarial attack to a certain extent, then such attacks can be defended. Although these methods achieve some success, a major difficulty is that it needs a large extra cost to collect adversarial examples and hard to apply on large-scale datasets.

Several studies~\cite{geirhos2017comparing,geirhos2018imagenet,baker2018deep} show that the prediction of CNNs is mainly from the texture of objects but not the shape. Also, \cite{liu2018analyzing} found that adversarial examples usually perturb a patch of the original image so that the perturbed patch looks like the texture of incorrect classes. For example, the adversarial example of the panda image is misclassified as a monkey because a patch of the panda skin is perturbed adversarially so that it alone looks like the face of a monkey (see Figure 11 in \cite{liu2018analyzing}). All previous works above suggest that the CNN learns textural information more than shape and the adversarial attack might come from textural-level perturbations. This is also correlated with robust features~\cite{tsipras2018robustness,ilyas2019adversarial,hosseini2019dropping,yang2019me} which has attracted more interest recently. Pixels which encode textural information contain high redundancy and may be easily deteriorated to the distribution of incorrect classes. However, shape information is more compact and might be a more robust feature.

\section{Defective Convolutional Neural Network}
\label{sec:Random Mask}
\subsection{Design of Defective Convolutional Layers}
In this subsection, we introduce our proposed defective convolutional neural networks and discuss the differences between our proposed method and related topics.

First, we briefly introduce the notations. For one convolutional layer, denote $x$ as the input and $z$ as the output of neurons in the layer. Note that $x$ may be the input image or the output of the last convolutional layer.
The input $x$ is usually a $M\times N\times K$ tensor in which $M/N$ are the height/width of a feature map, and $K$ is the number of feature maps, or equivalently, channels.
Denote $w$ and $b$ as the parameters (e.g., the weights and biases) of the convolutional kernel. Then a standard convolutional layer can be mathematically defined as below.

\textbf{Standard convolutional layer: }
\begin{eqnarray}
x'&=& w \otimes_{\text{conv}} x + b,\\
z &=& f(x'),
\end{eqnarray}
where $f(\cdot)$ is a non-linear activation function such as ReLU\footnote{Batch normalization is popularly used on $x'$ before computing $z$. Here we simply omit this.} and $\otimes_{\text{conv}}$ is the convolutional operation.

The convolutional filter receives signals in a patch and extracts local textural information from the patch. As mentioned in the introduction, recent works suggest that the prediction of standard CNNs strongly depends on such textural information, and noises imposed on the texture may lead to wrong predictions. Therefore, we hope to learn a feature extractor which does not solely rely on textural features and also considers other information. To achieve this goal, we introduce the \emph{defective convolutional layer} in which some neurons are purposely designed to be corrupted. Define $M_{\text{defect}}$ to be a binary matrix of size $M\times N\times K$. Our defective convolutional layer is defined as follows.

\textbf{Defective convolutional layer: }
\begin{eqnarray}
x'&=& w \otimes_{\text{conv}} x + b,\\
z' &=& f(x')\\
z &=& M_{\text{defect}} * z',
\end{eqnarray}

where $*$ denotes element-wise product.
$M_{\text{defect}}$ is a fixed matrix and is not learnable during training and testing. A simple visualization of a defective convolutional layer is shown in Figure~\ref{RandomMask}. From the figure, we can see that $M_{\text{defect}}$ plays a role of ``masking'' out values of some neurons in the layer. This disturbs the distribution of local textural information and decouples the correlation among neurons. With the masked output $z$ as input, the feature extractor of the next convolutional layer cannot accurately capture the local textural feature from $x$. As a consequence, the textural information is hard to pass through the defective CNN from bottom to top. To produce accurate predictions, the deep neural network has to find relevant signals other than the texture, e.g., the shape. Those corrupted neurons have no severe impact on the extraction of shape information since neighbors of those neurons in the same filter are still capable of passing the shape information to the next layer.

In this paper, we find that simply setting $M_{ \text{defect}}$ by random initialization is already helpful for learning a robust CNN. Before training, we sample each entry in $M_{\text{defect}}$ using Bernoulli distribution with keep probability $p$ and then fix $M_{\text{defect}}$ during training and testing. More discussions and ablation studies are provided in Section~\ref{sec:Experiment}.

\begin{figure}[h]
\centering
\includegraphics[width=3.3in]{./img/rdmask.png}
\vskip -0.1in
\caption{An illustration of three channels~(neuron-wise) from standard convolutional layer to defective convolutional layer. Black neurons are the defective neurons which are randomly selected. The output of defective neurons is fixed to zero during both training and testing phases. Note that the number of parameters in the network is not reduced after including defective neurons.\label{RandomMask}}
\vspace{-0.2in}
\end{figure}

As can be seen from Equation (3)-(5), the implementation of our defective convolutional layer is similar to the dropout operation~\cite{srivastava2014dropout}. To demonstrate the relationship and differences, we mathematically define the dropout as below.

\textbf{Standard convolutional layer +  dropout: }
\begin{eqnarray}
M_{\text{dp}} &\sim& \text{Bernoulli}(p) \\
x'&=& w \otimes_{\text{conv}} x + b\\
z' &=& f(x')\\
z &=& M_{\text{dp}}*z'.
\end{eqnarray}
The shape of $M_{\text{dp}}$ is the same as $M_{\text{defect}}$, and the value of each entry in $M_{\text{dp}}$ is sampled in each batch using some sampling strategies at each step during training. Generally, entries in $M_{\text{dp}}$ are independent and identically sampled in an online fashion using Bernoulli distribution with keep probability $p$. DropBlock~\cite{ghiasi2018dropblock} dropouts a connected block in each channel. In SpatialDropout~\cite{tompson2015efficient}, the dropout masks apply to whole channels. Note that the masked unit of our proposed method is a single neuron.

There are several significant differences between dropout and defective convolutional layer. First, the motivations behind the two methods are quite different. Dropout tries to reduce overfitting by preventing co-adaptations on training data. As the neurons in the feature maps still have full access to local textural features in testing, the model does not have to learn shape features. However, in our proposed architecture, defective neurons are fixed to be corrupted, and such neurons cannot contribute to local features. Second, the binary matrix $M_{\text{dp}}$ is sampled online during training and is removed during testing, while the binary matrix $M_{\text{defect}}$ in defective convolutional layers is predefined and keeps fixed in both training and testing. Third, places to apply and values of the keep probability $p$ for two methods are different. Dropout methods are usually applied to top layers, and $p$ is set to be large (e.g., $0.9$) ~\cite{tompson2015efficient,ghiasi2018dropblock}. For defective convolutional layer, we find using a small $p$ (e.g., $0.1$) and applying it to bottom layers are more effective.
\vspace{-0.1in}
\subsection{Defective CNN Relies Less on Texture but More on Shape}
\label{subsec:MoreShapeInfo}
In the defective CNN, some neurons are set to be corrupted during both training and testing, and we argue that this design can help the CNN make prediction relying less on textural information but more on shape information. In this subsection, we provide some empirical analyses to verify our intuition.

We design a particular image manipulation in which the local texture of the object in an image is preserved while the shape is destroyed. In detail, we divide an image into $k\times k$ patches and randomly relocate those patches to form a new image. A typical example is shown in Figure~\ref{RandomShuffleIllustration}. By relocating the patches, it is even hard for a human to recognize the object in the picture when $k$ is large.

\vspace{-0.1in}
\begin{figure}[h]
\begin{center}
\subfigure[original image]{\includegraphics[width=1in]{./img/11_6.png}}
\hspace{0.1in}
\subfigure[$2\times 2$ patches]{\includegraphics[width=1in]{./img/11_6_2_02.png}}
\hspace{0.1in}
\subfigure[$4\times 4$ patches ]{\includegraphics[width=1in]{./img/11_6_4_04.png}}
\hspace{0.1in}
\subfigure[$8\times 8$ patches]{\includegraphics[width=1in]{./img/11_6_8_02.png}}
\end{center}
\vspace{-0.2in}
\caption{An example image that is randomly shuffled after being divided into $2\times 2$, $4\times 4$ and $8\times 8$ patches respectively.
\label{RandomShuffleIllustration}}
\vskip -0.05in
\end{figure}

We manipulate a set of images and test whether a defective CNN and a standard CNN can make correct predictions. The experimental details are described as follows. First, we construct a defective CNN by applying defective convolutional layers to the bottom layers of a standard ResNet-18, and train the defective CNN along with a standard ResNet-18 on the ImageNet dataset. Then, we sample images from the validation set which are predicted correctly by both two CNNs.
We make manipulations to the sampled images by setting $k\in \{2,4,8\}$, feed these images to the networks and check their classification accuracy. The results in Table \ref{shuffle_table} show that when the shape information is destroyed but the local textural information is preserved, the defective CNN performs consistently worse than standard CNN, verifying that defective CNN makes predictions relying less on local textural information but more on shape information.

\begin{table}[h]
\begin{center}
\renewcommand\tabcolsep{5.0pt}
\begin{tabular}{cccc|c}
\toprule
\bf Model  & $2\times 2$ &$4\times 4$
& $8\times 8$ & IN $\rightarrow$ SIN
\\ \midrule
 Standard CNN &99.53\%	&84.36\%	&20.08\% &15.33\%
\\
 Defective CNN & 96.32\%	& 56.91\% & 9.04\% &20.20\%
\\ \bottomrule
\end{tabular}
\end{center}
\caption{Left three columns are the accuracy of classifying randomly shuffled images. The rightmost column is the accuracy of training on ImageNet and testing on Stylized-ImageNet. More results for different architectures can be found in Appendix~\ref{appendix:randomshuffle}. \label{shuffle_table}}
\vspace{-0.2in}
\end{table}

We also test the models on the Stylized-ImageNet~\cite{geirhos2018imagenet}, a stylized version of ImageNet, where the local textures of images are heavily distorted, while global object shapes remain. A model that more focuses on the shape cues should achieve higher performance on the Stylized-ImageNet while it is trained on the ImageNet.
We test on the same model used in the randomly shuffled experiments by feeding the images from the validation set of Stylized-ImageNet whose corresponding images in ImageNet can be correctly classified by both two tested models.
The result is listed in Table~\ref{shuffle_table}. It shows that the Defective CNN achieves higher transferring accuracy than the Standard CNN and thus verify our intuition.

From another perspective, if a model makes predictions relying less on textural information but more on shape information, the manipulation of the shape of objects will play a larger role in designing adversarial examples. To verify this, we train a defective CNN on CIFAR-10 dataset and Tiny-ImageNet dataset, and then apply an iterative attack method MIFGSM~\cite{dong2017boosting} on the validation set. Some examples are shown in Figure~\ref{cmp} and Appendix~\ref{appendix: more_aes}. We can see that adversarial examples against the defective CNN change the shape of the objects and may even fool humans as well. The phenomena not only supports the intuition, but also is consistent with the findings in \cite{tsipras2018robustness} that the representations learned by robust models tend to align better with salient data characteristics.

\begin{figure}[htp]
\vspace{-0.1in}
\centering
\subfigure[\scriptsize{Dog}]{\includegraphics[width=0.65in]{./img/9_0_5.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Bird}]{\includegraphics[width=0.65in]{./img/738_8_2.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Frog}]{\includegraphics[width=0.65in]{./img/451_9_6.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Ship}]{\includegraphics[width=0.65in]{./img/1724_1_8.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Mushroom}]{\includegraphics[width=0.65in]{./img/Tiny3_0_185.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Ladybug}]{\includegraphics[width=0.65in]{./img/Tiny1895_37_36.png}}
\\
\subfigure[\scriptsize{Airplane}]{\includegraphics[width=0.65in]{./img/9_0.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Ship}]{\includegraphics[width=0.65in]{./img/738_8.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Truck}]{\includegraphics[width=0.65in]{./img/451_9.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Automobile}]{\includegraphics[width=0.65in]{./img/1724_1.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Goldfish}]{\includegraphics[width=0.65in]{./img/Tiny3_0.png}}
\hspace{0.05in}
\subfigure[\scriptsize{Fly}]{\includegraphics[width=0.65in]{./img/Tiny1895_37.png}}
\caption{
Adversarial examples generated by Defective CNNs with perturbation scale  $\ell_\infty \in \{16, 32\}$. \textbf{First row}: the adversarial examples and the labels predicted by the CNNs. \textbf{Second row}: the corresponding original images and the ground truth labels. Detailed experimental settings and more examples can be found in Appendix~\ref{appendix: more_aes}. \label{cmp}}
\end{figure}

Furthermore, perturbations generated by standard CNNs and additive Gaussian noises usually change the textural information but hard to affect the shape information. A model is supposed to defend these attacks on textures better if it relies much on shape information for predictions. In Section~\ref{sec:Experiment}, we show that defective CNNs achieve higher defense performance than standard CNN against the two types of attack, further supporting our intuitions.

\section{Experiments}
\label{sec:Experiment}
In this section, we provide extensive experimental analyses on the robustness of defective CNNs against black-box attacks.
To better evaluate this point, we suggest that we should examine the target model against transfer-based attacks, decision-based attacks, and additive Gaussian noise.

In real-world tasks, attackers usually cannot access the parameters of the target models and thus need to transfer adversarial examples generated by their models. This setting of attack is referred to as transfer-based attacks~\cite{liu2016delving,kurakin2016adversarial}. Sometimes, attackers can get the final model decision and raise the more powerful decision-based attacks~\cite{brendel2017decision}. Both the two types of black-box attack are available in most real-world scenarios and should be considered. Recently, \cite{ford2019adversarial} bridge the adversarial robustness and corruption robustness \cite{hendrycks2018benchmarking}, and points out that a successful adversarial defend method should also effectively defense against additive Gaussian noise. Also, the additive Gaussian noise can be viewed as a type of black-box attack since the noise distribution has nothing to do with the parameters of tested models. Therefore, we also test the performance of target models against additive Gaussian noise.

We first test the robustness of defective CNN against transfer-based attacks, decision-based attacks, additive Gaussian noise, and then make ablation studies on possible design choices of defective CNN. Since we alter the network architecture and may pick up different potentially robust features following the argument in \cite{ilyas2019adversarial}, we also carefully study the robustness in the gray-box setting. Due to space limitation, we list the results of gray-box attacks, white-box attacks, and transfer-based attacks from ensemble models in Appendix~\ref{appendix:More Experiment Result}.

Note that, in this paper, all the successful defense rates except the rates listed in Table 2 are calculated on the samples whose corresponding original images can be classified correctly by the tested model. This can erase the influence of test accuracy that different models have different test accuracy on clean data, and thus help evaluate the robustness of models.

\vspace{-0.1in}
\subsection{Transfer-based Attack}
\label{sec:Experimental Settings}
\subsubsection{Experimental Settings}
\label{sec:Transfer-based Attack}
In this subsection, we evaluate the defense performance of networks with defective convolutional layers against transfer-based attacks.
We compare our proposed method with state-of-the-art defense methods~\cite{buckman2018thermometer,madry2017towards}. For fair comparisons, we follow \cite{buckman2018thermometer} to generate adversarial examples using wide residual networks~\cite{zagoruyko2016wide} with a depth of $32$ and a width factor of $4$. The \(4\)-block structure of ResNet-32 is shown in Appendix~\ref{appendix: network architecture}. The blocks are labeled $0, 1, 2, 3$ and the $0^{\text{th}}$ block is the first convolution layer. Both FGSM~\cite{goodfellow2014} and PGD~\cite{kurakin2016adversarial} attacks are run on the entire validation set of CIFAR-10 dataset. These two methods both have $\ell_\infty$ perturbation scale $8$ and PGD runs for \(7\) gradient descent steps with step size \(2\). The generated adversarial examples are used to attack target networks. For the target network, we use the same structure but applying defective convolutional layers to the $0^{\text{th}}$ and $1^{\text{st}}$ blocks with keep probability $p = 0.07$ and train the model using standard optimization method. As is mentioned in Section 3, our proposed method is essentially different from dropout, and thus we also take dropout methods as baselines. More specifically, we test SpatialDropout and DropBlock. For both methods, we follow the instruction from \cite{ghiasi2018dropblock} to apply dropout to the $3^{\text{rd}}$ block with $p = 0.9$. The block of DropBlock is set to be a $3\times 3$ square. Training details and curves can be found in Appendix~\ref{appendix:training_curve}.

Second, we test our proposed method in different architectures on the CIFAR-10 dataset. We apply defective convolutional layers, in a way which is similar to the experiment above, to five popular network architectures: ResNet-18~\cite{he2016deep}, ResNet-50, DenseNet-121~\cite{huang2017densely}, SENet-18~\cite{hu2017squeeze} and VGG-19~\cite{simonyan2014very}. For each architecture, we replace the standard convolutional layer with the defective version on the bottom layers. We then test the black-box defense performance against transfer-based attacks on 5000 samples from the validation set. Adversarial examples are generated by PGD, which runs for $20$ steps with step size $1$ and the $\ell_\infty$ perturbation scale is set to $16$. More results on the MNIST dataset and illustrations of where to apply the defective layers can be found in Appendix~\ref{appendix:More Experiment Result} and \ref{appendix: network architecture}.

\subsubsection{Experimental Results}
First, we compare with two strong defense methods~\cite{madry2017towards,buckman2018thermometer} and two dropout methods~\cite{tompson2015efficient,ghiasi2018dropblock}. The results are listed in Table~\ref{Table 0}. \cite{madry2017towards} is one adversarial training method that directly optimizes on adversarial examples in an online way. Based on adversarial training, \cite{buckman2018thermometer} proposed a method that discretizes inputs and achieved higher accuracy against transfer-based attack. The results show the strengths of our proposed method on both robustness and generalization, even though our model is only trained on \emph{clean data}. In addition, from the results of two dropout methods, we can conclude that spatial dropout and drop block do not improve the robustness of standard CNNs.

\begin{table}[h]
\begin{center}
\renewcommand\tabcolsep{5.0pt}
\begin{tabular}{cccc}
\toprule
\bf Model  &\bf FGSM &\bf PGD
&\bf Test Accuracy
\\ \midrule
Standard CNN &52.88\%	&15.98\%	&95.39\%
\\
Standard CNN + SD~\cite{tompson2015efficient}&51.63\%	&14.28\%	&95.98\%
\\
Standard CNN + DB~\cite{ghiasi2018dropblock}&51.55\%	&12.71\%	&95.81\%
\\
Adversarial Training~\cite{madry2017towards}&85.60\% &86.00\%	&87.30\%
\\
Thermometer(16)~\cite{buckman2018thermometer} &- &88.25\%	&89.88\%
\\
Thermometer(32)~\cite{buckman2018thermometer} &- &86.60\%	&90.30\%
\\
Defective CNN &\bf 86.24\%	& \bf 88.43\% & \bf 91.12\%
\\ \bottomrule
\end{tabular}
\end{center}
\caption{Black-box defense performances against transfer-based attacks. SD and DB denote spatial dropout and drop block, respectively. Detailed settings are listed in Section~\ref{sec:Transfer-based Attack}.}
\label{Table 0}
\end{table}

Second, we list the black-box defense results of applying defective convolutional layers to various architectures in Table~\ref{Table cifar2}. The results show that defective convolutional layers consistently improve the robustness of various network architectures against transfer-based attacks.

\begin{table}[H]
\begin{center}
\renewcommand\tabcolsep{4.0pt}
\scalebox{0.80}{
\begin{tabular}{ccccccc}
\toprule
\multicolumn{1}{c}{\bf Architecture}
&\bf $\text{ResNet-18}$
&\bf $\text{ResNet-50}$
&\bf $\text{DenseNet-121}$
&\bf $\text{SENet-18}$
&\bf $\text{VGG-19}$
&\bf Test Accuracy\\
\midrule
\multicolumn{1}{c}{ResNet-18} 	&5.98\%  & 0.94\%  & 14.14\% & 3.32\%  & 26.97\% & 95.33\% \\
0.5-Bottom &53.89\% & 33.05\% & 70.38\% & 57.52\% & 58.66\% & 93.39\% \\
0.3-Bottom &78.23\% & 67.64\% & 86.99\% & 82.46\% & 77.57\% & 91.83\% \\
\hline
\multicolumn{1}{c}{ResNet-50}&16.61\% & 0.22\%  & 14.60\% & 12.26\% & 42.38\% & 95.25\% \\
0.5-Bottom &51.55\% & 17.61\% & 62.69\% & 53.82\% & 62.73\% & 94.43\% \\
0.3-Bottom &71.63\% & 48.03\% & 80.94\% & 75.91\% & 75.72\% & 93.46\% \\
\hline
\multicolumn{1}{c}{DenseNet-121}&14.53\% & 0.60\%  & 2.98\%  & 7.79\%  & 31.57\% & 95.53\% \\
0.5-Bottom &35.07\% &  8.01\% & 34.21\% & 30.86\% & 45.28\% & 94.34\% \\
0.3-Bottom &58.19\% & 33.86\% & 62.32\% & 59.74\% & 62.09\% & 92.82\% \\
\hline
\multicolumn{1}{c}{SENet-18}&6.72\%  & 0.90\%  & 12.29\% & 2.23\%  & 26.86\% & 95.09\% \\
0.5-Bottom&52.95\% & 30.78\% & 66.81\% & 52.49\% & 57.45\% & 93.53\% \\
0.3-Bottom&74.73\% & 59.42\% & 84.31\% & 78.72\% & 75.04\% & 92.54\% \\
\hline
\multicolumn{1}{c}{VGG-19}&33.46\% & 14.16\% & 49.76\% & 29.98\% & 21.20\% & 93.93\% \\
0.5-Bottom&72.27\% & 59.70\% & 83.50\% & 77.93\% & 66.75\% & 91.73\% \\
0.3-Bottom&85.53\% & 79.20\% & 91.01\% & 88.51\% & 81.92\% & 90.11\% \\
 \bottomrule
\end{tabular}
}
\end{center}
\caption{Black-box defense performances against transfer-based attacks. Numbers in the middle mean the success defense rates. Networks in the first row are the source models for generating adversarial examples by PGD. 0.5-Bottom and 0.3-Bottom mean applying defective convolutional layers with keep probability $0.5$ and $0.3$ to the bottom layers of the network whose name lies just above them. The source and target networks are initialized differently if they share the same architecture. Detailed settings are listed in Section~\ref{sec:Transfer-based Attack}.\label{Table cifar2}}
\end{table}

\vspace{-0.1in}
\subsection{Decision-based Attack}
\label{sec:Decision-based}
\subsubsection{Experimental Settings}
In this subsection, we evaluate the defense performance of networks with defective convolutional layers against the decision-based attack. Decision-based attack performs based on the prediction of the model. It needs less information from the model and has the potential to perform better against adversarial defenses based on gradient masking. Boundary attack~\cite{brendel2017decision} is one effective decision-based attack. The attack will start from a point that is already adversarial by applying a large scale perturbation to the original image and keep decreasing the distance between the original image and the adversarial example by random walks. After iterations, we will get the final perturbation, which has a relatively small scale. The more robust the model is, the larger the final perturbation will be.

In our experiments, we use the implementation of boundary attack in Foolbox ~\cite{rauber2017foolbox}. It finds the adversarial initialization by simply adding large scale uniform noises on input images. We perform our method on ResNet-18 and test the performance on CIFAR-10 with 500 samples from the validation set. The 5-block structure of ResNet-18 is shown in Appendix Figure 2. The blocks
are labeled $0, 1, 2, 3, 4$ and the $0^{\text{th}}$ block is the first convolution layer. We apply the defective layer structure with keep probability $p=0.1$ to the bottom blocks (the $0^{\text{th}},1^{\text{st}},2^{\text{nd}}$ blocks). For comparison, we implement label smoothing~\cite{szegedy2016rethinking} with smoothing parameter $\epsilon = 0.1$ on a standard ResNet-18, and spatial dropout and drop block with the setting same as Section~\ref{sec:Transfer-based Attack}.

\subsubsection{Experimental Results}
We use the median squared to evaluate the performance, which is defined as $\ell_2$-distance of final perturbation across all samples proposed in \cite{brendel2017decision}. The score $S(M)$ is defined
as $\median_i \left(\frac{1}{N} \|P_i^M\|^2_2\right)$
, where $P_i^M \in \mathbb{R}^N$ is the final perturbation that the Boundary attack finds on model $M$ for the $i^{\text{th}}$ image. Before computing $P_i^M$, the images are normalized into $[0,1]^N$.

From the results in Table~\ref{Table decision}, we point out that spatial dropout and drop block can not enhance the robustness against the boundary attack. Neither does the label smoothing technique. This is consistent with the discovery in Section~\ref{sec:Transfer-based Attack}, and in \cite{papernot2016towards} where they point out that label smoothing is a kind of gradient masking method.
Also, the defective CNN achieves higher performance over the standard CNN.

\begin{table}[h]
\begin{center}
\begin{small}
\renewcommand\tabcolsep{5.0pt}
\begin{tabular}{cc}
\toprule
\bf Model  &\bf $S(M)$
\\ \midrule
Standard CNN & 7.3e-06
\\
Standard CNN + SD~\cite{tompson2015efficient}&7.2e-06
\\
Standard CNN + DB~\cite{ghiasi2018dropblock}&6.1e-06
\\
Standard CNN + LS~\cite{szegedy2016rethinking}&6.8e-06
\\
Defective CNN &\bf 3.5e-05
\\ \bottomrule
\end{tabular}
\end{small}
\end{center}
\caption{Black-box defense performances against decision-based attack. $S(M)$ is defined above. The larger value $S(M)$ has, the more robust the model is. SD, DB, and LS denote spatial dropout, drop block, and label smoothing, respectively. Detailed settings are listed in Section~\ref{sec:Decision-based}.
\label{Table decision}}
\end{table}

\vspace{-0.2in}
\subsection{Additive Gaussian Noise}
\subsubsection{Experimental Settings}
In this subsection, we evaluate the defense performance of networks with defective convolutional layers against additive Gaussian noise. Recently, \cite{ford2019adversarial} bridge the adversarial robustness and corruption robustness, and points out that a successful adversarial defense method should also effectively defense against additive Gaussian noise.
Also the Gaussian noises usually do not change the shape of objects, our models should have better defense performance. To see whether our structure is more robust in this setting, we feed input images with additive Gaussian noises to both standard and defective CNNs.

To obtain noises of scales similar to the adversarial perturbations, we generate i.i.d. Gaussian random variables \(x\sim N(0, \sigma^2)\), where \(\sigma\in\{1,2,4,8,12,16,20,24\) \(,28,32\}\), clip them to the range \([-2\sigma,2\sigma]\) and then add them to every pixel of the input image. Note that, the magnitude range of Gaussian noises used in our experiments covers all 5 severity levels used in \cite{hendrycks2018benchmarking}.
For CIFAR-10, we add Gaussian noises to $5000$ samples which are drawn randomly from the validation set and can be classified correctly by all the tested models. We place the defective layers with keep probability $p = 0.1$ on ResNet-18 in the same way as we did in Section~\ref{sec:Decision-based}.

\label{subsec:On the Robustness of Random Mask}
\subsubsection{Experimental Results}
The experimental results are shown in Figure~\ref{RandomNoise}. The standard CNN is still robust to small scale Gaussian noise such as $\sigma \leq 8$. After that, the performance of the standard CNN begins to drop sharply as $\sigma$ increases. In contrast, defective CNNs show far better robustness than the standard version. The defective CNN with keep probability $0.1$ can maintain high accuracy until $\sigma$ increase to $16$ and have a much slower downward trend as $\sigma$ increases.

\vspace{-0.1in}
\begin{figure}[h]
\centering
\includegraphics[width=2.7in]{./img/randnoise_3.png}
\caption{Defense performance against additive Gaussian noise. $p\text{-Bottom}$ means applying defective convolutional layers with keep probability $p$ to the bottom layers of a standard ResNet-18.
\label{RandomNoise}}
\end{figure}

\vspace{-0.2in}
\subsection{Ablation Studies}
\label{subsec:On the Properties of Random Mask}
There are several design choices of the defective CNN, which include the appropriate positions to apply defective convolutional layers, the benefit of breaking symmetry, the diversity introduced by randomness, as well as the extensibility of defective layers via structure adjustment. In this subsection, we conduct a series of comparative experiments and use black-box defense performance against transfer-based attacks as the evaluation criterion. In our experiments, we found that the performance is not sensitive to the choices on the source model to attack and the target model to defense. Without loss of generality, we only list the performances using DenseNet-121 as the source model and ResNet-18 as the target model on the CIFAR-10 dataset and leave more experimental results in Appendix~\ref{Appendix: resnet18 experiments}. The results are listed in Table~\ref{Tabel ablation}.

\begin{table}[h]
\begin{center}
\begin{small}
\renewcommand\tabcolsep{4.0pt}
\begin{tabular}{cccccc}
\toprule
\multicolumn{1}{c}{\bf Architecture} &\bf{$\text{FGSM}_{16}$} &\bf $\text{PGD}_{16}$ &\bf $\text{PGD}_{32}$ &\bf
$\text{CW}_{40}$
&\bf Test Accuracy\\%\specialrule{0em}{1pt}{1pt}
\midrule
ResNet-18 &14.91\% & 14.14\% & 7.16\%  & 8.23\%  & 95.33\% \\\hline\specialrule{0em}{1pt}{1pt}
\multicolumn{1}{l}{0.7-Bottom}& 23.29\% & 51.29\% & 37.00\% & 36.95\% & 94.03\% \\
\multicolumn{1}{l}{0.5-Bottom}& 30.86\% & 70.38\% & 56.36\% & 54.02\% & 93.39\% \\
\multicolumn{1}{l}{0.3-Bottom}&48.57\% & 86.99\% & 78.41\% & 73.70\% & 91.83\% \\
\multicolumn{1}{r}{0.7-Top} &14.62\% & 10.55\% & 4.91\%  & 7.88\%  & 95.16\% \\
\multicolumn{1}{r}{0.5-Top} &10.76\% & 11.06\% & 5.10\%  & 7.19\%  & 94.94\% \\
\multicolumn{1}{r}{0.3-Top} &11.23\% & 11.77\% & 5.80\%  & 10.10\% & 94.61\% \\
0.7-Bottom, 0.7-Top &24.15\% & 45.12\% & 30.24\% & 29.65\% & 94.16\% \\
0.7-Bottom, 0.3-Top &11.26\% & 33.67\% & 20.28\% & 23.31\% & 93.44\% \\
0.3-Bottom, 0.3-Top &27.43\% & 75.49\% & 62.78\% & 62.47\% & 89.78\% \\
0.3-Bottom, 0.7-Top &40.58\% & 82.77\% & 72.15\% & 68.58\% & 91.23\% \\\hline
0.5-Bottom &30.86\% & 70.38\% & 56.36\% & 54.02\% & 93.39\% \\
0.1-Bottom &79.93\% & 96.70\% & 94.68\% & 89.67\% & 87.68\% \\\hline
$\text{0.5-Bottom}_{\text{DC}}$ &12.15\% & 19.93\% & 11.20\% & 12.72\% & 95.12\% \\
$\text{0.1-Bottom}_{\text{DC}}$ &19.00\% & 53.87\% & 41.40\% & 44.80\% & 93.27\% \\\hline
$\text{0.5-Bottom}_{\text{SM}}$ &48.86\% & 85.00\% & 75.60\% & 72.07\% & 92.57\% \\
$\text{0.1-Bottom}_{\text{SM}}$ &39.40\% & 80.36\% & 72.43\% & 65.38\% & 74.28\% \\\hline
$\text{0.5-Bottom}_{\times 2}$  &20.78\% & 50.16\% & 34.24\% & 34.00\% & 94.12\% \\
$\text{0.1-Bottom}_{\times 2}$  &68.83\% & 93.35\% & 88.25\% & 82.74\% & 90.49\% \\
\bottomrule
\end{tabular}
\end{small}
\end{center}
\caption{
Ablation experiments of defective CNNs. Numbers in the middle mean the success defense rates. \textbf{$p\text{-Bottom}$} and \textbf{$p\text{-Top}$} mean applying defective layers with keep probability $p$ to bottom layers and top layers respectively. \textbf{$p\text{-Bottom}_{\text{DC}}$} means making whole channels defective with keep probability $p$. \textbf{$p\text{-Bottom}_{\text{SM}}$} means using the same defective mask in every channel with keep probability $p$. \textbf{$p\text{-Bottom}_{\times n}$} means increasing channel number to $n$ times at defective layers.
$\text{FGSM}_{16}$, $\text{PGD}_{16}$ and $\text{PGD}_{32}$ denote attack method FGSM with perturbation scale $\ell_\infty = 16$, PGD with perturbation scale $\ell_\infty = 16$ and $32$ respectively. $\text{CW}_{40}$ denotes CW attack method~\cite{DBLP:journals/corr/CarliniW16a} with confidence $\kappa = 40$.
\label{Tabel ablation}}
\end{table}

\textbf{Defective Layers on Bottom layers v.s Top Layers.}
\label{subsec:Bottom Layer Mask versus Top Layer Mask}
We apply defective layers with different keep probabilities to the bottom layers and the top layers of the original CNN, respectively. Comparing the results of the models with the same keep probability but different parts being masked, we find that applying defective layers to \emph{bottom} layers enjoys significantly higher success defense rates. Moreover, only applying defective layers to bottom layers can achieve better performance than applying defective layers on both the bottom and top layers. The bottom layers mainly contribute to detect the edges and shape, while the receptive fields of neurons in top layers are too large to respond to the location sensitive information. This corroborates the phenomena shown in \cite{zeiler2014visualizing,mordvintsev2015inceptionism}. Also, we find that the defense accuracy monotonically increases as the test accuracy decreases along with the keep probability (See the trend map in Appendix~\ref{appendix:trendmap}). The appropriate value for the keep probability mainly depends on the relative importance of generalization and robustness.

\textbf{Defective Neuron v.s. Defective Channel.}
\label{subsec:Random Mask versus Channel Mask} As our method independently selects defective neurons on different channels in a layer, we break the symmetry of the original CNN structure. To see whether this asymmetric structure would help, we try to directly mask whole channels instead of neurons using the same keep probability as the defective layer and train it to see the performance. This defective channel method does not hurt the symmetry while also leading to the same decrease in the number of convolutional operations. Table~\ref{Tabel ablation} shows that although our defective CNN suffers a small drop in test accuracy due to the low keep probability, we have a great gain in the robustness, compared with the defective-channel CNN.

\textbf{Defective Masks are Shared Among Channels or Not.}
\label{subsec:Random Mask versus Same Mask} The randomness in generating masks in different channels and layers allows each convolutional filter to focus on different input patterns. Also, it naturally involves various topological structures for local feature extraction instead of learning~\cite{dai2017deformable,chang2018structure}. We show the essentiality of generating various masks per layer via experiments that compare to a method that only randomly generates one mask per layer and uses it in every channel. Table~\ref{Tabel ablation} shows that applying the same mask to each channel will decrease the test accuracy. This may result from the limitation of expressivity due to the monotone masks at every channel of the defective layer.

\textbf{Increase the Number of Channels at Defective Layers.}
Although masking neurons does not reduce the parameters of the CNN, it reduces the number of convolutional operations and may decrease the expressive capacity of the CNN. To compensate for these defective positions, we increase the number of neurons at the defective layers by increasing the number of channels. Table~\ref{Tabel ablation}  shows that increasing channels does help the network with defective layers to obtain higher test accuracy while maintaining good robustness performance.

\section{Conclusion}
In this paper, we introduce and experiment on defective CNNs, a modified version of existing CNNs that makes CNNs capture more information other than local textures, especially the shape. We show that defective CNNs can achieve much better robustness while maintaining high test accuracy. More specifically, by using defective convolutional layers, we reach state-of-the-art performance against two transfer-based attack methods. Another insight resulting from our experiments is that the adversarial perturbations generated against defective CNNs can actually change the semantic information of images and may even “fool” humans. We hope that these findings bring more understanding on adversarial examples and the robustness of neural networks.

\bibliographystyle{splncs04}