text:
\section{Introduction}
The success of deep learning in computer vision is largely driven by Convolutional Neural Networks (CNNs). Starting from the milestone work AlexNet \cite{Krizhevsky2012}, CNNs keep pushing the frontier of computer vision and laying the foundation of modern recognition systems \cite{Simonyan2015,He2016,Tan2019}. Interestingly, the recently emerged Vision Transformer (ViT) \cite{dosovitskiy2020image} challenges the leading position of CNNs in computer vision. ViT offers a completely different roadmap on building recognition models---by applying the pure self-attention-based architecture to sequences of images patches, ViTs are able to attain competitive performance on a wide range of visual benchmarks compared to CNNs.

The recent studies on out-of-distribution robustness  \cite{bai2020vitsVScnns,zhang2021delving,paul2021vision} further heats up the debate between CNNs and Transformers. Unlike the situation on standard visual benchmarks where both models are closely matched, Transformers are much more robust than CNNs when testing out of the box. Moreover, Bai \emph{et al}\onedot \cite{bai2020vitsVScnns} argue that, rather than being benefited by the advanced training recipe provided in \cite{touvron2020training}, such strong out-of-distribution robustness comes inherently with the Transformer's self-attention-like architecture. For example, simply ``upgrading'' a pure CNN to a hybrid architecture (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, with both CNN blocks and Transformer blocks) can effectively improve out-of-distribution robustness \cite{bai2020vitsVScnns}.

\begin{figure}[t!]
\centering
\subfigure{
\includegraphics[width=0.233\textwidth]{figures/fig1_1_v5.pdf}}
    \subfigure{
\includegraphics[width=0.233\textwidth]{figures/fig1_2_v6.pdf}}
    \subfigure{
\includegraphics[width=0.233\textwidth]{figures/fig1_3_v5.pdf}}
\subfigure{
\includegraphics[width=0.233\textwidth]{figures/fig1_4_v6.pdf}}
\vspace{-1.5em}
\caption{Comparison of out-of-distribution robustness among ResNet, DeiT, and our enhanced ResNet (dubbed \emph{Robust-ResNet}). Though DeiT-S largely outperforms the vanilla ResNet, it performs worse than Robust-ResNet on these robustness benchmarks.}
\vspace{-1.3em}
\label{fig:best_results}
\end{figure}

Though it is generally believed that the architecture difference is the key factor that leads to the robustness gap between Transformers and CNNs, existing works all fail to answer which \emph{architectural elements} in Transformer should be attributed to such stronger robustness. The most relevant analysis is provided in \cite{bai2020vitsVScnns,shao2021adversarial}---both works point out that Transformer blocks, where self-attention operation is the pivot unit, are critical for robustness. Nonetheless, given a) Transformer block itself is already a compound design with multiple architectural elements, and b) Transformer also contains many other layers (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, patch embedding layer) in addition to Transformer blocks, the relationship between robustness and Transformer's architectural elements remains confounding. In this work, we take a closer look at the architecture design of Transformers, focusing on out-of-distribution robustness. More importantly, we aim to explore, with the help of the architectural elements from Transformers, \emph{whether CNNs can be robust learners as well}.

Our diagnose delivers three key messages for improving out-of-distribution robustness, from the perspective of neural architecture design. Firstly, patchifying images into non-overlapped patches can substantially contribute to out-of-distribution robustness; more interestingly, regarding the choice of patch size, we find the larger the better. Secondly, despite applying small convolutional kernels is a popular design recipe, we observe adopting a much larger convolutional kernel size (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, from $3\times3$ to $7\times7$, or even to $11\times11$) is necessary for securing model robustness on out-of-distribution samples. Lastly, as inspired by the recent work \cite{convnext}, we note aggressively reducing the number of normalization layers and activation functions is beneficial for out-of-distribution robustness; meanwhile, as a byproduct, the training speed could be accelerated by up to \raise.17ex\hbox{$\scriptstyle\sim$} 23\%, due to less normalization layers are used \cite{brock2021high}.

Our experiments verify that all these three architectural elements consistently and effectively improve out-of-distribution robustness on a set of CNN architectures.
The largest improvement is reported by integrating all of them into CNNs' architecture design---as shown in Figure \ref{fig:best_results}, without applying any self-attention-like components, our enhanced ResNet (dubbed Robust-ResNet) is able to outperform a similar-scale Transformer, DeiT-S, by 2.4\% on Stylized-ImageNet (16.2\% \vs 18.6\%), 0.5\% on ImageNet-C (42.8\% \vs 42.3\%), 4.0\% on ImageNet-R (41.9\% \vs 45.9\%) and 3.9\% on ImageNet-Sketch (29.1\% \vs 33.0\%). We hope this work can
help the community better understand the underlying principle of designing robust neural architectures.

\section{Related Works}

\paragraph{Vision Transformers.}
Transformers \cite{Vaswani2017}, which apply self-attention to enable global interactions between input elements, underpins the success of building foundation models in natural language processing
\cite{devlin2018bert,yang2019xlnet,dai2019transformer,radford2018improving,radford2019language,brown2020language,bommasani2021opportunities}. Recently, Dosovitskiy \emph{et al}\onedot \cite{dosovitskiy2020image} show Transformer is also applicable to computer vision, attaining competitive performance compared to CNNs on the challenging ImageNet classification task. Later works keep pushing the potential of Transformer on a variety of visual tasks, in both supervised learning \cite{touvron2020training,yuan2021tokens,liu2021swin,wang2021pyramid,yuan2021tokens,zhai2021scaling,touvron2021going,xue2021go} and self-supervised learning \cite{caron2021emerging,chen2021empirical,beit,zhou2021ibot,xie2021self,MaskedAutoencoders2021}, showing a seemly inevitable trend on replacing CNNs in computer vision.

\paragraph{CNNs striking back.}
There is a recent surge in work that aims to retake the position of CNNs as the favored neural architecture for building recognition models. Wightman \emph{et al}\onedot \cite{wightman2021resnet} demonstrate that, with the advanced training setup, the canonical ResNet-50 is able to achieve 80.4\% top-1 accuracy on ImageNet, largely outperforming its vanilla version (76.1\% top-1 accuracy). In addition, the prior works show modernizing CNNs' architecture design is essential---by either simply adopting Transformer's patchify layer \cite{convmixer} or aggressively bringing in every possible components from Transformer \cite{convnext}, CNNs are able to attain competitive performance w.r.t\onedot} \def\dof{d.o.f\onedot Transformers across a range of visual benchmarks.

Our work is closely related to ConvNeXt \cite{convnext}, while shifting the study focus from standard accuracy to robustness. Moreover, rather than specifically offering a unique neural architecture as in \cite{convnext}, this paper aims to provide a set of useful architectural elements that allows CNNs to be able to match, or even outperform, Transformers when measuring robustness.

\paragraph{Out-of-distribution robustness.}
Dealing with data of shifted distributions is a commonly encountered problem when deploying models in the real world. To simulate such challenges, several out-of-distribution benchmarks have been established, including measuring model performance on images with common corruptions \cite{Hendrycks2018} or with various renditions \cite{hendrycks2021many}. Recently, a set of works \cite{bai2020vitsVScnns,zhang2021delving,paul2021vision} find that Transformers are inherently much more robustness than CNNs on out-of-distribution samples. Our work is a direct follow-up of Bai \emph{et al}\onedot \cite{bai2020vitsVScnns}, whereas stands on the opposite side---\emph{we show CNNs can in turn outperform Transformers in out-of-distribution robustness}.

\section{Settings}
Following the previous work \cite{bai2020vitsVScnns}, in this paper the robustness comparison is fairly conducted between CNNs and Transformers based on the architecture of ResNet \cite{He2016} and ViT \cite{dosovitskiy2020image}, two of the most classic milestone architectures in the deep learning history.

\paragraph{CNN Block Instantiations.}
We consider four different block instantiations, as shown in Fig. \ref{fig:Block Instantiations}. The first block is Depth-wise ResNet Bottleneck block \cite{resnext}, in which the 3$\times$3 vanilla convolution layer in each block is replaced with a 3$\times$3 depth-wise convolution layer.
The second block is Inverted Depth-wise ResNet Bottleneck block, a design first proposed in \cite{mobilenetv2}, which is the same as Depth-wise ResNet Bottleneck block except that the hidden dimension is four times the input dimension. The third block is instantiated based on the second block by moving up the position of depthwise convolution layer as in ConvNeXT\cite{convnext}, which reduces the computation of the more complex 3$\times$3 depthwise convolution layer and puts more weight on the 1$\times$1 convolution layer.
Note that this block design is similar to a ConvNeXT block \cite{convnext}. But a ConvNeXT block comes with other modifications such as changes of activation and normalization layers.  Similarly, based on the second block, the forth block moves down the position of depthwise convolution layer.

Note that in all four blocks, depth-wise convolution are adopted, since numerous studies show that it achieves a better computation/accuracy trade-off  than a vanilla convolution \cite{mobilenet,mobilenetv2,mobilenetv3,efficientnet}. We replace the bottleneck building block in the original ResNet architecture with these four blocks, and term the resulting models as ResNet-DW, ResNet-Inverted-DW, ResNet-Up-Inverted-DW, and ResNet-Down-Inverted-DW, respectively.

\begin{figure*}[htb]
\vspace{-1.5em}
    \centering
    \includegraphics[width=\linewidth]{figures/block_instantiations.pdf}
\vspace{-2.5em}
\caption{Architectures of four different instantiations of CNN block.}
\label{fig:Block Instantiations}
\vspace{-1em}
\end{figure*}

\begin{table}[t!]
\centering
\caption{Comparison of various baseline models. We replace the ResNet Bottleneck block with four different stronger building blocks, and keep the computational cost of baseline models on par with DeiT counterparts. We observe significant gain in terms of clean image accuracy, but our CNN baseline models are still less robust than DeiT.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G      &79.7           &10.6    &50.1     &41.3      &29.1   \\
ResNet-Inverted-DW        &4.6G     &80.0           &10.8    &47.7     &41.9      &29.4   \\
ResNet-Up-Inverted-DW      &4.7G    &79.7           &12.9    &47.9     &42.9      &30.8   \\
ResNet-Down-Inverted-DW    &4.5G    &79.6           &12.1    &49.0     &42.5   &29.5\\
\bottomrule
\end{tabular}
}
\label{Experiment: Baseline Results}
\vspace{-1em}
\end{table}

\paragraph{Computational Cost.} In this work, we consider measuring the model size in terms of FLOPs, i.e., FLOating Point operations per second.
Here we note that directly replacing the ResNet bottleneck block with the above four instantiated blocks will significantly reduce the total FLOPs of the model, due to the usage of the depth-wise convolution.
To mitigate the computational cost loss, and to increase model performance following the spirit of ResNeXT \cite{resnext}, we change the channel setting of each stage from (64, 128, 256, 512) to (96, 192, 384, 768).
We then tune the block number in stage 3 to keep the total FLOPs of baseline models roughly the same as DeiT-S.
The final FLOPs of our baseline models are shown in Tab. \ref{Experiment: Baseline Results}.
Unless otherwise stated, all models considered in this work are of the similar scale as DeiT-S.

\paragraph{Robustness Benchmarks.} In this work, we extensively evaluate the model performance on out-of-distribution robustness using following benchmarks: 1) Stylized-ImageNet\cite{Geirhos2018}, which contains synthesized images with shape-texture conflicting cues; 2) ImageNet-C\cite{Hendrycks2018}, with various common image corruptions; 3) ImageNet-R \cite{hendrycks2021many}, which contains natural renditions of ImageNet object classes with different textures and local image statistics; 4) ImageNet-Sketch \cite{imagenetsketch}, which includes sketch images of the same ImageNet classes collected online.

\paragraph{Training Recipe.} CNNs may achieve better robustness by simply tuning the training recipe \cite{bai2020vitsVScnns,wightman2021resnet}.
Therefore in this work we deliberately apply the standard 300-epoch DeiT training recipe \cite{touvron2020training} for all models unless otherwise mentioned, so that the performance difference between different models can only be attributed to the architecture difference.
Specifically, we train all models using AdamW optimizer \cite{adamw};
We set the initial learning rate to 5e-4, and apply the cosine learning rate scheduler to decrease it; Besides weight decay, we additionally adopt six data augmentation \& regularization strategies (i.e., RandAug \cite{randaugment},
MixUp \cite{mixup} ,CutMix \cite{cutmix}, Random Erasing \cite{randomerasing}, Stochastic Depth \cite{stochasticdepth}, and Repeated Augmentation \cite{repeatedaugment}) to regularize training.

\paragraph{Baseline Results.} We train six models with the recipe specified above. The results are shown in Tab. \ref{Experiment: Baseline Results}. For simplicity, we use ``IN'', ``S-IN'', ``IN-C'', ``IN-R'', and ``IN-SK'' as abbreviations for ``ImageNet'', ``Stylized-ImageNet'', ``ImageNet-C'', ``ImageNet-R'', and ``ImageNet-Sketch''.
We already know that DeiT-S shows better robustness generalization than ResNet50 from \cite{bai2020vitsVScnns}. Interestingly, even when equipped with stronger depth-wise convolution layers, the ResNet architecture can only attain similar clean image accuracy while still being non-trivially less robust compared with the DeiT counterpart, suggesting the hidden magic lies in the architecture design of Vision Transformer.

\section{Component Diagnosis}
In this section, we introduce three effective architectural design inspired from Transformers which work surprisingly well on CNNs as well.
Concretely, they are 1) patchifying input images (Section~\ref{sec:patchifying}),
b) enlarging the kernel size (Section~\ref{sec:large_kernel}), and finally 3) reducing the number of activation layers and normalization layers (Section~\ref{sec:removing_act_norm}), all of which are conducted under a fair standard.
Especially, the first design has a non-negligible effect on network FLOPs, which we mitigate by adding back in the third stage.
Next, we will reveal how combining all of them delivers a pure CNN architecture that is as robust as, or even more
robust than, Transformers.

\begin{table}[t!]
\centering
\vspace{-.5em}
\caption{The performance of different baseline models equipped with ViT-style patchify stem. ``PX'' refers to that the model has a ViT-style patchify stem with patch size set to X.
The block number in stage 3 is increased when patch size is larger. With larger patch size, the models attain better robustness but worse clean accuracy. The results of the original ResNet50 and DeiT-S are included for reference.}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1   &41.3      &29.1   \\
\rowcolor{mygray}
+ P4             &4.6G    &79.6     &11.4    &51.2    &40.5      &28.3   \\
\rowcolor{mygray}
+ P8             &4.6G    &79.9     &12.7    &47.6    &42.5      &30.1   \\
\rowcolor{mygray}
+ P16            &4.6G    &78.5     &\textbf{17.0}    &\textbf{44.3}    &\textbf{44.8}      &\textbf{32.0}   \\
\midrule
ResNet-Inverted-DW                &4.6G    &80.0     &10.8    &47.7    &41.9      &29.4   \\
\rowcolor{mygray}
+ P4             &4.5G    &80.2     &11.2    &48.8    &41.5      &29.8   \\
\rowcolor{mygray}
+ P8             &4.6G    &79.8     &14.8    &45.4    &44.0      &31.8   \\
\rowcolor{mygray}
+ P16            &4.6G    &77.9     &\textbf{17.2}    &\textbf{42.5}    &\textbf{44.9}      &\textbf{32.4}   \\
\midrule
ResNet-Up-Inverted-DW                &4.7G    &79.7     &12.9    &47.9    &42.9      &30.8  \\
\rowcolor{mygray}
+ P4             &4.5G    &80.0     &13.5    &48.9    &42.3      &30.7   \\
\rowcolor{mygray}
+ P8             &4.7G    &79.8     &15.7    &46.3    &44.0      &31.4   \\
\rowcolor{mygray}
+ P16            &4.5G    &77.9     &\textbf{17.0}    &\textbf{43.9}    &\textbf{44.6}      &\textbf{32.0}   \\
\midrule
ResNet-Down-Inverted-DW                &4.5G    &79.6     &12.1    &49.0    &42.5   &29.5   \\
\rowcolor{mygray}
+ P4             &4.4G    &79.8     &12.4    &48.6    &41.8      &29.4   \\
\rowcolor{mygray}
+ P8             &4.5G    &79.6     &15.2    &46.2    &43.5      &\textbf{30.7}   \\
\rowcolor{mygray}
+ P16            &4.6G    &78.0     &\textbf{16.2}    &\textbf{43.8}    &\textbf{43.8}      &30.6   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Patch Embedding Ablation}
\vspace{-1em}
\end{table}

\subsection{Patchify Stem}
\label{sec:patchifying}
A CNN or a Transformer usually down-samples the input image to a proper feature map size at the beginning of the network. A standard ResNet employs a 7$\times$7 convolution layer with stride 2, followed by a 3$\times$3 max pooling with stride 2, resulting in a 4$\times$ resolution reduction. ViT, however, adopts a much more aggressive downsampling strategy. It partitions an input image into p$\times$p non-overlapping patches and map each patch to a $d$-dimension feature vector with a trainable linear projection. In practice, this step is implemented by a convolution layer with kernel size $p$ and stride $p$, where $p$ is typically set to 16. Here we refer to the layers before typical ResNet block in ResNet, or the layer before self-attention block in ViT, as \textit{stem}. Some works argue for the importance of stem setup in CNN \cite{convmixer} or Transformers \cite{convstem}, but none of them investigate this module in the prospective of robustness. We replace the ResNet-style stem in the baseline models with the ViT-style patchify stem, a convolution layer with kernel size $p$ and stride $p$, where $p$ varies from 4 to 16. We keep the total stride of the model fixed (so a 224$\times$224 input image will always lead to a 7$\times$7 feature map before the final global pooling layer), which is achieved by setting stride=1 in the first block of some stages. Specifically, the original ResNet sets stride=2 for the first block in stage 2, 3, 4.
We set stride=1 for the first block in stage 2 when employing 8$\times$8 patchify stem, and set stride=1 for the first block in stage 2 and 3 when employing 16$\times$16 patchify stem.
To ensure a fair comparison, we add some blocks in stage 3 to keep the total FLOPs roughly the same as before.

\begin{table}[t!]
\centering
\caption{The performance of different baseline models equipped with ViT-style patchify stem and convolution stem. ``PX'' refers to that the model has a ViT-style patchify stem with patch size set to X; ``C'' refers to that it has a convolutional stem. }
\vspace{-0.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1   &41.3      &29.1   \\
\rowcolor{mygray}
+ P16            &4.6G   &78.5     &\textbf{17.0}    &\textbf{44.3}    &\textbf{44.8}      &\textbf{32.0}   \\
\rowcolor{mygray}
+ C              &4.5G    &79.6     &16.2    &48.4    &43.0      &30.2   \\
\midrule
ResNet-Inverted-DW       &4.6G    &80.0     &10.8    &47.7    &41.9      &29.4   \\
\rowcolor{mygray}
+ P16                    &4.6G    &78.1     &\textbf{17.4}    &\textbf{42.9}    &\textbf{44.7}     &\textbf{32.4}   \\
\rowcolor{mygray}
+ C      &4.6G    &79.9     &16.9    &46.2    &43.2      &31.2   \\
\midrule
ResNet-Up-Inverted-DW    &4.7G    &79.7     &12.9    &47.9    &42.9      &30.8  \\
\rowcolor{mygray}
+ P16                    &4.5G    &77.9     &1\textbf{7.0}    &\textbf{43.9}    &\textbf{44.6}     &\textbf{32.0}   \\
\rowcolor{mygray}
+ C    &4.5G    &79.1     &16.6    &48.4    &43.2      &30.3   \\
\midrule
ResNet-Down-Inverted-DW  &4.5G    &79.6     &12.1    &49.0    &42.5   &29.5   \\
\rowcolor{mygray}
+ P16                    &4.6G    &78.0     &16.2    &\textbf{43.8}    &\textbf{43.8}      &\textbf{30.6}   \\
\rowcolor{mygray}
+ C  &4.6G    &79.3     &\textbf{16.4}    &49.0    &42.4      &29.2   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Conv Stem Ablation}
\vspace{-1em}
\end{table}

We can observe in Tab. \ref{Experiment: Baseline Results} that increasing patch size of the ViT-style patchify stem leads to increased performance on robustness benchmarks, though at the potential cost of clean accuracy.
Specifically, for ResNet-DW, when patch size=8, the performance on all robustness benchmarks are boosted by at least 0.6\%; when patch size=16, the performance on all robustness benchmarks are boosted by at least 1.2\%, and the biggest improvement is 6.6\% on Stylized-ImageNet.
Surprisingly, this simple operation demonstrates significant potential for closing the robustness gap between CNNs and Transformers,
suggesting that a large portion of the robustness of ViT comes from the patchify stem.

Recent findings also suggest replacing the ViT-style patchify stem with a small number of stacked 2-stride 3$\times$3 convolution layers can greatly ease optimization and thus boost the clean accuracy \cite{graham2021levit,convstem}. To verify its effectiveness on robustness benchmarks, we also implemented the convolution stem for ViT-S in \cite{graham2021levit}, with a stack of 4 3$\times$3 convolution layers with stride 2.
The results are shown as follows in Tab. \ref{Experiment: Conv Stem Ablation}. Surprisingly, though the use of convolutional stem does attain higher clean accuracy, it appears to be not as helpful as ViT-style patchify stem on out-of-distribution robustness. It suggests that the clean accuracy and out-of-distribution robustness are not always positively correlated. And useful design for improving clean accuracy may not necessary attribute to better robustness, which again, shows the importance of searching for methods that can boost robustness in addition to solely clean accuracy.

\subsection{Large Kernel Size}
\label{sec:large_kernel}
One of the most critical properties of self-attention operation that distinguish it from the classic convolution operation, is that it operates on the whole input image or feature map and thus has a global receptive field.
Even before the rise of Vision Transformer, the importance of capturing long-range dependency has been demonstrated for CNNs as well.
One typical example is the Non-local Neural Network which has been show to be highly effective for both static and sequential image recognition even when equippped with only one non-local block~\cite{nonlocal}.
Still, the most popular design is to stack many 3$\times$3 convolution layers to gradually increase the network receptive field as it goes deeper.

In this section, we try to mimic the behaviour of the self-attention block, by increasing the kernel size of the depth-wise convolution layer.
We experiment with various kernel sizes: 5, 7, 9, 11, and 13. The results on different robustness benchmarks are shown as in Fig. \ref{fig: Kernel Size Ablation}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/point_all_v7.pdf}
    \vspace{-2.5em}
    \caption{Robustness evaluation of model with varying kernel size. Different colors and shapes denote different models.}
    \vspace{-1em}
    \label{fig: Kernel Size Ablation}
\end{figure}

From Fig. \ref{fig: Kernel Size Ablation}, we observe that larger kernel sizes generally come with both better clean accuracy and stronger robustness.
We can also see that the performance gain gradually saturates when the kernel size becomes too large.

Since using convolutions of larger kernels will incur notable extra computation, here we would like to highlight that this particular component design would be impossible without the help of depth-wise convolution layer, which significantly reduces the computational cost of large kernel convolution layers in all four blocks. For example, if directly changing the kernel size in ResNet50 from 3 to 5, the total FLOPs of the resulted model would be 7.4G, which is way larger than its Transformer counterpart. Thanks to depth-wise convolution layer, increasing kernel size from 3 to 13 typically only increase the FLOPs by 0.3G, which is relatively small compared to the FLOPs of DeiT-S (4.6G), except for ResNet-Inverted-DW. Because of the large channel dimension in the Inverted Bottleneck design, increasing kernel size from 3 to 13 brings a total of 1.4G FLOPs increase, rendering unfair comparison to some extent.
Nevertheless, the trend for all four architectures is the same. And the extra computational cost brought by large kernel size can be mitigated by employing a patchify stem with large patch size so that our final models will still be on the same scale as DeiT-S (see Sec. \ref{sec:components combination} for the FLOPs of models with multiple proposed designs).

\setlength{\tabcolsep}{2pt}
\begin{table}[t!]
\centering
\caption{The performance of models with differently removed activation layers. We use \cmark to denote keeping a activation layer and \xmark to denote removing a activation layer. The marks here are in order. For example, a \cmark in the first column of the ``ReLU'' column refers to that the first relu activation in the block is keepped; a \xmark in the second column of ``ReLU'' column refers to that the first relu activation in the block is removed.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}
\toprule
\multirow{2}{*}{Architecture} & \multirow{2}{*}{FLOPs}  & \multicolumn{3}{c|}{ReLU}  & \multirow{2}{*}{IN \textcolor{red}{$\uparrow$}}  & \multirow{2}{*}{S-IN \textcolor{red}{$\uparrow$}}  & \multirow{2}{*}{IN-C \textcolor{red}{$\downarrow$}}   & \multirow{2}{*}{IN-R \textcolor{red}{$\uparrow$}} & \multirow{2}{*}{IN-SK \textcolor{red}{$\uparrow$}} \\ \cline{3-5}
&&$1^{st}$&$2^{nd}$&$3^{rd}$&&&&&\\
\midrule
\midrule
ResNet50   &4.1G    & \cmark &\cmark &\cmark              &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S      &4.6G   &\cmark&\cmark&\cmark              &79.8     &16.2    &42.8    &41.9      &29.1   \\
\midrule
\multirow{4}{*}{ResNet-DW}  &\multirow{4}{*}{4.8G}   &\cmark&\cmark&\cmark               &79.7     &10.6    &50.1   &41.3      &29.1   \\
  &&\cmark&\xmark&\xmark              &74.8     &8.3     &58.6    &37.1      &24.5   \\
    & &\xmark&\cmark&\xmark             &73.2    &8.6    &60.5      &36.2   &23.3\\
  &  &\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\cmark             &\cellcolor{mygray}79.4     &\cellcolor{mygray}11.7    &\cellcolor{mygray}50.5    &\cellcolor{mygray}42.8      &\cellcolor{mygray}29.8   \\
\midrule
\multirow{4}{*}{ResNet-Inverted-DW} & \multirow{4}{*}{4.6G}   &\cmark&\cmark&\cmark        &80.0     &10.8    &47.7    &41.9      &29.4   \\
     & &\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark           &\cellcolor{mygray}80.7     &\cellcolor{mygray}12.3    &\cellcolor{mygray}46.4    &\cellcolor{mygray}45.3      &\cellcolor{mygray}31.9   \\
  &  &\xmark&\cmark&\xmark               &80.3     &11.2    &48.5    &44.0      &30.9   \\
  &  &\xmark&\xmark&\cmark          &72.5     &9.7     &59.5    &37.5      &24.7   \\
\midrule
\multirow{4}{*}{ResNet-Up-Inverted-DW} & \multirow{4}{*}{4.7G} &\cmark&\cmark&\cmark      &79.7     &12.9    &47.9    &42.9      &30.8  \\
    &  &\cmark&\xmark&\xmark      &67.8     &9.4     &65.0    &33.0      &19.7  \\
  &&\cellcolor{mygray}\xmark&\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark                 &\cellcolor{mygray}80.1     &\cellcolor{mygray}13.0    &\cellcolor{mygray}47.3    &\cellcolor{mygray}44.2      &\cellcolor{mygray}30.0   \\
  &&\xmark&\xmark&\cmark            &68.7     &8.9     &64.4    &33.4      &20.2   \\
\midrule
\multirow{4}{*}{ResNet-Down-Inverted-DW} &\multirow{4}{*}{4.5G}  &\cmark&\cmark&\cmark    &79.6     &12.1    &49.0    &42.5   &29.5   \\
  & &\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark             &\cellcolor{mygray}80.7     &\cellcolor{mygray}13.0    &\cellcolor{mygray}46.6    &\cellcolor{mygray}45.9      &\cellcolor{mygray}32.8    \\
  &  &\xmark&\cmark&\xmark           &70.1     &9.9     &63.3    &35.2      &21.8   \\
 &    &\xmark&\xmark&\cmark      &68.8     &8.3     &65.1    &33.7      &21.9   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Few Act Ablation}
\vspace{-1em}
\end{table}

\begin{table}[t!]
\centering
\caption{The performance of models with differently removed activation and normalization layer. Note that the results are obtained with the best choice of only one activation layer in a block based on the results of Tab. \ref{Experiment: Few Act Ablation}. For simplicity, we only draw marks to denote if a normalization layer is kept in this table. We use \cmark to denote keeping a normalization layer and \xmark to denote removing a normalization layer. For example, a \cmark in the first column of the ``BN'' column refers to that the first batch normalization in the block is kept; a \xmark in the second column of ``BN'' column refers to that the first batch normalization in the block is removed. It is observed that the optimal choice always lead to best robustness. }
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}
\toprule
\multirow{2}{*}{Architecture} & \multirow{2}{*}{FLOPs}  & \multicolumn{3}{c|}{BN}  & \multirow{2}{*}{IN \textcolor{red}{$\uparrow$}}  & \multirow{2}{*}{S-IN \textcolor{red}{$\uparrow$}}  & \multirow{2}{*}{IN-C \textcolor{red}{$\downarrow$}}   & \multirow{2}{*}{IN-R \textcolor{red}{$\uparrow$}} & \multirow{2}{*}{IN-SK \textcolor{red}{$\uparrow$}} \\ \cline{3-5}
&&$1^{st}$&$2^{nd}$&$3^{rd}$&&&&&\\
\midrule
\midrule
ResNet50   &4.1G    & \cmark &\cmark &\cmark              &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S      &4.6G   &\cmark&\cmark&\cmark              &79.8     &16.2    &42.8    &41.9      &29.1   \\
\midrule
\multirow{4}{*}{ResNet-DW}  &\multirow{4}{*}{4.8G}   &\cmark&\cmark&\cmark               &79.7     &10.6    &50.1   &41.3      &29.1   \\
   &&\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark               &\cellcolor{mygray}79.4     &\cellcolor{mygray}11.6    &\cellcolor{mygray}49.3    &\cellcolor{mygray}43.3      &\cellcolor{mygray}29.9 \\
    & &\xmark&\cmark&\xmark               &79.5     &10.6    &49.5    &43.1      &29.3   \\
   &  &\xmark&\xmark&\cmark   &79.3     &11.1    &50.7    &42.5      &30.2   \\
\midrule
\multirow{4}{*}{ResNet-Inverted-DW} & \multirow{4}{*}{4.6G}   &\cmark&\cmark&\cmark        &80.0     &10.8    &47.7    &41.9      &29.4   \\
     & &\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark           &\cellcolor{mygray}80.4     &\cellcolor{mygray}12.9    &\cellcolor{mygray}47.0    &\cellcolor{mygray}45.8      &\cellcolor{mygray}32.4     \\
   &  &\xmark&\cmark&\xmark               &80.0     &11.2    &49.3    &44.1      &31.2   \\
  &  &\xmark&\xmark&\cmark          &80.1     &13.0    &46.8    &44.3      &30.8   \\
\midrule
\multirow{4}{*}{ResNet-Up-Inverted-DW} & \multirow{4}{*}{4.7G} &\cmark&\cmark&\cmark      &79.7     &12.9    &47.9    &42.9      &30.8  \\
    &  &\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark     &\cellcolor{mygray}80.5     &\cellcolor{mygray}14.3    &\cellcolor{mygray}45.0    &\cellcolor{mygray}47.2      &\cellcolor{mygray}33.1  \\
   &&\xmark&\cmark&\xmark                 &80.4     &12.9    &47.0    &45.8      &32.4     \\
   &&\xmark&\xmark&\cmark              &80.1     &13.0    &46.8    &44.3      &30.8   \\
\midrule
\multirow{4}{*}{ResNet-Down-Inverted-DW} &\multirow{4}{*}{4.5G}  &\cmark&\cmark&\cmark    &79.6     &12.1    &49.0    &42.5   &29.5   \\
  & &\cellcolor{mygray}\cmark&\cellcolor{mygray}\xmark&\cellcolor{mygray}\xmark            &\cellcolor{mygray}80.5     &\cellcolor{mygray}13.6    &\cellcolor{mygray}46.4    &\cellcolor{mygray}45.9      &\cellcolor{mygray}33.3    \\
   &  &\xmark&\cmark&\xmark          &80.7     &13.1    &46.1    &45.5      &31.7    \\
 &    &\xmark&\xmark&\cmark      &80.8     &12.7    &46.5    &46.1      &33.1     \\
\bottomrule
\end{tabular}}
\label{Experiment: Few Norm Act Ablation}
\vspace{-1em}
\end{table}

\subsection{Reducing Activation and Normalization Layers}
\label{sec:removing_act_norm}

A typical Vision Transformer block usually has fewer activation and normalization layers than a ResNet block \cite{dosovitskiy2020image,liu2021swin}, an architecture design found to be very effective in ConvNeXT \cite{convnext}. In this work, we directly borrow the idea of reducing activation layers and normalization layers in ConvNeXT for all four block instantiations, and find it to be beneficial to clean accuracy as well as out-of-distribution robustness. Specifically, in a typical ResNet block, there is one normalization layer and one activation layer following each convolution layer, resulting in a total of three normalization layers and activation layers in one block. In our implementation, for each block, we remove two normalization layers and activation layers, and leave only one normalization layer and activation layer. The results of the models with differently removed activation layers are shown in Tab.~\ref{Experiment: Few Act Ablation}. And the results of the models with differently removed normalization and activation layers are shown in Tab.~\ref{Experiment: Few Norm Act Ablation}. Note that we only ablate the position of removing normalization layers with the optimal position of activation layer shown in Tab.~\ref{Experiment: Few Act Ablation}, since we find that only removing normalization layers does not result in performance improvement in the preliminary experiments.

We empirically find that among all experimental settings, leaving only one activation layer after the convolution layer where the channel dimension expands (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, the number of output channel is larger than that of the input channels), and leaving one normalization layer after the first convolution layer yields the best result.
We also visualize the best choice in Fig. \ref{fig:best few norm act}.
For instance, we observe 1.4\% improvement on Stylized-ImageNet (12.9\% \vs 14.3\%), 2.9\% improvement on ImageNet-C (47.9\% \vs 45.0\%), 4.3\% improvement on ImageNet-R (42.9\% \vs 47.2\%), and 2.3\% improvement on ImageNet-Sketch (30.8\% \vs 33.1\%) for ResNet-Up-Inverted-DW. Additionally, fewer activation layers and normalizations lead to less computation, which helps speep up training (e.g., up to 23\% speed up simply by removing a few normalization layers).

\begin{figure}[t!]
    \centering
    \includegraphics[width=\linewidth]{figures/optimal_block.pdf}
    \vspace{-2.5em}
    \caption{The optimal position of normalization and activation layers for different block architectures.}
    \vspace{-1em}
    \label{fig:best few norm act}
\end{figure}

\section{Components Combination}
\label{sec:components combination}
In this section, we investigate whether all components combined will lead to better results.
We choose a 16$\times$16 patchify stem with patch size, 11$\times$11 kernel size, and the corresponding optimal position for placing normalization and activation layer for all architectures (except for ResNet-Inverted-DW, we use 7$\times$7 kernel size as we empirically find using too large kernel size (e.g., 11$\times$11) can lead to unstable training).
As shown in Tab. \ref{Experiment: Patch Embedding + Large Kernel Size}, Tab.~\ref{Experiment: Patch Embedding + Few Norm Act}, Tab. \ref{Experiment: Large Kernel Size + Few Norm Act}, and Tab.~\ref{Experiment: Patch Embedding + Large Kernel Size + Few Norm Act}, \textit{we can see these simple designs not only work well when they are individually employed with ResNet, but even work better when they are simultaneously employed}. Also, when all three designs are used, we see now ResNet beats DeiT on all four out-of-distribution benchmarks . These results again verify the effectiveness of our proposed architecture design, and also implies that a pure CNN without any self-attention-like block achieves just as good robustness as ViT.
From now on, for simplicity, we will term ResNet-DW with our choice of ViT-style patchify stem, large kernel size, and fewer activation and normalization layers as Robust-ResNet-DW (and similar for others).

\begin{table}[t!]
\centering
\caption{The performance of different baseline models equipped with ViT-style patchify stem and large kernel size. The models with the suffix ``PX'' refers to that it has a ViT-style patchify stem with patch size set to X. ``KX'' refers to that the model uses blocks with a depth-wise convolution layer with kernel size X. }
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1   &41.3      &29.1   \\
\rowcolor{mygray}
+ P16 + K11              &4.5G    &79.0     &18.8    &42.5   &45.2      &32.4   \\
\midrule
ResNet-Inverted-DW       &4.6G    &80.0     &10.8    &47.7    &41.9      &29.4   \\
\rowcolor{mygray}
+ P16 + K7               &4.6G    &79.2     &19.8    &41.1    &46.4      &33.7   \\
\midrule
ResNet-Up-Inverted-DW    &4.7G    &79.7     &12.9    &47.9    &42.9      &30.8  \\
\rowcolor{mygray}
+ P16 + K11              &4.4G    &78.0     &17.2    &43.7    &47.1      &31.2   \\
\midrule
ResNet-Down-Inverted-DW  &4.5G    &79.6     &12.1    &49.0    &42.5   &29.5   \\
\rowcolor{mygray}
+ P16 + K11              &4.6G    &78.1     &18.9    &43.7    &42.9      &29.6   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Patch Embedding + Large Kernel Size}
\end{table}

\begin{table}[t!]
\centering
\caption{The performance of different baseline models equipped with ViT-style patchify stem and less activation and normalization layers. The models with the suffix ``PX'' refers to that it has a ViT-style patchify stem with patch size set to X. ``NormXActY'' refers to that the model has only one normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in the block.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2     &42.8     &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1    &41.3       &29.1   \\
\rowcolor{mygray}
+ P16 + Norm1Act3     &4.6G    &78.7     &16.2    &43.0    &46.3       &33.1   \\
\midrule
ResNet-Inverted-DW       &4.6G    &80.0     &10.8    &47.7    &41.9      &29.4   \\
\rowcolor{mygray}
+ P16 + Norm1Act1     &4.6G    &79.3     &17.5    &41.7    &46.1      &32.9   \\
\midrule
ResNet-Up-Inverted-DW    &4.7G    &79.7     &12.9    &47.9    &42.9      &30.8  \\
\rowcolor{mygray}
+ P16 + Norm1Act2     &4.5G    &79.1     &18.6    &41.5    &48.0      &33.8   \\
\midrule
ResNet-Down-Inverted-DW  &4.5G    &79.6     &12.1    &49.0    &42.5      &29.5   \\
\rowcolor{mygray}
+ P16 + Norm1Act1     &4.6G    &79.7     &18.0    &40.7    &47.1      &33.4   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Patch Embedding + Few Norm Act}
\end{table}

\begin{table}[t!]
\centering
\caption{The performance of different baseline models equipped with large kernel size and less activation and normalization layers. ``KX'' refers to that the model uses blocks with a depth-wise convolution layer with kernel size X. ``NormXActY'' refers to that the model has only one normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in the block.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8      &16.2     &42.8     &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1    &41.3       &29.1   \\
\rowcolor{mygray}
+ K11 + Norm1Act3     &5.0G    &80.4     &13.3    &45.2    &43.9       &30.8   \\
\midrule
ResNet-Inverted-DW       &4.6G    &80.0     &10.8    &47.7    &41.9       &29.4   \\
\rowcolor{mygray}
+ K7 + Norm1Act1      &5.0G    &80.8     &13.1    &46.7        &45.7       &32.1   \\
\midrule
ResNet-Up-Inverted-DW    &4.7G    &79.7     &12.9    &47.9    &42.9       &30.8  \\
\rowcolor{mygray}
+ K11 + Norm1Act2     &4.9G    &81.4     &15.8    &42.9    &48.3       &33.8   \\
\midrule
ResNet-Down-Inverted-DW  &4.5G    &79.6     &12.1    &49.0    &42.5       &29.5   \\
\rowcolor{mygray}
+ K11 + Norm1Act1     &4.6G    &81.3     &14.2    &43.6    &46.1       &32.8   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Large Kernel Size + Few Norm Act}
\end{table}

\begin{table}[t!]
\centering
\caption{The performance of different baseline models equipped with ViT-style patchify stem, large kernel size, and less activation and normalization layers. The models with the suffix ``PX'' refers to that it has a ViT-style patchify stem with patch size set to X. ``KX'' refers to that the model uses blocks with a depth-wise convolution layer with kernel size X. ``NormXActY'' refers to that the model has only one normalization layer after the Xth convolution layer and one activation layer after the Yth convolution layer in the block.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
ResNet50                 &4.1G    &78.4      &9.9      &51.2     &39.2      &27.7   \\
DeiT-S                   &4.6G    &79.8     &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                &4.8G    &79.7     &10.6    &50.1   &41.3      &29.1   \\
\rowcolor{mygray}
+ P16 + K11 + Norm1Act3             &4.5G    &79.4     &18.6    &42.3   &45.9      &33.0   \\
\midrule
ResNet-Inverted-DW       &4.6G    &80.0     &10.8    &47.7    &41.9      &29.4   \\
\rowcolor{mygray}
+ P16 + K7 + Norm1Act1              &4.6G    &79.0     &19.5    &42.1    &45.9      &32.8   \\
\midrule
ResNet-Up-Inverted-DW    &4.7G    &79.7     &12.9    &47.9    &42.9      &30.8  \\
\rowcolor{mygray}
+ P16 + K11 + Norm1Act2             &4.4G    &79.2     &20.2    &40.9    &48.7      &35.2   \\
\midrule
ResNet-Down-Inverted-DW  &4.5G    &79.6     &12.1    &49.0    &42.5   &29.5   \\
\rowcolor{mygray}
+ P16 + K11 + Norm1Act1             &4.6G    &79.9     &19.3    &41.6    &46.0      &32.8   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Patch Embedding + Large Kernel Size + Few Norm Act}
\vspace{-1em}
\end{table}

\begin{table}[t!]
\centering
\caption{The robustness generalization of different models when DeiT-S serves as the teacher model. The results of DeiT-S here are included for reference; it is trained by the default recipe without knowledge distillation.}
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Student Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\midrule
\midrule
DeiT-S                                  &4.6G    &79.8     &16.2    &42.8    &41.9      &29.1   \\
\midrule
ResNet-DW                               &4.8G  &79.5  &10.2  &50.8  &40.9  &28.9 \\
\rowcolor{mygray}
\textbf{Robust}-ResNet-DW               &4.5G    &79.3     &20.1    &41.2    &46.5    &33.2   \\
\midrule
ResNet-Inverted-DW                       &4.6G    &80.1     &11.0    &47.9    &42.4    &30.4 \\
\rowcolor{mygray}
\textbf{Robust}-ResNet-Inverted-DW       &4.6G    &79.1     &19.4    &42.4    &45.9    &33.0   \\
\midrule
ResNet-UpInvertedDW                     &4.7G  &79.9  &14.1  &48.4  &43.6  &31.0 \\
\rowcolor{mygray}
\textbf{Robust}-ResNet-Up-Inverted-DW     &4.4G    &79.3     &19.6    &41.1    &49.3    &35.4   \\
\midrule
ResNet-DownInvertedDW                   &4.5G    &79.7     &12.3    &48.5    &42.3    &30.4 \\
\rowcolor{mygray}
\textbf{Robust}-ResNet-Down-Inverted-DW   &4.6G    &79.9     &18.9    &41.0    &46.4    &33.5   \\
\bottomrule
\end{tabular}
}
\label{Experiment: Knowledge Distillation}
\vspace{-1em}
\end{table}

\section{Knowledge Distillation}
\label{sec:knowledge distillation}
Knowledge Distillation \cite{hinton2015distilling} is a training heuristic aiming at transferring the knowledge of a stronger teacher model to a weaker student model with lower capacity. Usually, the student model can achieve better or at least similar performance than the teacher model in knowledge distillation. Yet, directly employing knowledge distillation to let ResNet-50 (student model) learn from DeiT-S (teacher model) has been found to be less effective: ResNet50 still underperforms DeiT-S by a significant margin; While surprisingly when the model roles are switched, the student model DeiT-S is still able to remarkably outperform the teacher model ResNet-50 \cite{bai2020vitsVScnns}.
It is therefore concluded that the key to achieving good robustness of DeiT lies in its architecture that cannot be transferred to ResNet via knowledge distillation. We run these experiments again with our training recipe and models with all three architectural designs combined. We can clearly conclude from Tab. \ref{Experiment: Knowledge Distillation} that with the help of our proposed architectural elements from ViT, a ResNet does perform better than DeiT on out-of-distribution samples. In contrast, the baseline models, though attaining good performance on clean ImageNet, are not as robust as the teacher model DeiT-S.

\section{Larger Models}
So far we have only demonstrated the effectiveness of our proposed models on a relative small model scale. To show that they are also beneficial to larger models, we change our model to match the total FLOPs of DeiT-Base.
We change the base channel dimensions to (128, 256, 512, 1024), and over 20 blocks are added in stage 3 of the network. We then train it with the ConvNeXT-B recipe \cite{convnext} and compare its performance with DeiT-Base, as shown in Tab.~\ref{Experiment: Final Base Model Results}. We have also tried directly applying DeiT recipe and find that brings little improvement. As can be seen from Tab.~\ref{Experiment: Final Base Model Results}, our proposed Robust-ResNet-Base models still performs favorably against DeiT-B, indicating the great potential of our method when scaling up models.

\begin{table}[h!]
\vspace{-1em}
\centering
\caption{Comparing the robustness of Robust-ResNet and DeiT at a larger scale. }
\vspace{-.5em}
 \resizebox{\linewidth}{!}{
\begin{tabular}{c|c|c|c|c|c|c}
\toprule
Architecture   & FLOPs & IN (\textcolor{red}{$\uparrow$})  & S-IN (\textcolor{red}{$\uparrow$})  & IN-C (\textcolor{red}{$\downarrow$})   & IN-R (\textcolor{red}{$\uparrow$}) & IN-SK (\textcolor{red}{$\uparrow$})  \\
\shline
DeiT-B                                            &17.6G    &81.8     &19.6    &37.9       &44.7      &32.0  \\ \hline
\textbf{Robust}-ResNet-Base-DW                &17.2G    &80.5     &19.2    &38.8   &46.6      &34.2
\\
\textbf{Robust}-ResNet-Base-InvertedDW        &17.1G    &81.6     &22.1    &35.5    &50.2    &36.7
\\
\textbf{Robust}-ResNet-Base-UpInvertedDW     &17.1G     &81.9     &23.1    &35.1    &50.5    &37.1
\\
\textbf{Robust}-ResNet-Base-DownInvertedDW    &17.3G     &81.3     &22.2    &35.6    &49.7      &37.8  \\
\bottomrule
\end{tabular}
}
\label{Experiment: Final Base Model Results}
\vspace{-1.5em}
\end{table}

\section{Conclusion}
While the recent work \cite{bai2020vitsVScnns} concludes that Transformers are more robust than CNNs on out-of-distribution samples, it also suggests that Transformer's self-attention-like architectures is the key. In this paper, we take a closer look at the Transformer architecture, and find several useful designs other than the self-attention block. We find that combining these designs and introducing them into ResNet can lead to a pure CNN architecture that can be as robust as, sometimes even more robust than, a Vision Transformer model of the same scale. We hope our results can encourage the community to rethink the robustness comparison between Transformers and CNNs, and meanwhile, help the community re-examine and better understand robust architecture design.

\section*{Acknowledgement}
This work is supported by a gift from Open Philanthropy, TPU Research Cloud (TRC) program, and Google Cloud Research Credits program.

\bibliographystyle{splncs04}

redpajama_set_name: RedPajamaArXiv


For more information on their office refurbishment services, call the Merseyside based company M D R Interiors on 0151 428 4877 or visit them at 15 Quarry St South.
Information about M D R Interiors of Liverpool is provided by Bizwiki, the UK company directory. You can click here to report any errors.


It was early March, and Sabina FooFat was tracing her fingers along an aerial photo of Squamish, pausing here and there to tap one of the district's abundant brownfield sites. A planner with the district, FooFat is in charge of tooling the municipality's energy plan, part of which envisions those parcels of scorched earth redeveloped with energy-efficient infrastructure.
"It's crazy to think of buildings built 20 years ago that had baseboard heaters below windows," she said, face bewildered by the carefree ignorance of yesteryears.
Discouragement comes in the form of a district energy system. In this sense, the word district doesn't apply to District of Squamish, but rather a group of buildings energized by the same system. Using a network of subterranean pipes and boilers, district energy systems circulate hot water to heat buildings plugged into the system.
FooFat and Heinzman, along with Brent Leigh of the Squamish Sustainability Corporation, took a tour through North Vancouver and Vancouver recently, stopping along the way to visit infrastructure built by those communities.
For North Vancouver, it all started in 1998 with a feasibility study for Lower Lonsdale, Versatile Pacific and Central Lonsdale. Three years later, the municipality leveraged $20,000 in funding from the Federation of Canadian Municipalities (FCM) to conduct further study into mini energy plants in Lower Lonsdale and Versatile Pacific. Successful studies produced another $4 million in green grants from FCM, money used to fund the creation, design and operation of a district system.
The Lonsdale Energy Corporation emerged from that backdrop. The company uses gas-fired condensing boilers able to provide about 900 kilowatts of energy. There are three plants on the go, each with about 15 boilers able to service 10 buildings.


text:
Archive | Translated
Marxism and the Dialectics of Ecology
By John Bellamy Foster on October 1, 2016 in Articles, Other Major Scholarly Journal Articles, Translated
The recovery of the ecological-materialist foundations of Karl Marx's thought, as embodied in his theory of metabolic rift, is redefining both Marxism and ecology in our time, reintegrating the critique of capital with critical natural science. This may seem astonishing to those who were reared on the view that Marx's ideas were simply a synthesis […]
Marxism and Ecology
By John Bellamy Foster on December 1, 2015 in Articles, Other Major Scholarly Journal Articles, Translated
To link Marxism and ecological transition may seem at first like trying to bridge two entirely different movements and discourses, each with its own history and logic: one having mainly to do with class relations, the other with the relation between humans and the environment. However, historically socialism has influenced the development of ecological thought […]
Crossing the River of Fire
By John Bellamy Foster on February 1, 2015 in Articles, Other Major Scholarly Journal Articles, Translated
Naomi Klein's new book, This Changes Everything [argues that the source of the looming crisis from climate change] is not the planet, which operates according to natural laws, but rather the economic and social system in which we live, which treats natural limits as mere barriers to surmount. It is now doing so on a planetary scale, […]
By John Bellamy Foster on November 1, 2014 in Articles, Other Major Scholarly Journal Articles, Translated
Not since the Great Depression of the 1930s has it been so apparent that the core capitalist economies are experiencing secular stagnation, characterized by slow growth, rising unemployment and underemployment, and idle productive capacity. Consequently, mainstream economics is finally beginning to recognize the economic stagnation tendency that has long been a focus in these pages, […]
The fortieth anniversary of Harry Braverman's Labor and Monopoly Capital is the occasion here for a reassessment of his work as a whole. Braverman's analysis of the degradation of work is shown to have been only a part of a much larger argument he was developing on the structure of the U.S. working class. Building on his […]
By John Bellamy Foster on August 1, 2014 in Articles, Other Major Scholarly Journal Articles, Translated
The United States came out of the Second World War as the hegemonic power in the world economy. The war had lifted the U.S. economy out of the Great Depression by providing the needed effective demand in the form of endless orders for armaments and troops. Real output rose by 65 percent between 1940 and […]
Stagnation and Financialization
By John Bellamy Foster on May 2, 2014 in Articles, Other Major Scholarly Journal Articles, Translated
More than six years after the beginning of the Great Recession in the United States, and nearly five years since it was officially declared over in this country, the core economies of the capitalist world system remain crisis-ridden. The jobs lost in the downturn in the United States have not yet been fully recovered and […]
The Theory of Monopoly Capitalism: An Elaboration of Marxian Political Economy (New Edition)
By John Bellamy Foster on April 1, 2014 in Authored, Books, Translated
In 1966, Paul Baran and Paul Sweezy published Monopoly Capital, a monumental work of economic theory and social criticism that sought to reveal the basic nature of the capitalism of their time. Their theory, and its continuing elaboration by Sweezy, Harry Magdoff, and others in Monthly Review magazine, influenced generations of radical and heterodox economists. […]
Metabolism, Energy and Entropy in Marx's Critique of Political Economy
By John Bellamy Foster on January 1, 2014 in Book Chapters, Translated
Until recently, most commentators, including ecological Marxists, have assumed that Marx's historical materialism was only marginally ecologically sensitive at best, or even that it was explicitly anti-ecological. However, research over the last decade has demonstrated not only that Marx deemed ecological materialism essential to the critique of political economy and to investigations into socialism, but […]
Marx and the Rift in the Universal Metabolism of Nature
The rediscovery over the last decade and a half of Marx's theory of metabolic rift has come to be seen by many on the left as offering a powerful critique of the relation between nature and contemporary capitalist society. The result has been the development of a more unified ecological world view transcending the divisions […]

redpajama_set_name: RedPajamaCommonCrawl


Risingbd Desk: A UK patient's HIV has become "undetectable" following a stem cell transplant - in only the second case of its kind, doctors report in Nature.
International Desk: A group of scientists has called for a ban on the development of weapons controlled by artificial intelligence (AI).
Risingbd Desk: New results suggest ageing brains can potentially be rejuvenated, at least in mice, according to researchers.
Risingbd Desk: Astronomers say they have the first evidence of a head-on collision between two planets in a distant star system.
Risingbd Desk: Rising temperatures will change the colour of the world's oceans, making them more blue in the coming decades say scientists.
Desk Report: Egyptian archaeologists uncovered a Pharaonic tomb containing 50 mummies dating back to the Ptolemaic era, in Minya, south of Cairo, the ministry of antiquities said on Saturday.
Risingbd Desk: More people over the age of 75 should be taking statins, scientists have said, following a review of research.
Risingbd Desk: Researchers have genetically modified chickens that can lay eggs that contain drugs for arthritis and some cancers.
Risingbd Desk: Funding to tackle 33 significant diseases has reached its highest level since figures were taken, says a survey which has tracked this for 11 years.
International Desk: UK scientists have succeeded in cutting a 2km hole through the West Antarctic Ice Sheet to its base.
Desk Report: BMW's new concept should be music to the eyes and ears of anyone who has been stuck behind a caravan on a B road.
Risingbd Desk: Mysterious signals have been picked up from distant galaxies.


text:
PDF2 Add To My Favorites | Version: 10/06/11 - Chaptered 09/16/11 - Enrolled 09/02/11 - Amended Senate 08/31/11 - Amended Senate 07/05/11 - Amended Senate 12/06/10 - Introduced
AB-29 Governor's Office of Business and Economic Development.(2011-2012)
The Legislature finds and declares all of the following:
(a) In 2008, California had the eighth largest economy in the world, with a gross state product of over $1.8 trillion.
(b) California's significance in the global marketplace results from a variety of factors, including:
(1) Its strategic west coast location that provides direct access to the growing markets and technology centers of Mexico, Latin America, Japan, China, and Southeast Asia.
(2) Its diverse regional economies.
(3) Its ethnically diverse population and well-educated workforce.
(4) Its access to a wide variety of venture and other private capital.
(5) Its rich inventory of pioneering businesses.
(6) Its culture of entrepreneurship, particularly in the areas of computer technology, biotechnology, medical devices and diagnostic tools, and clean and green technology.
(7) Its access to world-class research and educational institutions.
(c) It is therefore in California's best interest to have a designated Office of Business and Economic Development within the Governor's office to provide a visible point of contact and assistance to companies, entrepreneurs, and small business owners, seeking information on business growth and economic development opportunities, as well as technical assistance for navigating the state's regulatory environment.
(d) It is further in California's best interest to reexamine and reframe the state government's apparatus for the delivery of services to the business community, particularly the small business community, to better position California to compete in the global economy.
Chapter 1.6 (commencing with Section 12096) is added to Part 2 of Division 3 of Title 2 of the Government Code, to read:
CHAPTER 1.6. Governor's Office of Business and Economic Development
This chapter shall be known, and may be cited, as the Economic Revitalization Act.
Article 1. Definitions
12096.1.
For purposes of this chapter:
(a) "Office" means the Governor's Office of Business and Economic Development.
(b) "Director" means the Director of the Governor's Office of Business and Economic Development.
Article 2. General
(a) The Governor's Office of Business and Economic Development, also known as "GO-Biz," is hereby established in state government within the Governor's office. The office shall be under the direct control of a director, who shall be responsible to the Governor.
(b) The Governor shall appoint the Director of the Governor's Office of Business and Economic Development who shall perform all duties, exercise all powers, assume and discharge all responsibilities, and carry out and effect all purposes vested by law in the office, including contracting for professional or consultant services in connection with the work of the office.
(c) The salary of the Director of the Governor's Office of Business and Economic Development shall be fixed pursuant to Section 12001.
Article 3. Powers and Duties
The office shall serve the Governor as the lead entity for economic strategy and the marketing of California on issues relating to business development, private sector investment, and economic growth. In this capacity, the office may:
(a) Recommend to the Governor and the Legislature new state policies, programs, and actions, or amendments to existing programs, advance statewide economic goals and respond to emerging economic problems and opportunities, and ensure that all state policies and programs conform to the adopted state economic and business development goals.
(b) Coordinate the development of policies and criteria to ensure that federal grants administered or directly expended by state government advance statewide economic goals and objectives.
(c) Market the business and investment opportunities available in California by working in partnership with local, regional, federal, and other state public and private institutions to encourage business development and investment in the state.
(d) Provide, including, but not limited to, all of the following:
(1) Economic and demographic data.
(2) Financial information to help link businesses with state and local public and private programs.
(3) Workforce information, including, but not limited to, labor availability, training, and education programs.
(4) Transportation and infrastructure information.
(5) Assistance in obtaining state and local permits.
(6) Information on tax credits and other incentives.
(7) Permitting, siting, and other regulatory information pertinent to business operations in the state.
(e) Establish a well-advertised telephone number, an interactive Internet Web site, and an administrative structure that effectively supports the facilitation of business development and investment in the state.
(f) Encourage collaboration among research institutions, startup companies, local governments, venture capitalists, and economic development agencies to promote innovation.
(g) In cooperation with the federal government, foster relationships with overseas entities to improve the state's image as a destination for business investment and expansion.
(h) Conduct research on the state's business climate, including, but not limited to, research on how the state can remain on the leading edge of innovation and emerging sectors.
(i) Support small businesses by providing information about accessing capital, complying with regulations, and supporting state initiatives that support small business.
(a) Each member of the Governor's cabinet shall identify a senior manager within his or her agency who shall coordinate business support activities with the office.
(b) For the 2012–13 fiscal year and subsequent fiscal years, the office shall develop a budget in coordination with the Department of Finance pursuant to the state budgeting process, including the permanent transfer of associated positions.
Article 4. California Business Investment Services Program
(a) The California Business Investment Services Program is hereby created within the Governor's Office of Business and Economic Development.
(b) The program shall be under the direct authority of the director.
(c) The purpose of the program is to serve employers, corporate executives, business owners, and site location consultants who are considering California for business investment and expansion.
(d) In implementing the program, the director shall establish and implement a process for convening teams on key business development situations, including, but not limited to, attracting new businesses, relocation of large manufacturers, or the closure of a large business employer.
(e) In implementing the program, the director shall work cooperatively with local, regional, federal, and other state public and private marketing institutions and trade organizations in attracting, retaining, and helping businesses grow and be successful in California.
Article 5. Small Business Advocate
(a) The Legislature finds and declares that it is in the public interest to aid, counsel, assist, and protect, insofar as is possible, the interests of small business concerns in order to preserve free competitive enterprise and maintain a healthy state economy.
(b) In order to advocate the causes of small business and to provide small businesses with the information they need to survive in the marketplace, there is created within the Governor's Office of Business and Economic Development the Office of Small Business Advocate.
(c) The advocate shall post on his or her Internet Web site the name and telephone number of the small business liaison designated pursuant to Section 14846.
The following definitions apply to this article, unless otherwise indicated:
(a) "Advocate" means the Small Business Advocate who is also the Director of the Office of Small Business Advocate.
(b) "Director" means the Director of the Office of Small Business Advocate.
(c) "Office" means the Office of Small Business Advocate.
(a) The Director of the Office of Small Business Advocate shall be appointed by, and shall serve at the pleasure of, the Governor.
(b) The Governor shall appoint the employees who are needed to accomplish the purposes of this article.
(c) The duties and functions of the advocate shall include all of the following:
(1) Serve as the principal advocate in the state on behalf of small businesses, including, but not limited to, advisory participation in the consideration of all legislation and administrative regulations that affect small businesses, and advocacy on state policy and programs related to small businesses on disaster preparedness and recovery including providing technical assistance.
(2) Represent the views and interests of small businesses before other state agencies whose policies and activities may affect small business.
(3) Enlist the cooperation and assistance of public and private agencies, businesses, and other organizations in disseminating information about the programs and services provided by state government that are of benefit to small businesses, and information on how small businesses can participate in, or make use of, those programs and services.
(4) Consult with experts and authorities in the fields of small business investment, venture capital investment, and commercial banking and other comparable financial institutions involved in the financing of business, and with individuals with regulatory, legal, economic, or financial expertise, including members of the academic community, and individuals who generally represent the public interest.
(5) Seek the assistance and cooperation of all state agencies and departments providing services to, or affecting, small business, including the small business liaison designated pursuant to Section 14846, to ensure coordination of state efforts.
(6) Receive and respond to complaints from small businesses concerning the actions of state agencies and the operative effects of state laws and regulations adversely affecting those businesses.
(7) Counsel small businesses on how to resolve questions and problems concerning the relationship of small business to state government.
(8) Maintain, publicize, and distribute an annual list of persons serving as small business ombudsmen throughout state government.
(9) Consult with the Department of Transportation in the development and administration of the Small and Emerging Contractor Technical Assistance Program established pursuant to Article 2.6 (commencing with Section 14137) of Chapter 2 of Part 5.
(a) Each agency of the state shall furnish to the advocate the reports, documents, and information that are public records and that the director deems necessary to carry out his or her functions under this chapter.
(b) The advocate shall prepare and submit a written annual report to the Governor and to the Legislature that describes the activities and recommendations of the office, including an evaluation of the efforts of state agencies and, where appropriate, specific departments, that significantly regulate small businesses to assist minority and other small business enterprises, and making recommendations that may be appropriate to assist the development and strengthening of minority and other small business enterprises.
(c) The advocate may establish a centralized interactive telephone referral system and Internet Web site to assist small and minority businesses in their operations, including governmental requirements, such as taxation, accounting, and pollution control, and to provide information concerning the agency from which more specialized assistance may be obtained. The advocate may establish and advertise a telephone number and an Internet Web site address to serve this centralized interactive telephone referral system and Internet Web site.
In addition to his or her other responsibilities under this article, the advocate shall do the following:
(a) Develop on the office's Internet Web site in consultation with the appropriate state agencies, and update as necessary, information about emergency preparedness, responses to emergencies, and recovery strategies for small businesses.
(b) Conduct at least one public meeting every year in coordination with the appropriate state agencies, to share best practices for small business disaster preparedness. The meetings shall be held in consultation with regional and statewide small business organizations and shall take place in different locations throughout the state.
Article 7 (commencing with Section 65054) of Chapter 1.5 of Division 1 of Title 7 of the Government Code is repealed.


When nature collides with science, potent powers are ignited. Such is the thinking behind one of the UK's principal AI insight developers. As a leading-researcher in Artificial Intelligence, they privilege technology as a route for resolving the complex mysteries of the universe.
Wanting to celebrate the company's collective thinking, they hosted a one-day event and outdoor party for delegates and their families. With our background in imaginative tech-production and set designs, the agency asked us to bring their vision alive.
Located in Chiswick House and Gardens, we provided an audio-visual system channeling the innovative ingenuity of the agency's tech developments. To cover the large-scale space, we installed a far-reaching PA System and lights, in keeping with the festival-themed brief.
Expressing the feel of Big-Tent entertainment, we designed an aesthetic as bold as our performers. Vibrant garlands and floral festoons aided the celebratory tone. Bespoke painted signs, decorative outdoor-friendly furniture and a gypsy-caravan photo-booth exhibited a boho-chic vibe for family-friendly festival-fun.
Further entertainment was curated by our specialist team. Face painters, balloon modelers, soft-play attractions, roaming bands and storytellers set the scene in motion for the little-ones.
Carrying the party on into the night, a DJ played chilled-out beats, Henna artists were on-hand to embellish attendees and a fairground including traditional rides and stalls set a new-standard in outdoor-events, with the night culminating in a high energy performance from Hackney Colliery Band.
This large scale event required a week-long build involving all of our departments from production to entertainment to set and furniture styling. The result? An impactive day that like the agency, brought a vision of the outstanding possibilities when tech meets nature.


The barber shave is a bit of a dying art in with the advent of advanced razors, but they remain one of the few decadent "beauty" treatments men get to enjoy. No matter how many blades you have on your Mach X razor, you will never get as close as a trained professional who is not having to negotiate through a mirror. Furthermore, one of the keys to a close and comfortable shave is the preparation – steaming towel and creamy balm to soften the skin and whiskers. Something few men shaving take enough time to do. And having the hot towel on your face in a chair is as relaxing as it is soothing.
I treated all my groomsmen to a classic shave on my wedding day and we had as much fun (and looked as almost as smart) as the bridesmaids who enjoyed the beauty parlor. And on our recent trip sailing in the Greek isles, we came upon an old school barber offering shaves and it was one of the decadent high points of the holiday sitting in the chair being pampered while listening to the sounds of the bustling port town outside.
Hug a Sheep Day today! Aka "I love ewe" day for all ovinophiles If you hadn't herd, Finolhu has adopted the fluffy cloven-hoofed ruminants as their resort mascot in a nod to their location in the eponymous Baa Atoll (or should be say "Baaaaaa Atoll"). Various woolly figures are found all over the island (see photos). So maybe skip the lamb curry today out of respect.
I always delight in finding things at resorts that I haven't come across before in other resorts. I especially delight in finding things I had never come across ever in my life. So it was with Olhuveli's "Pakora", a kind of ghee-fried fritter. London and the UK is not short of Indian cuisine, but I have never had a pakora in decades of living here. Not to mention visiting the Indian Ocean neighbour for nearly as long. And not only did the resort offer the treat, but they had a special "Pakora Station" which made several variations on the culinary theme fresh to order.
I've also decide to add a "Station" tag to the blog to highlight the distinctive "fresh, made to order" preparations available at buffets around the Maldives.
Coconuts are the bounty of the Maldivian plots of sand. On nearly any island, some staff member will – often a gardener with a machete for clearing – will grab you one, open up a whole with a few well-placed hacks and let you sip its succulent coconut water. When you're done, he will also slice the fruit open so you can nibble on the tender coconut meat inside. A modest gift of a few dollars is always appreciated in return. Finolhu has parked up a more stylish outlet for noshing on some nucifera. An iconic VW bus in pastel robin's egg blue with a coconut stand right by the pool so you can sample with tropical treat whenever the mood strikes.
Ma'm, would you like filter cropping for breakfast today?


text:
eco.mont – Journal on Protected Mountain Areas Research and Management es una revista científica interdisciplinaria revisada por pares, publica también informes (no revisados por pares), sobre informe de gestión.  De periodicidad semestralmente desde 2009 por el Instituto de Investigación Interdisciplinaria de Montaña de la Academia de Ciencias de Austria en Innsbruck, que reúne artículos sobre áreas protegidas en regiones de montaña. Desde 2015 se aplica el modelo open access a todas las contribuciones.

eco.mont se centra en las áreas de investigación de la Biología, Geografía y Ecología, entre otros. Los enfoques regionales son los Alpes y otras áreas de montaña en Europa y a nivel global. Actualmente, eco.mont está incluido en la base de datos Web of Science (SCIE) y en Scopus. En 2017 el factor de impacto era 0,656. La revista se encuentra en el DOAJ. El DOI de la revista es doi.org/10.1553/eco.mont .

Véase también
 Revista científica
 Geografía regional
 Portal:Geografía
 Portal:Biología
 Portal:Ecología

Referencias

Revistas de geografía
Revistas de biología
Revistas científicas en inglés
Revistas científicas de Austria
Revistas científicas semestrales
Publicaciones fundadas en 2009

redpajama_set_name: RedPajamaWikipedia


text:
Category Archives: animal rights
animal rights, environment, Fail!, Profiteering
How Vikings Hunted Themselves Off of Greenland – Atlas Obscura
January 15, 2020 nedhamson
Even marauding Norsemen can be guilty of entrepreneurial overreach.
Source: How Vikings Hunted Themselves Off of Greenland – Atlas Obscura
animal rights, environment
What to do if you meet a coyote? Wave a big stick and don't run away | CBC News
October 14, 2019 nedhamson
"Try to make yourself imposing and loud," he said. "In the vast majority of cases, that will cause the coyote to alter its course probably go into the adjacent field or woods." Gregory said it's not a good idea to run away, although that might be the natural instinct. "Whether or not the coyote is actually bold or aggressive, it can just trigger an instinct to chase," he said. If the coyote does not retreat, Gregory advised backing away while continuing to face it until it is out of sight, then returning to your vehicle.
Source: What to do if you meet a coyote? Wave a big stick and don't run away | CBC News
animal rights, environment, Feminism, Global Politics, Rock on-Peace Out
The Launch of Fire Drill Fridays – Jane Fonda
Speaking today at the launch of Fire Drill Fridays. 17-year-old Jerome Foster 11, who has been striking for Climate every Friday in front of the White House. Left to right: Medea Benjamin of Code Pink, me, Naomi Klein and Annie Leonard, head of Greenpeace USA With me granddaughter, Vasser Turner Seydell Vasser, being arrested…her first time Me being arrested…not my first time It was a grand launch. The weather was perfect. All the speakers were wonderful, the press coverage was awesome. White plastic "handcuffs" were put on each of us, we were taken to the nearby police station for a few hours. The plastic handcuffs hurt more than the metal ones and I discovered that it's not easy for an 82-year-old to get in and out of a police paddy wagon without the use of her hands. All the police women and men were super polite and nice. There were about 16 of us all together, divided into 2 cells with the doors left open. We used the time, a few hours, to organize and plan. Then we each paid $50 (Fire Drill Friday provided the funds for those who didn't have) and were released. We then walked to a nearby coffeehouse and debriefed. This has all come together very fast. I only moved to D.C. 2 w
Source: The Launch of Fire Drill Fridays – Jane Fonda
animal rights, Fail!, planetkillers
Opinion | The Crisis for Birds Is a Crisis for Us All – The New York Times
September 19, 2019 nedhamson
The mass disappearance of North American birds is a dire warning about the planet's well-being.
— Read on www.nytimes.com/2019/09/19/opinion/crisis-birds-north-america.html
animal rights, environment, innovation
Charting a sustainable path in a land of peat, oil palm and pollution
Indonesia is the world's top palm oil producer — yet too often, yields come at the expense of nature's health.
Source: Charting a sustainable path in a land of peat, oil palm and pollution
animal rights, environment, Fail!, planetkillers, Profiteering
The wrong kind of trees: Ireland's afforestation meets resistance | World news | The Guardian
July 7, 2019 nedhamson
Coillte, a state-owned commercial forestry business, started buying land and planting here in the 1960s. Private companies, encouraged by tax breaks, followed. Farm land vanished as Sitkas multiplied. They now number an estimated 34.5m – more than 1,000 for each inhabitant. The trees mature in about 30 years – exponentially faster than oak – and are then felled, making way for a fresh plantation. "The forest closed in bit by bit," said McCaffrey, a farmer, who now feels surrounded. The trees eclipse sunlight, exude mist and block wifi and phone networks, inducing isolation, he said. "It's a death sentence for the townlands." These non-native trees carpet the soil with acidic needles and smother wildlife, said Natalia Beylis, an artist. "A lot of people find them spooky because they don't have life in them. They're silent except on the edges."
Source: The wrong kind of trees: Ireland's afforestation meets resistance | World news | The Guardian
animal rights, avian influenza, environment, Fail!, healthcare, pandemic, planetkillers, Profiteering
Avian Flu Diary: Nature: Bat Influenza Receptors In Other Mammals (Including Humans) Bats flourish and overpopulate when humans overproduce fruits that bats feed on! Game on! and who knows when we strike out!
February 27, 2019 nedhamson
#13,894 Exactly 7 years ago today ( Feb 27, 2012 ) we looked at an announcement from scientists from the U.S. CDC and the Universid…
Source: Avian Flu Diary: Nature: Bat Influenza Receptors In Other Mammals (Including Humans)
animal rights, environment, Fail!, planetkillers, Profiteering, Traitor
Top Leader at Interior Dept. Pushes a Policy Favoring His Former Client – The New York Times
As a lobbyist and lawyer, David Bernhardt fought for years on behalf of a group of California farmers to weaken Endangered Species Act protections for a finger-size fish, the delta smelt, to gain access to irrigation water. As a top official since 2017 at the Interior Department, Mr. Bernhardt has been finishing the job: He is working to strip away the rules the farmers had hired him to oppose.
animal rights, Anti-Humane, environment, Fail!, Gun-violence, Just strange, planetkillers
Alaska Hunter Who Killed Cubs in Bear Den Gets 3 Months in Jail – The New York Times
Based on the camera footage, which was summarized by prosecutors in a news release, Andrew Renner and his son approached the den on Esther Island, in Prince William Sound, on April 14. Owen Renner fatally shot the sleeping mother bear with a rifle, prosecutors said, causing the cubs to start shrieking inside the den. Andrew Renner then fatally shot the cubs. Prosecutors said the recovered video footage showed the Renners butchering the mother bear and taking the remains away in bags. Two days later, the camera caught them returning to the area to collect the bullet shells and dispose of the bear cubs' bodies, according to a report from Alaska's Department of Public Safety.
animal rights, Gun-violence, planetkillers, Profiteering
How Did Rifles With an American Stamp End Up in the Hands of African Poachers? – The New York Times
December 25, 2018 nedhamson
The question is at the heart of investigations by Congress and federal agencies into whether an American gun manufacturer is entangled in the shadowy world of arms smuggling and wildlife poaching.


text:
J. Ford Little
Primary Practice Areas
Business Litigation and Commercial Law
Vanderbilt University, J.D., 1989
University of Tennessee, B.S., 1986
Selected to America's Top 100 Bet-the Company Litigators® in 2019.
Listed in Best Lawyers in America in Construction Law and Construction Litigation since 2010.
Named the Best Lawyers in America 2013 and 2018 Knoxville Construction Litigation Lawyer of the Year.
Fellow, Knoxville Bar Foundation since 2010
Past President of The University of Tennessee Alumni Association
Past Chair of the University of Tennessee Past Presidents Council and University of Tennessee Alumni Association Board of Governors
Served on Board of Directors of the Helen Ross McNabb Center, 2011-2017
Co-Editor for the Tennessee section of the Product Liability Desk Reference
+Full Bio
J. Ford Little is a member of the firm whose primary areas of practice include construction law, commercial litigation, and product liability/toxic tort. He has an AV Rating by Martindale-Hubbell and has been a member of Best Lawyers in America© by U.S. News & World Report and Best Lawyers®since 2009. In 2019, Ford was selected as one of America's Top 100 Bet-the Company Litigators.®
Mr. Little is an experienced trial lawyer who has been involved in the trials and appeals of cases before federal and state courts around the country. In addition, he has represented clients in mediations and arbitrations in a number of states.
Mr. Little assists clients in the construction industry, including owners, general contractors, subcontractors, and sureties at every stage of the construction project, including the initiation of the project, the preparation of contract documents, bid preparation, bid mistakes, bid protests, change order negotiations, preparation of project correspondence, disputes concerning defective work and delays, claim preparation, terminations, lien claims, and final dispute resolution through negotiation, mediation, arbitration or trial.
Mr. Little, a native of Knoxville, Tennessee, graduated from The University of Tennessee in 1986, with high honors, with a B.S. in accounting and obtained his J.D. from Vanderbilt University in 1989. He is a member of the Knoxville Bar Association and the Tennessee Bar Association. In 2010, Mr. Little was named a Fellow in the Knoxville Bar Foundation. He is also a member of the State Board of the Associated Builders and Contractors, and is involved in the Associated General Contractors of Tennessee, Construction Financial Management Association, and the Tennessee Association of Construction Counsel. Mr. Little is past president of The University of Tennessee Alumni Association and past chair of The University of Tennessee Past Presidents Council. He has served on The University of Tennessee Alumni Association Board of Governors and UTK Chancellor's Associates. He currently serves on the Board of Directors of the Helen Ross McNabb Foundation. He is a past member of the boards of the Helen Ross McNabb Center, Dogwood Arts Festival, and other philanthropic organizations. He is married and has two children, James and Carlen.
Mr. Little has been a speaker at numerous legal and construction industry seminars on topics such as construction law, litigation, and workers' compensation. He is a co-editor for the Tennessee section of the Product Liability Desk Reference.
(865) 215- 1027 (Direct)
J. Ford Little2017-04-212019-10-30http://wmbac.com/wp-content/uploads/2017/08/wm_logo-1.jpgWoolf, McClane, Bright, Allen & Carpenter, PLLChttps://wmbac.com/wp-content/uploads/2017/04/little-j.-ford.jpg200px200px
Howard E. Jarvis


text:
LGBT Member Disappointed in Manny Pacquiao's Win Over Timothy Bradley. Here is What He Had to Say!
In what was likely his final bout inside the ring, Manny 'Pacman' Pacquiao once again carried the pride and hope of the Filipinos for one last time and delivered with a convincing win to end his illustrious boxing career.
For years Pacquiao has been the epitome of the sport of boxing, bringing pride and honor to his country in representing the Philippine flag with each punch thrown.
His victory over Bradley was celebrated by many of his fans, but other's weren't too pleased with Pacquiao's triumph.
A certain member of the LGBT community expressed frustration and disappointment in Pacquiao's win after recalling the boxer's controversial statements that shook the entire world.
Many people also believe that Pacquiao will be filed with a disqualification case. Since he is running for a position in politics, some have concluded that his aired fights are considered illegal and his exposure will certainly affect his candidacy.Nevertheless, many rejoiced when Pacquiao brought home another win that made many Filipinos proud. Despite the controversial statements, many LGBT members are still in support of Pacquiao for bringing honor and glory to the Philippines through his successful boxing career.
What do you think of the comments? Feel free to let us know your own ideas and opinions through the comments section below!
CREDITS: TrendingNewsPortal
Coorms: LGBT Member Disappointed in Manny Pacquiao's Win Over Timothy Bradley. Here is What He Had to Say!
https://1.bp.blogspot.com/-gYJ_CcI-gIE/Vw1XUVjA3hI/AAAAAAAAKX4/0mKJNIyis6IZNewbDZwxBnp4EA23GIMMgCLcB/s640/LBGT-To-Pacquiao.jpg
https://1.bp.blogspot.com/-gYJ_CcI-gIE/Vw1XUVjA3hI/AAAAAAAAKX4/0mKJNIyis6IZNewbDZwxBnp4EA23GIMMgCLcB/s72-c/LBGT-To-Pacquiao.jpg
https://www.coorms.net/2016/04/lgbt-member-disappointed-in-manny-pacquiaos-triumph-over-timothy-bradley.html


Pizzy's sister Machia was the stronger of the two. She was also found under a car nearby. She is striving and making a good recovery.
Adorable Bertie. Now 3 years and still a real softy until his brother tries to take his bone!!
Tasha is pretty o;d now bt doing very well apart from some stiff joints.


From Hwy 23 to south on Townline Rd, to Forest Ave, west to Fenner Rd , south to property.
The details on FENNER ROAD: This land located in Lamartine, WI 54937 is currently for sale for $49,900. FENNER ROAD is a – square foot land with – beds and – baths that has been on Estately for 267 days. This land is in the attendance area of Oakfield Jr Sr High School and Reynolds Elementary School.


NHRA top fuel drag racing tends to get overlooked on these pages, despite it being a raucous good time. Here's a wallpaper from Eric Rieskamp from the 2010 NHRA US Nationals qualifying of Brandon Bernstein's Copart top fuel dragster. So fast.


Air pollution is now the world's fourth leading cause of early death. It killed some 5.5 million people in 2013 alone. And levels of this pollution in the major cities of several countries, including China and India, often exceed WHO's recommended limit.
#Chandra_Venkataraman is a c... read morehemical engineer at the Indian Institute of Technology in Mumbai. The biggest culprit in the country is the traditional burning of biomass, such as wood and dried dung, she found.
Deaths from air pollution will rise by 2030, the researchers say. India and some other countries are likely to increase their emissions as their population grows and burns more fossil fuels.
#be a part of the Solution, not the pollution!
Support #Go_Green_initiative - Reduce, Recycle and Reuse!
"Air pollution is important. And globally, it's a major health risk."


text:
Statistical and Actuarial Sciences
Articles | Videos | Podcasts
Research Talent
Science Students' Council
Master's in Environment and Sustainability
Master of Data Analytics
Alumni and their Perspectives
Donation Announcements
Alumni Award of Achievement
Smart Systems and Computational Applications
Chris Kapulkin
Wild Lab
Lyle Muller
Funny-man Chris Kapulkin brings together elements of stand-up comedy and homomorphic encryption in his daily activities at Western's Department of Mathematics. While perhaps not a duo as well-entrenched as fish and chips, he certainly makes it work. "When I started teaching, I realized that too much intense math is a recipe for students to zone out. Adding humour to my lessons helped hold the interest of the room a bit longer," he laughs. Evidently, Kapulkin's sense of humour was more than enough to hold the attention of a lecture room – if you're lucky, you can actually catch him performing stand-up at comedy clubs around Southern Ontario.
"It's not all fun and games though," smirks Kapulkin. His work on homomorphic encryption and homotopy theory involves making sense of vast sets of data. "If you think of a very simple plot, you can have a point that falls along an x-axis and a y-axis. That's easy to visualize. But what about if you're measuring the medical health of an entire population? Then you have millions of data points and dozens of axes," he explains. Being able to visualize such clouds of information is crucial for recognizing patterns. "Data lives in thousands of dimensions!" exclaims Kapulkin. "From an Excel sheet, you would never be able to wrap your head around the data, let alone recognize the stories they tell through patterns."
Being able to recognize shapes from bare numbers has real-world applications that affect everyone. Driven by these uses of advanced mathematics, Kapulkin's group has applied their work to create secure online elections as well as ensuring the accuracy of software underpinning crucial daily activities like banking and air traffic control.
Data lives in thousands of dimensions! Being able to recognize shapes from bare numbers has real-world applications that affect everyone."
In a related example, researchers used homotopy theory to identify a novel type of cancer from patients reporting symptoms slightly out of line with a pre-existing dataset. Using math to diagnose cancer may not seem intuitive but with computational power skyrocketing, our ability to process massive datasets is improving.
Kapulkin is also looking to apply the fundamentals of his work to encrypting valuable datasets, like information gathered by governments or large corporations. "It's a zillion dollar question – how can we analyze the data without actually seeing it? This is essential to maintain privacy and security," he explains.
The downtime during this pandemic has inspired Kapulkin to beef up his written comedic material for use once the comedy clubs reopen. And just like his stand-up, his lectures this semester have been all been delivered from a virtual stage to an attentive audience who clearly appreciate the content and its delivery.
The Faculty of Science, North Campus Building, Room 240
London, Ontario, Canada, N6A 5B7
Tel: 519-661-2111 ext. 89192
science@uwo.ca
Science Technology Services


What Have You Been Watering Lately?
Hello and welcome to Week 44 of #HOMwork!
We can learn a lot from nature. There are a lot of parallels between how your creativity (both personally and professionally) grows and how a plant grows. What you give your time and attention to will grow, just like a plant would. This goes for both positive and negative attention; if you're feeding negative thoughts they will persist and flourish. If there's a part of your life that you're neglecting it will wilt.
Oftentimes, we talk about all the things we want to do, but if we don't pick up the watering can and tend to those ideas...they will never grow. Consistent maintenance will keep a plant healthy and stimulate new growth, and the same applies for our creative careers. Practicing something a bit each day and dedicating time & care to it will keep us healthy and will ultimately flourish into a thriving skill or career.
This week, I want you to pay attention to the things you're watering in your own life. Where are you spending your time and attention, and is that the best place for it? Are you feeding negative thoughts? Is there an area in your life that is thirsty and needs more tending to? Just like plants, the many branches (heh heh heh) of our lives need different amounts of care to grow to their fullest potential, so take some time this week to consider where to point your watering can next.
This week's #HOMwork assignment is to letter AND illustrate your favorite plant-related phrase surrounded by your favorite (or relevant to the piece) plants.
As always, post your work using the hashtag #HOMwork. In the caption reflect on why the phrase resonates with you and what you're watering these days, personally or professionally!
Don't forget to tag me @homsweethom in your posts so I can see all the gorgeous foliage. I'll be sharing my faves in my Stories all week!