text:
\section{Introduction}

After receiving paper reviews, authors may optionally submit a rebuttal to address the reviewers' comments, which will be limited to a {\bf one page} PDF file.
Please follow the steps and style guidelines outlined below for submitting your author response.

The author rebuttal is optional and, following similar guidelines to previous CVPR conferences, is meant to provide you with an opportunity to rebut factual errors or to supply additional information requested by the reviewers.
It is NOT intended to add new contributions (theorems, algorithms, experiments) that were absent in the original submission and NOT specifically requested by the reviewers.
You may optionally add a figure, graph, or proof to your rebuttal to better illustrate your answer to the reviewers' comments.

Per a passed 2018 PAMI-TC motion, reviewers should refrain from requesting significant additional experiments for the rebuttal or penalize for lack of additional experiments.
Authors should refrain from including new experimental results in the rebuttal, especially when not specifically requested to do so by the reviewers.
Authors may include figures with illustrations or comparison tables of results reported in the submission/supplemental material or in other papers.

Just like the original submission, the rebuttal must maintain anonymity and cannot include external links that reveal the author identity or circumvent the length restriction.
The rebuttal must comply with this template (the use of sections is not required, though it is recommended to structure the rebuttal for ease of reading).

\subsection{Response length}
Author responses must be no longer than 1 page in length including any references and figures.
Overlength responses will simply not be reviewed.
This includes responses where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.

\section{Formatting your Response}

{\bf Make sure to update the paper title and paper ID in the appropriate place in the tex file.}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The top margin should begin 1 inch (2.54 cm) from the top edge of the page.
The bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the page.

Please number any displayed equations.
It is important for readers to be able to refer to any particular equation.

Wherever Times is specified, Times Roman may also be used.
Main text should be in 10-point Times, single-spaced.
Section headings should be in 10 or 12 point Times.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Figure and table captions should be 9-point Roman type as in \cref{fig:onecol}.

List and number all bibliographical references in 9-point Times, single-spaced,
at the end of your response.
When referenced in the text, enclose the citation number in square brackets, for example~\cite{Alpher05}.
Where appropriate, include the name(s) of editors of referenced books.

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{0.5in} \rule{0.9\linewidth}{0pt}}

   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
   \label{fig:onecol}
\end{figure}

To avoid ambiguities, it is best if the numbering for equations, figures, tables, and references in the author response does not overlap with that in the main paper (the reviewer may wonder if you talk about \cref{fig:onecol} in the author response or in the paper).
See \LaTeX\ template for a workaround.

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
Please ensure that any point you wish to make is resolvable in a printed copy of the response.
Resize fonts in figures to match the font in the body text, and choose line widths which render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your response in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it is almost always best to use \verb+\includegraphics+, and to specify the  figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}

{\small
\bibliographystyle{ieee_fullname}

\section{Introduction}
\label{sec:intro}

Deep convolutional neural networks (CNN) \cite{krizhevsky2017imagenet,lecun2015deep,he2016deep} have made remarkable achievements in various visual tasks, e.g., image recognition \cite{he2016deep} and object detection \cite{ren2015faster}. Despite the excellent feature extraction and discrimination ability, CNNs are generally treated as black-box models due to their complex architectures, high-dimensional feature spaces, and an enormous number of learnable parameters. Such lack of interpretability hinders their successful application in fields that require understandable and transparent decisions \cite{rudin2019stop}, such as disease diagnosis \cite{tjoa2020survey}, financial risk assessment \cite{liu2020predicting}, and autonomous driving \cite{kim2017interpretable}.

Recently, increasing attention has been dedicated to the development of interpretable deep-learning models \cite{koh2017understanding,alvarez2018towards,chen2019looks,bohle2022b}.
A particularly interesting strategy is the prototype-based interpretable models, e.g., prototypical part network (ProtoPNet) \cite{chen2019looks,donnelly2022deformable}.
These methods are inherently interpretable since they can explain the model's decisions by showing image classification activation maps associated with a set of class-specific image prototypes.
These prototypes are automatically learned from training samples, with classification score being computed by comparing testing image parts to the learned training prototypes.

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{figures/Fig.1.pdf}

    \caption{The difference between the learning of trivial and support prototypes. (a) Trivial prototypes: the separation loss pushes the prototypes of different classes as far as possible from the classification boundary. (b) Support prototypes: our new closeness loss enforces the prototypes of different classes to be as close as possible to the classification boundary.}
    \label{fig:protopnet}
    \vspace{-8pt}
\end{figure}

ProtoPNet~\cite{chen2019looks} is trained to learn a classifier from a set of class-specific prototypes by minimising the cross-entropy classification loss and two additional regularisation losses, namely: 1) a clustering loss that pulls together training images to prototypes of their own class; and 2) a separation loss that pushes apart training images from all prototypes of other classes.
More specifically, the clustering loss minimises the distance of each image patch to at least one prototype of its own class, while the separation loss maximises the distance between all image patches and all other class prototypes.
The combination of these two losses
pushes the prototypes as far as possible from the classification boundary, but still within the class distribution, so we call them trivial (i.e., easy-to-learn) prototypes, as shown in Fig.~\ref{fig:protopnet}(a).
We also display these trivial prototypes in Fig.~\ref{fig:twomoons}(a), where we present the ProtoPNet learning results for the two-moon problem, depicting the training points (red and blue points) and prototypes (green and black stars) in both the data and feature spaces, learned with a feed-forward neural network\footnote{The network has an input layer of 2 nodes, a hidden layer of 256 nodes (activated by tanh), and an output layer of 2 nodes (activated by sigmoid).}.
Notice that the trivial prototypes are located as far as possible from the classification boundary in the feature space.
Similar to ProtoPNet, the support vector machine (SVM)~\cite{cortes1995support} classifier is trained by minimising a loss function that learns a set of support vectors.
Different from the ProtoPNet's prototypes, these support vectors are located as close as possible to the classification boundary, as shown in Fig.~\ref{fig:twomoons}(c).
Given that the prototypes in ProtoPNet and support vectors in SVM play similar roles in classification problems, we argue that the ProtoPNet's loss function may lead to suboptimal classification results because of the trivial prototypes being learned.

In this paper, we propose an alternative learning strategy of ProtoPNet from the SVM perspective, to force the learned prototypes to resemble support vectors of SVM and be located as close as possible to the classification boundary with the goal of increasing classification accuracy.
The strategy consists of a new closeness loss that minimises the distance between prototypes of different classes.
As shown in Fig.~\ref{fig:protopnet}(b), our new loss enforces the prototypes to move closer to the classification boundary, as also demonstrated by Fig.~\ref{fig:twomoons}(b) that reveals the support prototypes produced by the introduction of our new closeness loss are indeed more similar to the support vectors.
Furthermore, in order to improve interpretability, we propose a new ProtoPNet classifier that integrates the support and trivial prototypes (named ST-ProtoPNet), where the goal is to produce two distinctive and complementary sets of prototypes to obtain more meaningful classification explanations.
Due to the different natures of the two sets of prototypes, they can also enable further improvements in terms of classification accuracy.

The major contributions of this work are:
\begin{enumerate}
    \item We provide the first study that makes an analogy between the prototype learning from ProtoPNet methods and support vector learning from SVM, where we investigate if by following SVM's support vector learning strategy and pulling the prototypes to be as close as possible to the classification boundary, it is possible to improve ProtoPNet's classification accuracy.

    \item We present a new ST-ProtoPNet method to exploit both support and trivial prototypes for the interpretable image classification, where the two sets of prototypes can provide complementary information to improve both interpretability and classification accuracy.
    \item We conduct extensive experiments on three benchmark datasets which show that our ST-ProtoPNet method outperforms current state-of-the-art (SOTA) methods in terms of classification accuracy.
\end{enumerate}
In our experiments, we also demonstrate that the combination of the two types of prototypes contributes to richer interpretability, where trivial prototypes tend to focus on both local parts of the visual object of interest and the background, while support prototypes mainly focus on visually similar object parts of different classes.

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{figures/Fig.2.png}
    \vspace{-15pt}
    \caption{Two-moon classification results from ProtoPNet and SVM classifiers. (a) Trivial prototypes (stars) and training samples (circles) in the feature (top) and data (bottom) spaces. (b) Support prototypes (stars) and training samples (circles) in the feature (top) and data (bottom) spaces. Note that in (a) and (b), each learned prototype is projected onto the nearest training sample in the feature space. (c) Support vectors (stars) and training samples (circles) from a Radial Basis Function (RBF) kernel based SVM. }
    \label{fig:twomoons}
    \vspace{-5pt}
\end{figure}

\section{Related Work}
\label{sec:relatedwork}

In this section, we first review relevant studies on classification interpretability where we focus on prototype-based methods, and then we briefly review support vector machine (SVM) classification.
Finally, we provide a short survey on ensemble classification for interpretability.

\subsection{Classification Interpretability}

The interpretation of classification results produced by deep neural networks can be achieved by a variety of post-hoc explanation techniques, such as explanatory surrogates \cite{lundberg2017unified,zhang2018interpreting,shitole2021one}, counterfactual examples \cite{goyal2019counterfactual,teney2020learning,kenny2021generating}, and saliency visualisation \cite{simonyan2013deep,zeiler2014visualizing,zhou2016learning,selvaraju2017grad}.

In comparison with post-hoc explanations, prototype-based interpretability is directly present in the model's inner computations.
ProtoPNet~\cite{chen2019looks} is the original work that uses class-specific prototypes for interpretable image classification tasks.
Similar to ProtoPNet, TesNet \cite{wang2021interpretable} constructs class-specific transparent basis concepts on Grassmann manifold for the interpretable classification.
Derived from ProtoPNet, Deformable ProtoPNet \cite{donnelly2022deformable} employs spatially-flexible and deformable prototypes to adaptively capture meaningful object features.
In ProtoPShare \cite{rymarczyk2021protopshare}, a data-dependent merge-pruning method is presented to share prototypes among classes, which can reduce the number of prototypes used for classification.
Alternatively, ProtoPool \cite{rymarczyk2022interpretable} introduces a fully differentiable prototype assignment strategy to reduce the number of prototypes.
In Proto2Proto \cite{keswani2022proto2proto}, a knowledge distillation method is designed to transfer interpretability from a teacher ProtoPNet to a shallow student ProtoPNet.
ProtoTree \cite{nauta2021neural} integrates the prototype learning into a binary neural decision tree that can explain its predictions by tracing a decision path throughout the tree.
ViT-NeT \cite{kim2022vit} further establishes the prototype neural tree structure on visual transformers \cite{dosovitskiy2020image}.

Because of the ability to self-explain classification results, prototype-based interpretability (e.g., ProtoPNet) has been widely utilised not only in the computer vision applications above, but also in medical imaging \cite{barnett2021case,kim2021xprotonet,wang2022knowledge} and face recognition \cite{trinh2021interpretable}.
However, an open question faced by these methods is if the prototypes being learned are the optimal ones in terms of classification and interpretability.

\subsection{SVM vs Prototype-based Classification}

To better understand the optimality of prototypes, we consider the support vector machine (SVM)~\cite{cortes1995support} classifier that finds support vectors to represent classes.
More specifically, SVM learns the maximum-margin classifier defined by a classification boundary that maximises the distance to the closest training samples, which are the support vectors for the classes.
The testing of SVM classifiers consists of computing a weighted similarity between a testing sample and the support vectors.
It is interesting to note that the testing of prototype-based classifiers is also based on measuring the similarity between a testing image and a set of class-specific prototypes learned from the training process.
Although the testing of SVM and prototype-based classifiers are similar, their training procedures are quite different.
First, the training of a prototype-based classifier learns a fixed number of prototypes~\cite{chen2019looks,donnelly2022deformable}, while the SVM classifier learns to weight a variable number of support vectors from the training set.
Second, in prototype-based classifiers~\cite{chen2019looks,donnelly2022deformable}, the learned prototypes tend to be as far as possible from the classification boundary, which is contrary to the SVM training objective mentioned above.

The study of deep learning methods from an SVM theoretical perspective is a rich area of research~\cite{domingos2020every,pruthi2020estimating,chen2021equivalence} with huge potential given the impact that both techniques have had in the whole society for the last decades.
However, there are many practical questions that need to be addressed, e.g., how to scale the kernel computation for large-scale datasets, how to shorten the training process~\cite{pruthi2020estimating}, and how to integrate the learning of the SVM classifier with deep-learning features.
In this paper, we focus on adapting the learning of ProtoPNet's prototypes to make them similar to SVM's support vectors, by forcing prototypes to be as close as possible to the classification boundary.

\subsection{Ensemble Classification}

Ensemble classification~\cite{dong2020survey} is a traditional machine learning approach that combines the results from multiple classifiers, with the goals of improving learning generalisation and classification calibration.
The use of interpretable ensemble strategy to improve the classification accuracy has been explored in \cite{chen2019looks,wang2021interpretable,donnelly2022deformable}, which is achieved by summing the classification logits of multiple prototype-based classifiers (e.g., ProtoPNets trained with different CNN backbones).
In this work, we obtain the interpretable ensemble classification by combining the predictions of two ProtoPNets with highly distinctive prototypes (i.e., support prototypes and trivial ones), which is different from previous studies \cite{chen2019looks,wang2021interpretable,donnelly2022deformable} where the type of prototypes each individual classifier produces is very similar given that the same objective function is employed for each classifier.
More specifically, the ensemble classification used in this paper targets the utilisation of two complementary sets of prototypes, particularly when the prototypes are learned from quite different objective functions, such as the ones for learning the support and trivial prototypes.

\section{Preliminaries}
\label{sec:preliminaries}

We assume to have a training set $\mathcal{D} = \{ (\mathbf{x}_n,\mathbf{y}_n) \}_{n=1}^{|\mathcal{D}|}$,
where $\mathbf{x} \in \mathcal{X} \subset \mathbb{R}^{H \times W \times R}$ is an image with $R$ colour channels and $\mathbf{y} \in \mathcal{Y} \subset \{0,1\}^C$ is a one-hot vector representation of the image class label.
The interpretable ProtoPNet \cite{chen2019looks,donnelly2022deformable} is trained to learn a set of prototypes $\mathcal{P}=\{\mathbf{p}_m\}_{m=1}^{M}
$, where $\mathbf{p}_m \in \mathbb{R}^ {\rho_1 \times \rho_2 \times D}$,
with each of the $C$ classes containing $M/C$ prototypes.
Without loss of generality, we assume  $\rho_1=\rho_2=1$, but the extension to general values is trivial.
A typical ProtoPNet comprises four components: a CNN backbone, add-on layers, a prototype layer, and a fully connected (FC) layer.
An input image $\mathbf{x}$ is fed to the CNN backbone $f_{\theta}:\mathcal{X} \to \mathcal{F}$ (parameterised by $\theta \in \Theta$, where $\mathcal{F} \subset \mathbb{R}^{ \bar{H} \times \bar{W} \times \bar{D}}$) and then passed on to the add-on layers, denoted by $f_{\omega}:\mathcal{F} \to \mathcal{V}$ (parameterised by $\omega \in \Omega$), to produce a feature map $\mathbf{V} \in \mathcal{V} \subset \mathbb{R}^{\bar{H} \times \bar{W}  \times D}$.
The prototype layer computes the similarity between the feature map $\mathbf{V}$ and the $M$ $D$-dimensional prototypes $\{ \mathbf{p}_m\}_{m=1}^{M} $ to generate $M$ similarity maps $\mathbf{S}_m^{(i,j)} = {\rm{sim}}(\mathbf{V}(i,j,:), \mathbf{p}_m)$, where
$i \in \{1,...,\bar{H}\}$,
$j \in \{1,...,\bar{W}\}$,
and $\rm{sim(\cdot,\cdot)}$ represents a similarity measure, e.g., cosine similarity \cite{donnelly2022deformable} and projection metric \cite{wang2021interpretable}.
The prototype layer outputs $M$ similarity scores from max-pooling $\mathcal{S} = \Big\{\underset{i \in \{1,...,\bar{H}\}, j \in \{1,...,\bar{W}\}}\max \; \mathbf{S}_m^{(i,j)}\Big\}_{m=1}^M$, which are fed to the FC layer $f_{\phi}:\mathcal{S} \to \Delta$, parameterised by $\phi \in \Phi$,
to produce the classification prediction $\hat{\mathbf{y}} \in \Delta \subset [0,1]^C$, where $\Delta$ denotes the probability space for $C$ classes.

\section{ST-ProtoPNet}

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{figures/Fig.3.pdf}

    \caption{The architecture of our proposed ST-ProtoPNet method for the interpretable image classification.

    }
    \label{fig:archetecture}
    \vspace{-10pt}
\end{figure}

An overview of our proposed ST-ProtoPNet method is illustrated in Fig.~\ref{fig:archetecture}, which comprises a shared CNN backbone $f_{\theta}(\cdot)$, two interpretable ProtoPNet classification branches, namely: 1) the support ProtoPNet
represented by add-on layers $f_{\omega^{(s)}}(\cdot)$, prototype layer with support prototypes $\mathcal{P}^{(s)}$, and FC layer $f_{\phi^{(s)}}(\cdot)$ which outputs the classification probability distribution $\hat{\mathbf{y}}^{(s)} \in \Delta$; and 2) the trivial ProtoPNet branch with its add-on layers $f_{\omega^{(t)}}(\cdot)$, trivial prototypes $\mathcal{P}^{(t)}$, and FC layer $f_{\phi^{(t)}}(\cdot)$ that generates probability predictions $\hat{\mathbf{y}}^{(t)} \in \Delta$.
The final classification is obtained by combining the classification logits from both the support and trivial ProtoPNets.
In our implementation, we construct the support and trivial ProtoPNet mainly based on the original ProtoPNet \cite{chen2019looks} and TesNet \cite{wang2021interpretable}, as explained below.

\subsection{Support ProtoPNet}
\label{sec:support}

The support ProtoPNet is designed to produce support (i.e., hard-to-learn) prototypes for classification that are as close as possible to the classification boundary, as shown in Fig.~\ref{fig:protopnet} and Fig.~\ref{fig:twomoons}.
The loss function to optimise the support ProtoPNet branch is defined as:
\begin{equation}
\begin{split}
    \theta^*,&\omega^{(s)*},\mathcal{P}^{(s)*},\phi^{(s)*} = \\ &\arg\min_{\theta,\omega^{(s)},\mathcal{P}^{(s)},\phi^{(s)}} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}}\ell_{spt}(\mathbf{x},\mathbf{y},\theta,\omega^{(s)},\mathcal{P}^{(s)},\phi^{(s)}).
\end{split}
    \label{eq:main_loss_support}
\end{equation}

The loss for each training sample $(\mathbf{x},\mathbf{y}) \in \mathcal{D}$ in Eq.~\eqref{eq:main_loss_support} above is represented by:
\begin{equation}
\begin{split}
    \ell_{spt}(\mathbf{x},\mathbf{y},\theta,\omega^{(s)},\mathcal{P}^{(s)},\phi^{(s)}) = \ & \ \ell_{ce}(\mathbf{x},\mathbf{y},\theta,\omega^{(s)},\mathcal{P}^{(s)},\phi^{(s)})  \\
    &- \lambda_1 \ell_{ct}(\mathbf{x},\mathbf{y}, \theta,\omega^{(s)},\mathcal{P}^{(s)})  \\
    &+ \lambda_2 \ell_{sp}(\mathbf{x},\mathbf{y}, \theta,\omega^{(s)},\mathcal{P}^{(s)})  \\
    &- \lambda_3 \ell_{cls}(\mathcal{P}^{(s)})  \\
    &+ \lambda_4 \ell_{ort}(\mathcal{P}^{(s)}),
\end{split}
\label{eq:support_PPN}
\end{equation}
where $\lambda_1$, $\lambda_2$, $\lambda_3$, and $\lambda_4$ are hyper-parameters to balance each term, $\ell_{ce}(\cdot)$ denotes the cross-entropy classification loss, $\ell_{ct}(\cdot)$ and $\ell_{sp}(\cdot)$ represent the clustering and separation losses, respectively, which are introduced to regularise the ProtoPNet's training, as follows:
\begin{equation}
    \ell_{ct}(\mathbf{x},\mathbf{y}, \theta,\omega^{(s)},\mathcal{P}^{(s)}) = \max_{\mathbf{p}\in\mathcal{P}^{(s)}_{\mathbf{y}}} \max_{\mathbf{v} \in\mathbf{V}^{(s)}}  \rm{sim}(\mathbf{v},\mathbf{p}),
    \label{eq:support_ct}
\end{equation}
\begin{equation}
    \ell_{sp}(\mathbf{x},\mathbf{y}, \theta,\omega^{(s)},\mathcal{P}^{(s)}) = \max_{\mathbf{p}\notin\mathcal{P}^{(s)}_{\mathbf{y}}} \max_{\mathbf{v} \in\mathbf{V}^{(s)}}  \rm{sim}(\mathbf{v},\mathbf{p}),
    \label{eq:support_sp}
\end{equation}
where $\mathbf{V}^{(s)} = f_{\omega^{(s)}}(f_{\theta}(\mathbf{x}))$ is the feature map extracted from the input image $\mathbf{x}$,
$\mathbf{v}$ represents one of the $\bar{H} \times \bar{W}$ feature vectors in $\mathbf{V}^{(s)}$ obtained by matrix vectorisation,
$\mathbf{p}$ is a normalised prototype (i.e., unit vector) in $\mathcal{P}^{(s)}$,
${\rm{sim}}(\cdot,\cdot)$ is one of the similarity functions defined in Section ~\ref{sec:preliminaries},
and $\mathcal{P}^{(s)}_{\mathbf{y}}$ denotes the set of prototypes of class $\mathbf{y}$.
The clustering loss in Eq. \eqref{eq:support_ct} and separation loss in Eq. \eqref{eq:support_sp} aim to learn a meaningful feature space in which the image features of a certain class are clustered around the prototypes of the class, and also well separated from those of other classes.

As mentioned in Section~\ref{sec:intro}, the effect of the clustering and separation losses above tend to push the prototypes of different classes as far as possible from the classification boundary, resulting in trivial prototypes, as displayed in Fig.~\ref{fig:protopnet}(a) and Fig.~\ref{fig:twomoons}(a). In order to learn the proposed support prototypes, we introduce the following novel closeness loss $\ell_{cls}$ to explicitly enforce the prototypes of different classes to be close to each other, which is formulated as:
\begin{equation}

    \ell_{cls}(\mathcal{P}^{(s)}) = \sum_{c_1 = 1}^{C-1} \sum_{c_2 = c_1 + 1}^{C} \min_{\mathbf{p}_m\in\mathcal{P}_{c_1}, \mathbf{p}_n\in\mathcal{P}_{c_2}}
    \mathbf{p}_m^{\top} \mathbf{p}_n.
    \label{eq:support_cls}
\end{equation}

During training, the closeness loss $\ell_{cls}$ maximises the pair-wise prototype similarity, in the form of dot product $\mathbf{p}_m^{\top} \mathbf{p}_n$ between different classes in Eq.~\eqref{eq:support_cls}, with the goal of pulling the prototypes close to the classification boundary.
As the prototypes move gradually towards the classification boundary, they are able to capture harder and harder visual features from training samples.
On the other hand, since the prototypes are located near the classification boundary, this can put pressure on the support ProtoPNet's learning and enforce it to learn highly discriminative
feature representations to achieve accurate classification, which is beneficial to extract more meaningful semantic information from training samples.

Ideally, each prototype of a class should focus on unique object parts of the training images (e.g., head, tail, and claw of birds), so that the prototypes can represent rich and diverse visual patterns.
However, there is no particular constraints to guarantee such prototype diversity and the issue of prototype duplication \cite{donnelly2022deformable} often occurs in the ProtoPNet family of models.
To encourage the intra-class prototype diversity, we employ an orthonormality loss \cite{wang2021interpretable} so that prototypes within a class can represent dissimilar visual patterns of training samples, which is defined as:
\begin{equation}
    \ell_{ort}(\mathcal{P}^{(s)}) = \sum_{c = 1}^{C} \| {\mathbf{P}_c}^{\top} \mathbf{P}_c -\mathbf{I}_{M/C} \|_F^2,
    \label{eq:support_orth}
\end{equation}
where $\|\cdot\|_F^2$ represents Frobenius norm,
$\mathbf{P}_c \in D \times \mathbb{R}^{ (M/C)}$ stands for a matrix composed of the prototypes of class $c$ (prototypes in each column of $\mathbf{P}_c$ are normalised),
and $\mathbf{I}_{M/C}$ is an identity matrix of size $M/C \times M/C$.

\subsection{Trivial ProtoPNet}

As described in Section ~\ref{sec:support}, the support ProtoPNet is developed to learn support (i.e., hard-to-learn) prototypes by forcing them to be close to the classification boundary.
Considering that training samples contain not only hard visual features but also important trivial ones that the support prototypes cannot fully capture, we propose to also learn trivial prototypes to provide complementary classification information, and exploit both the support and trivial prototypes for improved interpretable classification.

The loss objective to train the trivial ProtoPNet branch is defined as follows:
\begin{equation}
\begin{split}
    \theta^*,&\omega^{(t)*},\mathcal{P}^{(t)*},\phi^{(t)*} = \\ &\arg\min_{\theta,\omega^{(t)},\mathcal{P}^{(t)},\phi^{(t)}} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}}\ell_{trv}(\mathbf{x},\mathbf{y},\theta,\omega^{(t)},\mathcal{P}^{(t)},\phi^{(t)}).
\end{split}
    \label{eq:main_loss_trivial}
\end{equation}
The loss for each training image $(\mathbf{x},\mathbf{y}) \in \mathcal{D}$ in Eq.~\eqref{eq:main_loss_trivial} above is represented by:
\begin{equation}
\begin{split}
    \ell_{trv}(\mathbf{x},\mathbf{y},\theta,\omega^{(t)},\mathcal{P}^{(t)},\phi^{(t)}) = \ & \ \ell_{ce}(\mathbf{x},\mathbf{y},\theta,\omega^{(t)},\mathcal{P}^{(t)},\phi^{(t)})  \\
    &- \lambda_1 \ell_{ct}(\mathbf{x},\mathbf{y}, \theta,\omega^{(t)},\mathcal{P}^{(t)})  \\
    &+ \lambda_2 \ell_{sp}(\mathbf{x},\mathbf{y}, \theta,\omega^{(t)},\mathcal{P}^{(t)})  \\
    &+ \lambda_3 \ell_{dsc}(\mathcal{P}^{(t)}) \\
    &+ \lambda_4 \ell_{ort}(\mathcal{P}^{(t)}),
\end{split}
\label{eq:trivial_PPN}
\end{equation}
where $\lambda_1$, $\lambda_2$, $\lambda_3$, and $\lambda_4$ are hyper-parameters,
$\ell_{ce}(\cdot)$ is the cross-entropy loss,
the clustering loss $\ell_{ct}$, separation loss $\ell_{sp}$, and orthonormality loss $\ell_{ort}$ are the same as in the support ProtoPNet defined in Eq. ~\eqref{eq:support_ct}, \eqref{eq:support_sp} and \eqref{eq:support_orth}, respectively.

The trivial ProtoPNet targets the learning of easy prototypes that are far from the classification boundary and have a good discrimination ability.
To help achieve this, we introduce a new discrimination loss $\ell_{dsc}$ to facilitate the inter-class separability between prototypes of different classes.
This is formulated by minimising the pair-wise prototype similarities of different classes, as follows:
\begin{equation}

    \ell_{dsc}(\mathcal{P}^{(t)})  = \sum_{c_1 = 1}^{C-1} \sum_{c_2 = c_1 + 1}^{C} \max_{\mathbf{p}_m\in\mathcal{P}_{c_1}, \mathbf{p}_n\in\mathcal{P}_{c_2}} \mathbf{p}_m^{\top} \mathbf{p}_n.
    \label{eq:ell_dis}
\end{equation}

\subsection{Training and Testing}

\textbf{Training.} Following the training strategies in the ProtoPNet family~\cite{chen2019looks,donnelly2022deformable}, the training procedure of our ST-ProtoPNet consists of 3 stages:
1) stochastic gradient descent (SGD) optimisation of the CNN backbone, add-on layers, and prototype layer, using a fixed FC layer initialised with +1.0 and -0.5 for correct and incorrect connection weights, respectively;
2) prototype projection by updating each prototype with its nearest latent training image patch; and
3) optimisation of the FC layer, with an additional $L_1$ regularisation on the incorrect connection weights (initially fixed at -0.5).
In each stage, we alternate the optimisation of each branch of the ST-ProtoPNet between mini-batches.

\textbf{Testing.} To exploit the complementary results from both branches of the ST-ProtoPNet, the final classification is obtained from the summed logits predicted by the two branches.
It is worth noticing that this ensemble strategy introduces no loss of interpretablity but improved accuracy.

\section{Experiments}
\label{sec:experiment}

We perform experiments on three fine-grained classification benchmark datasets: CUB-200-2011 \cite{wah2011caltech}, Stanford Cars \cite{krause20133d}, and Stanford Dogs \cite{khosla2011novel}.
To achieve fair comparison, we follow previous studies \cite{chen2019looks,wang2021interpretable} by applying offline data augmentations (e.g., random rotation, skew, shear, and left-right flip) on the cropped CUB and cropped Cars datasets (using bounding boxes provided).
We also validate our method on the full CUB and full Dogs datasets, and employ the same online data augmentation methods (e.g., random affine transformation and left-right flip) as used in Deformable ProtoPNet \cite{donnelly2022deformable}.
All images are resized to $224 \times 224$ pixels as network input.

\subsection{Experimental Settings}

The proposed ST-ProtoPNet method is evaluated on the following CNN architectures: VGG-16, VGG-19, ResNet-34, ResNet-50, ResNet-152, DenseNet-121, and DenseNet-161.
All CNN backbones are pre-trained on ImageNet~\cite{deng2009imagenet}, except for ResNet-50, which is pre-trained on iNaturalist~\cite{van2018inaturalist} for the experiment on full CUB~\cite{donnelly2022deformable}.
The add-on layers include two $1 \times 1$ convolutional layers.
For simplicity, we utilise the same prototype dimension $D=64$ for all CNN backbones on the three datasets.
For cropped CUB and Cars datasets, following~\cite{wang2021interpretable}, we use 10 prototypes (5 support and 5 trivial) per class and the projection metric in the similarity function ${\rm{sim}}(\cdot,\cdot)$.
In full CUB and Dogs datasets, to ensure comparison fairness with Deformable ProtoPNet~\cite{donnelly2022deformable} that uses 10 $2 \times 2$ (full CUB) and 10 $3 \times 3$ (full Dogs) deformable prototypes per class, we utilise the same total number of prototypes, i.e., 40 $1 \times 1$ (20 support and 20 trivial) for full CUB and 90 $1 \times 1$ (45 support and 45 trivial) for full Dogs.
Also, we employ the cosine similarity in ${\rm{sim}}(\cdot,\cdot)$ and obtain $14 \times 14$ ($\bar{H}=\bar{W}=14$) feature maps by upsampling the original $7 \times 7$ feature maps via a bi-linear interpolation step, as in \cite{donnelly2022deformable}.
In our implementation, we empirically set $\lambda_1 = 0.8$, $\lambda_2 = 0.48$ and $0.08$ for the support and trivial ProtoPNet branches respectively, $\lambda_3 = 1.0$, and $\lambda_4 = 0.001$.
More details about the experimental setup can be found in the supplementary material.

\subsection{Performance Comparison}

\begin{table*}[]
\setlength{\tabcolsep}{1.0 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{c|cccccc|cccccc}
\hline
\multirow{2}{*}{Method} & \multicolumn{6}{c|}{CUB}                                                                                & \multicolumn{6}{c}{Cars}                                                                                \\
\cline{2-13}
                        & VGG16               & VGG19               & ResNet34            & ResNet152           & Dense121            & Dense161            & VGG16               & VGG19               & ResNet34            & ResNet152           & Dense121            & Dense161             \\
\hline
Baseline                & 73.3 ± 0.2          & 74.7 ± 0.4          & 82.2 ± 0.3          & 80.8 ± 0.4          & 81.8 ± 0.1          & 82.1 ± 0.2          & 87.3 ± 0.4          & 88.5 ± 0.3          & \textbf{92.6 ± 0.3} & \textbf{92.8 ± 0.4} & 92.0 ± 0.3          & 92.5 ± 0.3           \\
ProtoPNet \cite{chen2019looks}              & 77.2 ± 0.2          & 77.6 ± 0.2          & 78.6 ± 0.1          & 79.2 ± 0.3          & 79.0 ± 0.2          & 80.8 ± 0.3          & 88.3 ± 0.2          & 89.4 ± 0.2          & 88.8 ± 0.1          & 88.5 ± 0.3          & 87.7 ± 0.1          & 89.5 ± 0.2           \\
TesNet \cite{wang2021interpretable}                 & 81.3 ± 0.2          & 81.4 ± 0.1          & 82.8 ± 0.1          & 82.7 ± 0.2          & 84.8 ± 0.2          & 84.6 ± 0.3          & 90.3 ± 0.2          & 90.6 ± 0.2          & 90.9 ± 0.2          & 92.0 ± 0.2          & 91.9 ± 0.3          & 92.6 ± 0.3           \\
Trivial ProtoPNet       & 80.8 ± 0.2          & 81.2 ± 0.2          & 82.5 ± 0.2          & 83.1 ± 0.3          & 83.9 ± 0.3          & 84.6 ± 0.3          & 90.1 ± 0.2          & 90.7 ± 0.2          & 91.1 ± 0.2          & 91.5 ± 0.2          & 91.4 ± 0.3          & 92.4 ± 0.3           \\
Support ProtoPNet       & 81.7 ± 0.2          & 81.8 ± 0.3          & 83.0 ± 0.1          & 83.6 ± 0.2          & 84.7 ± 0.2          & 85.2 ± 0.3          & 90.9 ± 0.2          & 90.8 ± 0.2          & 91.0 ± 0.2          & 91.8 ± 0.2          & 91.7 ± 0.2          & 92.7 ± 0.3           \\
ST-ProtoPNet (ours)   & \textbf{82.9 ± 0.2} & \textbf{83.2 ± 0.2} & \textbf{83.5 ± 0.1} & \textbf{84.1 ± 0.2} & \textbf{85.4 ± 0.2} & \textbf{86.1 ± 0.2} & \textbf{91.1 ± 0.2} & \textbf{91.7 ± 0.2} & 91.4 ± 0.1          & 92.0 ± 0.2          & \textbf{92.3 ± 0.3} & \textbf{92.7 ± 0.2}  \\
\hline
\end{tabular}

}
\vspace{-6pt}
\caption{Classification accuracy (\%) on cropped CUB-200-2011 and Stanford Cars by competing methods using different CNN backbones.}
\label{tab:croppedCUBCars}
\end{table*}

Table \ref{tab:croppedCUBCars} presents the classification accuracy (averaged across 5 runs) of our proposed ST-ProtoPNet on cropped CUB and cropped Cars datasets, where the Baseline is represented by non-interpretable black-box CNN models.
As can be seen, our ST-ProtoPNet outperforms other competing methods across all backbone architectures for the task of bird species classification.
Also, our method achieves the best results for the car model identification task when using VGG and DenseNet architectures as the CNN backbone.
In particular, our VGG19-based ST-ProtoPNet reaches an average accuracy of 83.2\% and 91.7\% on CUB and Cars, respectively, surpassing other methods with the most improvements across all backbones.
Moreover, the support ProtoPNet generally performs better than methods utilising only trivial prototypes (e.g., ProtoPNet, TesNet, and Trivial ProtoPNet), showing the importance of learning support prototypes for the interpretable image classification.
It is worth noting that our proposed ST-ProtoPNet produces superior performance over the support ProtoPNet method, indicating that both support and trivial prototypes are useful and can provide complementary information for achieving accurate and interpretable classification.

\begin{table*}[]
\setlength{\tabcolsep}{0.6 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{c|cccccccc|cccccccc}
\hline
\multirow{2}{*}{Method} & \multirow{2}{*}{Prototype} & \multicolumn{7}{c|}{CUB}                                                                                & \multirow{2}{*}{Prototype} & \multicolumn{7}{c}{Dogs}                                                                                \\
\cline{3-9}\cline{11-17}
                        &                            & VGG16         & VGG19         & ResNet34      & ResNet50      & ResNet152     & Dense121      & Dense161      &                            & VGG16         & VGG19         & ResNet34      & ResNet50      & ResNet152     & Dense121      & Dense161       \\
\hline
Baseline                & –                          & 70.9          & 71.3          & 76.0          & 78.7          & 79.2          & 78.2          & 80.0          & –                          & 75.6          & 77.3          & 81.1          & 83.1          & 85.2          & 81.9          & 84.1           \\
ProtoPNet \cite{chen2019looks}              & 1×1p, 10pc                 & 70.3          & 72.6          & 72.4          & 81.1          & 74.3          & 74.0          & 75.4          & 1×1p, 10pc                 & 70.7          & 73.6          & 73.4          & 76.4          & 76.2          & 72.0          & 77.3           \\
ProtoPNet \cite{chen2019looks}              & 1×1p, 40pc                 & 72.9          & 74.2          & 74.1          & 84.8          & 76.0          & 76.6          & 78.5          & 1×1p, 90pc                 & 73.9          & 75.3          & 76.1          & 78.1          & 79.7          & 75.4          & 78.8           \\
TesNet \cite{wang2021interpretable}                 & 1×1p, 10pc                 & 75.8          & 77.5          & 76.2          & 86.5          & 79.0          & 80.2          & 79.6          & 1×1p, 10pc                 & 74.3          & 77.1          & 80.1          & 82.4          & 83.8          & 80.3          & 83.8           \\
TesNet \cite{wang2021interpretable}                 & 1×1p, 40pc                 & 77.6          & 79.2          & 76.5          & 87.3          & 80.1          & 80.9          & 81.3          & 1×1p, 90pc                 & 78.5          & 79.6          & 81.2          & 83.3          & 84.5          & 82.1          & 85.2           \\
Deformable ProtoPNet \cite{donnelly2022deformable}  & 2×2p, 10pc              & 75.7          & 76.0          & 76.8          & 86.4          & 79.6          & 79.0          & 81.2          & 3×3p, 10pc                 & 75.8          & 77.9          & 80.6          & 82.2          & 86.5          & 80.7          & 83.7           \\
Trivial ProtoPNet       & 1×1p, 40pc                 & 80.0          & 79.5          & 77.5          & 87.2          & 80.8          & 81.1          & 82.1          & 1×1p, 90pc                 & 78.6          & 80.4          & 82.6          & 85.0          & 87.0          & 82.3          & 85.9           \\
Support ProtoPNet       & 1×1p, 40pc                 & 80.4          & 80.0          & \textbf{78.4} & 87.5          & 80.2          & 81.5          & 82.4          & 1×1p, 90pc                 & 79.0          & 80.6          & 83.0          & 85.1          & \textbf{87.3} & 82.6          & 86.2           \\
ST-ProtoPNet (ours)   & 1×1p, 10pc                 & 76.8          & 77.6          & 77.4          & 86.6          & 78.7          & 78.6          & 80.6          & 1×1p, 10pc                 & 74.2          & 77.2          & 80.8          & 84.0          & 85.3          & 79.4          & 84.4           \\
ST-ProtoPNet (ours)    & 1×1p, 40pc                 & \textbf{81.0} & \textbf{80.2} & 78.2          & \textbf{88.0} & \textbf{81.2} & \textbf{81.8} & \textbf{82.7} & 1×1p, 90pc                 & \textbf{79.1} & \textbf{80.9} & \textbf{83.4} & \textbf{85.7} & 87.2          & \textbf{82.9} & \textbf{86.6}  \\
\hline
\end{tabular}

}
\vspace{-5pt}
\caption{Classification accuracy (\%) on full CUB-200-2011 and Stanford Dogs datasets by competing approaches using different CNN backbones, where $k$pc represents $k$ prototypes per class and $\rho_1$×$\rho_2$p denotes the spatial shape of the prototypes.}
\label{tab:fullCUBDogs}
\end{table*}

Table \ref{tab:fullCUBDogs} shows the classification results of different methods on full CUB and full Dogs datasets.
In both datasets, the classification accuracy of the original ProtoPNet method is generally worse than the non-interpretable counterpart (Baseline) under many CNN backbones.
On the other hand, the accuracy by the trivial ProtoPNet and support ProtoPNet are substantially better than those by Baseline, ProtoPNet, and Deformable ProtoPNet.
However, the proposed ST-ProtoPNet achieves more significant performance gains that result in the best classification accuracy across most CNN backbones, particularly when using a large number of prototypes (i.e., 40 $1\times1$ prototypes per class for CUB and 90 $1\times1$ prototypes per class for Dogs), which demonstrates the effectiveness of utilising both the trivial and support prototypes for the interpretable image classification.
Additionally, when using a smaller number of prototypes, i.e, 10 $1\times1$ prototypes per class, our ST-ProtoPNet method still exhibits competitive classification accuracy across multiple CNN backbones.

We further compare our ST-ProtoPNet with other deep-learning methods that can provide different levels of interpretability on CUB dataset, with results shown in Table \ref{tab:interprlevel}, where * and ** denote ensembled models trained with different CNN backbones or hyper-parameters.
As can be observed, an ensemble of three ST-ProtoPNets can achieve high classification accuracy (87.9\% on cropped images, 88.2\% on full images), outperforming competing methods that are also based on an ensemble of three models (e.g., ProtoTree, TesNet, and ProtoPool).
Moreover, the ensemble of five ST-ProtoPNets exceeds all other competing methods and obtains the best classification accuracy of 88.1\% and 88.4\% on cropped and full CUB images, respectively.
Experimental results on Stanford Cars and Stanford Dogs are presented in the supplementary material.

\begin{table}[]
\setlength{\tabcolsep}{0.6 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{llcc}
\hline
Interpretability                                    & Method                 & \multicolumn{2}{c}{Accuracy (\%)}                    \\
\hline
None                                                & B-CNN \cite{lin2015bilinear}                  & 85.1 (b)          & 84.1 (f)                    \\
\hline
\multirow{2}{*}{Object-level attention}             & CAM \cite{zhou2016learning}                   & 70.5 (b)          & 63.0 (f)                    \\
                                                    & CSG \cite{liang2020training}                   & \textbf{82.6} (b) & \textbf{\textbf{78.5}}~(f)  \\
\hline
\multirow{5}{*}{Part-level attention}               & PA-CNN \cite{krause2015fine}                 & 82.8 (b)          & --                          \\
                                                    & MG-CNN \cite{wang2015multiple}                & \textbf{83.0} (b)          & 81.7 (f)                    \\
                                                    & MA-CNN \cite{zheng2017learning}                & --                & 86.5 (f)                    \\
                                                    & RA-CNN \cite{fu2017look}                & --                & 85.3 (f)                    \\
                                                    & TASN \cite{zheng2019looking}                  & --                & \textbf{\textbf{87.0}}~(f)  \\
\hline
\multirow{10}{*}{Part-level attention + Prototypes} & Region \cite{huang2020interpretable}                & 81.5 (b)          & 80.2 (f)                    \\
                                                    & ProtoPNet* \cite{chen2019looks}            & 84.8 (b)          & 81.1 (f)                    \\
                                                    & ProtoTree* \cite{nauta2021neural}            & --                & 86.6 (f)                    \\
                                                    & TesNet* \cite{wang2021interpretable}               & 86.2 (b)          & 83.5 (f)                    \\
                                                    & ProtoPool* \cite{rymarczyk2022interpretable}            & 87.5 (b)          & --                          \\
                                                    & ST-ProtoPNet* (ours)         & 87.9 (b)          & 88.2 (f)                     \\
                                                    & ProtoTree** \cite{nauta2021neural}           & ~--               & 87.2 (f)                    \\
                                                    & Deformable ProtoPNet** \cite{donnelly2022deformable} & ~--               & 87.8 (f)                    \\
                                                    & ProtoPool** \cite{rymarczyk2022interpretable}           & 87.6 (b)          & --                          \\
                                                    & ST-ProtoPNet** (ours) & \textbf{88.1} (b) & \textbf{88.4} (f)                         \\
\hline
\end{tabular}

}
\vspace{-5pt}
\caption{Classification accuracy and interpretability of different methods on CUB-200-2011. "b" and "f" denote the model is trained and tested on cropped and full images, respectively. *: Ensembled of three models. **: Ensembled of five models.
}
\label{tab:interprlevel}
\end{table}

\subsection{Visualisation Analysis}

\begin{figure*}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{figures/Fig.4.pdf}
    \vspace{-20pt}
    \caption{Visual comparison between the support (a) and trivial (b) prototypes from cropped CUB-200-2011, where each row exhibits prototypes of the same class. In each pair, the first column shows the original image with a prototype indicated in a yellow bounding box, the second column demonstrates the prototype's corresponding activation map. }
    \label{fig:visualprototypes}
    \vspace{-14.5pt}
\end{figure*}

To better highlight the differences between the support and trivial prototypes, we select 8 categories of birds with visually similar features from cropped CUB-200-2011 to form a subset, and show the learned prototypes of our VGG19-based ST-ProtoPNet in Fig.~\ref{fig:visualprototypes}, where each prototype is visualised by projecting onto its nearest training image patch in the latent feature space \cite{chen2019looks}.
We can see in Fig.~\ref{fig:visualprototypes}(a) that the support prototypes can capture subtle and fine visual features of different classes.
Importantly, notice that the support prototypes only focus on relevant bird parts, such as the head and belly.
This is reasonable since our learning algorithm is designed to produce prototypes that are as close as possible to the classification boundary, where the image prototypical parts should be discriminative but at the same time visually similar among different classes.
By contrast, it is observed in Fig.~\ref{fig:visualprototypes}(b) that the trivial prototypes tend to focus not only on the relevant bird parts but also the background regions.
For example, some trivial prototypes of the Laysan Albatross and Sooty Albatross classes capture the sea surface as they often appear with the sea background.
We argue that this is because the trivial ProtoPNet may treat the background as the most easy-to-learn pattern, focusing less on the target's visual parts of the class.

Fig.~\ref{fig:reasoning} displays an example of the interpretable reasoning process of our ST-ProtoPNet method in classifying a testing bird image.
As can be seen, each ProtoPNet branch calculates its own classification logits (weighted sum of similarity scores), which is then combined to generate the final predictions.
To be specific, when classifying a Brewer Blackbird, the support prototypes are quite active on the belly or lower surface of the bird.
Meanwhile, the trivial prototypes mostly have high activations on the bird's head and tail.
In this case, the support ProtoPNet branch obtains a relatively higher similarity score (22.913), compared with the trivial branch (20.248).
Note that our ST-ProtoPNet exploits both the support and trivial prototypes to capture much richer representations of the target object from different perspectives, which enables the production of the final decision in a complementary way.
Notably, the ensemble classification improves interpretability and accuracy using richer representations for the target object.
More examples of visual prototypes and interpretable reasoning on Cars and Dogs are provided in the supplementary material.

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.0\linewidth]{figures/Fig.5.pdf}
    \vspace{-17pt}
    \caption{An example of the interpretable reasoning of our ST-ProtoPNet for classifying a testing Brewer Blackbird image. }
    \label{fig:reasoning}
    \vspace{-10pt}
\end{figure}

\subsection{Ablation Study}

\textbf{The Closeness and Discrimination Losses.}
In order to validate the effectiveness of our proposed
closeness loss in Eq.~\eqref{eq:support_cls} and discrimination loss in Eq.~\eqref{eq:ell_dis}, we first conduct ablation studies on full CUB and full Dogs datasets using ResNet34 as the CNN backbone, with results provided in Table~\ref{tab:ablationLoss}.
We can notice that both the closeness and discrimination losses can improve the classification accuracy, compared with the Baseline ProtoPNet trained with only clustering, separation, and orthonormality losses.
Importantly, the closeness loss introduces a larger performance improvement since it aims at learning support (i.e., hard-to-learn) prototypes that lie near the classification boundary.

\begin{table}[]
\setlength{\tabcolsep}{3.2 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{cccccccc}
\hline
\multirow{2}{*}{Method} & \multirow{2}{*}{$\ell_{ct}$} & \multirow{2}{*}{$\ell_{sp}$} & \multirow{2}{*}{$\ell_{ort}$} & \multirow{2}{*}{$\ell_{dsc}$} & \multirow{2}{*}{$\ell_{cls}$} & \multicolumn{2}{c}{Accuracy (\%)}  \\
                        &                                                  &                                                  &                                                   &                                                   &                                                   & CUB & Dogs                    \\
\hline
Baseline ProtoPNet      & $\checkmark$                     & $\checkmark$                    & $\checkmark$                                      &                                                   &                                          & 76.7    & 80.9                       \\
Trivial ProtoPNet       & $\checkmark$                     & $\checkmark$                    & $\checkmark$                                      & $\checkmark$                                      &                                          & 77.5    & 82.6                        \\
Support ProtoPNet       & $\checkmark$                     & $\checkmark$                    & $\checkmark$                                      &                                                   & $\checkmark$                             & 78.4    & 83.0                        \\
\hline
\end{tabular}

}
\vspace{-6pt}
\caption{Ablation analysis of the
closeness loss $\ell_{cls}$ in Eq.~\eqref{eq:support_cls} to learn support prototypes, and
discrimination loss $\ell_{dsc}$ in Eq.~\eqref{eq:ell_dis} to learn trivial prototypes on full CUB-200-2011 and Stanford Dogs.}
\label{tab:ablationLoss}
\end{table}

\textbf{Combining Support and Trivial Prototypes.}
We also investigate the importance of integrating the two complementary sets of support and trivial prototypes for achieving improved classification.
To achieve this, we first train a two-branch model where both branches learn the same type of prototypes and the final result is produced by the ensemble of them (Trivial Ensemble and Support Ensemble).
Besides, for our ST-ProtoPNet, we also provide results of its individual branches (Trivial Branch and Support Branch).
Table~\ref{tab:ablationEnsemble} gives the experimental results on cropped CUB dataset.
We can notice that combining the two different types of prototypes (ST-ProtoPNet) achieves superior performance over combining only the same type of prototypes (Trivial Ensemble and Support Ensemble), indicating that our performance improvements are from not only the ensemble strategy but also the two complementary sets of prototypes.
Also, ST-ProtoPNet indeed exhibits higher accuracy than its individual branches, further verifying that the results from the two branches are complementary, and the combination of them are effective to improve the final classification accuracy.

\begin{table}[]
\setlength{\tabcolsep}{0.7 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{ccccccc}
\hline
Method              & VGG16               & VGG19               & ResNet34            & ResNet152           & Dense121            & Dense161             \\
\hline
Trivial Ensemble    & 81.4 ± 0.3          & 81.8 ± 0.2          & 82.7 ± 0.2          & 83.2 ± 0.3          & 84.4 ± 0.2          & 85.0 ± 0.3           \\
Support Ensemble    & 82.1 ± 0.2          & 82.4 ± 0.3          & 83.0 ± 0.2          & 83.7 ± 0.3          & 84.8 ± 0.2          & 85.5 ± 0.2           \\
Trivial Branch      & 81.0 ± 0.2          & 81.1 ± 0.3          & 82.4 ± 0.2          & 82.9 ± 0.3          & 84.1 ± 0.3          & 84.8 ± 0.3           \\
Support Branch      & 81.5 ± 0.3          & 81.8 ± 0.3          & 82.8 ± 0.2          & 83.4 ± 0.3          & 84.6 ± 0.2          & 85.4 ± 0.2           \\
ST-ProtoPNet (ours) & \textbf{82.9 ± 0.2} & \textbf{83.2 ± 0.2} & \textbf{83.5 ± 0.1} & \textbf{84.1 ± 0.2} & \textbf{85.4 ± 0.2} & \textbf{86.1 ± 0.2}  \\
\hline
\end{tabular}

}
\vspace{-6pt}
\caption{Ablation study of the combination of support and trivial prototypes for improved classification on cropped CUB-200-2011.}
\label{tab:ablationEnsemble}
\vspace{-9pt}
\end{table}

\section{Conclusion and Future Work}

In this paper, we propose the ST-ProtoPNet method to exploit both support (i.e., hard-to-learn) and trivial (i.e., easy-to-learn) prototypes for the interpretable image classification, where the two distinctive sets of prototypes can provide complementary results.
In addition, our ST-ProtoPNet is a general approach that can be easily applied to existing prototype-based interpretable models.
One limitation for our method is that we empirically adopt the same number of support and trivial prototypes and the same total number of prototypes for each class.
Considering the different learning difficulties and imbalanced training samples among classes in other real-world datasets, e.g., ImageNet \cite{deng2009imagenet}, a better way to adaptively learn a flexible number of support and trivial prototypes is needed and deserves to be further investigated in our future work.
Furthermore, given that we mimic the behaviour of the support vectors of SVM classifier to obtain the support prototypes by forcing them to be as close as possible to the classification boundary, we plan to develop new methods to learn prototypes with gradient-based kernel computations, e.g., neural tangent kernel \cite{jacot2018neural} and path kernel \cite{domingos2020every}.

{\small
\bibliographystyle{ieee_fullname}

\section{Training Details}
\label{sec:traindetails}

In our experiments, the proposed ST-ProtoPNet is trained with Adam \cite{kingma2014adam} optimiser.
The training process starts from a warm-up stage for 5 epochs, where the pre-trained CNN backbone is frozen and we only optimise the add-on layers and prototype layers, using a learning rate of $3 \times 10^{-3}$.
Then, we jointly train the CNN backbone, add-on layers, and prototype layers, where the learning rate for the CNN backbone is chosen as $1 \times 10^{-4}$ (CUB \cite{wah2011caltech}), $1 \times 10^{-4}$ (Cars \cite{krause20133d}), $1 \times 10^{-5}$ (Dogs \cite{khosla2011novel}).
In the third stage of FC layer optimisation, we utilise a learning rate of $1 \times 10^{-4}$ for 10 iterations.
In cropped CUB and cropped Cars, we decrease the learning rate by a factor of 0.2 per 2 epochs and use a weight decay ($5 \times 10^{-4}$ for VGG and ResNet, $1 \times 10^{-3}$ for DenseNet) on the CNN backbone and add-on layers.
In full CUB, we decrease the learning rate by a factor of 0.2 per 20 epochs, a weight decay of $1 \times 10^{-4}$ is applied on all CNN backbones and add-on layers.
In full Dogs, we decrease the learning rate by a factor of 0.2 per 10 epochs, a weight decay of $5 \times 10^{-4}$ is applied on all CNN backbones and add-on layers.
For cropped CUB and cropped Cars, the two $1 \times 1$ convolutional layers in the add-on layers are activated by a tanh function, as a form of normalisation.
Note that, we alternate the optimisation of each branch of ST-ProtoPNet among mini-batches during training.

We implemented our ST-ProtoPNet method with Pytorch \cite{paszke2019pytorch} framework and performed experiments on a machine with 4 NVIDIA GeForce RTX 3090 Ti GPUs.

\section{Additional Results on Cars and Dogs}
\label{sec:additionalresults}

Tables \ref{tab:interprlevel_cars} and \ref{tab:interprlevel_dogs} demonstrate the classification results of deep-learning based methods that have different levels of interpretability on Stanford Cars and Stanford Dogs datasets, respectively.

As we can observe in Table \ref{tab:interprlevel_cars}, an ensemble of three ST-ProtoPNets (based on VGG19, ResNet34, and DenseNet121) reaches a high accuracy of 93.8\% on Stanford Cars, outperforming the three ensembled TesNet method (93.1\%) that uses the same CNN backbones.
Notice that our three ensembled ST-ProtoPNet also exceeds the three ensembled ProtoPNet, ProtoTree, and ProtoPool methods by a margin of at least 2.4\%.
Additionally, the ensemble of five ST-ProtoPNets, based on VGG19, ResNet34, ResNet152, DenseNet121, and DenseNet161, achieves the best classification accuracy of 94.1\%.

In Table \ref{tab:interprlevel_dogs}, we can see that a single ResNet152 based ST-ProtoPNet model has a high accuracy of 87.2\% on full Stanford Dogs.
An ensemble of three ST-ProtoPNets, based on ResNet50, ResNet152, and DenseNet161, performs better than other three ensembled methods utilising the same CNN backbones, e.g., ProtoPNet and Deformable ProtoPNet.
The ensemble of five ST-ProtoPNets, using VGG19, ResNet50, ResNet152, DenseNet121, and DenseNet161, obtains the state-of-the-art classification accuracy of 88.7\%, surpassing other competing methods that are also based on an ensemble of five models, such as ProtoPNet and Deformable ProtoPNet.

\begin{table}[]
\setlength{\tabcolsep}{1.0 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{llc}
\hline
Interpretability                                   & Method                & Accuracy       \\
\hline
None                                               & B-CNN \cite{lin2015bilinear}                 & 91.3           \\
\hline
Object-level attention                             & CSG \cite{liang2020training}                   & 91.6           \\
\hline
\multirow{3}{*}{Part-level attention}              & MA-CNN \cite{zheng2017learning}               & 92.8           \\
                                                  & RA-CNN \cite{fu2017look}               & 92.5           \\
                                                  & TASN \cite{zheng2019looking}                 & \textbf{93.8}  \\
\hline
\multirow{9}{*}{Part-level attention + Prototypes} & Region \cite{huang2020interpretable}               & 90.9           \\
                                                  & ProtoPNet* \cite{chen2019looks}           & 91.4           \\
                                                  & ProtoTree* \cite{nauta2021neural}           & 90.5           \\
                                                  & TesNet* \cite{wang2021interpretable}              & 93.1           \\
                                                  & ProtoPool* \cite{rymarczyk2022interpretable}           & 91.1           \\
                                                  & ST-ProtoPNet* (ours)  & 93.8           \\
                                                  & ProtoTree** \cite{nauta2021neural}          & 91.5           \\
                                                  & ProtoPool** \cite{rymarczyk2022interpretable}          & 91.6           \\
                                                  & ST-ProtoPNet** (ours) & \textbf{94.1}  \\
\hline
\end{tabular}

}
\caption{Classification accuracy and interpretability of different methods on Stanford Cars. *: Ensembled of three models. **: Ensembled of five models. }
\label{tab:interprlevel_cars}
\end{table}

\begin{table}[]
\setlength{\tabcolsep}{0.5 mm}
\resizebox{\linewidth}{!}{

\begin{tabular}{llc}
\hline
Interpretability                                   & Method                 & Accuracy       \\
\hline
\multirow{2}{*}{Part-level attention}             & FCAN \cite{liu2016fully}                  & 84.2           \\
                                                  & RA-CNN \cite{fu2017look}                & 87.3           \\
\hline
\multirow{9}{*}{Part-level attention + Prototypes} & ProtoPNet \cite{chen2019looks}             & 79.7           \\
                                                  & Deformable ProtoPNet \cite{donnelly2022deformable}  & 86.5           \\
                                                  & ST-ProtoPNet (ours)    & 87.2           \\
                                                  & ProtoPNet* \cite{chen2019looks}            & 83.6           \\
                                                  & Deformable ProtoPNet* \cite{donnelly2022deformable} & 87.1           \\
                                                  & ST-ProtoPNet* (ours)   & 88.6           \\
                                                  & ProtoPNet** \cite{chen2019looks}           & 84.1           \\
                                                  & Deformable ProtoPNet** \cite{donnelly2022deformable} & 87.3           \\
                                                  & ST-ProtoPNet** (ours)   & \textbf{88.7}  \\
\hline
\end{tabular}

}
\caption{Classification accuracy and interpretability of different methods on Stanford Dogs. *: Ensembled of three models. **: Ensembled of five models. }
\label{tab:interprlevel_dogs}
\end{table}

\section{More Examples of Prototype Visualisation}
\label{sec:prototype}

In Fig.~\ref{fig:visualprototypescar} and Fig.~\ref{fig:visualprototypesdog}, we provide more examples of visual prototypes learned by our top-performing models on Stanford Cars and Stanford Dogs datasets.
Specifically, we display prototypes of Dense161-based and ResNet152-based ST-ProtoPNet for Stanford Cars and Stanford Dogs, respectively.
Following \cite{chen2019looks,donnelly2022deformable}, all these prototypes are visualised by projecting onto their nearest training patch in the latent feature space.

\subsection{Stanford Cars}

In Fig.~\ref{fig:visualprototypescar}, we illustrate the visual prototypes of four classes (Acura RL Sedan 2012, Acura TL Sedan 2012, Acura TL Type-S 2008, and Acura TLS Sedan 2012) learn from cropped Stanford Cars dataset.
We can notice from Fig.~\ref{fig:visualprototypescar}(a) that the support prototypes mainly capture visually similar features of different classes, such as light and front bumper.
By contrast, the trivial prototypes in Fig.~\ref{fig:visualprototypescar}(b) are inclined to focus not only on other car parts, such as wheel, door, and window.
For the cropped Cars dataset, the target objects of cars usually occupy a large area in the whole image, so we rarely observe background regions in trivial prototypes.

\begin{figure*}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{cvpr2023-author_kit-v1_1-1/latex/figures/car_prototypes.pdf}

    \caption{Visual comparison between the support (a) and trivial (b) prototypes from cropped Stanford Cars, where each row exhibits prototypes of the same class. In each pair, the first column shows the original image with a prototype indicated in a yellow bounding box, the second column demonstrates the prototype's corresponding activation map. }
    \label{fig:visualprototypescar}
    \vspace{10pt}
\end{figure*}

\subsection{Stanford Dogs}

In Fig.~\ref{fig:visualprototypesdog}, we demonstrate typical prototypes of five classes (Rhodesian Ridgeback, Toy Terrier, Komondor, Rottweiter, and Dhole) from full Stanford Dogs dataset.
We can observe from Fig.~\ref{fig:visualprototypesdog}(a) that the support prototypes often contain similar visual patterns of different classes, e.g., dog's eye, mouth, and head.
By contrast, trivial prototypes in Fig.~\ref{fig:visualprototypesdog}(b) are usually from dog's neck, leg, and belly (lower surface) regions.
We also notice that two trivial prototypes capture background regions (the rightmost ones of Rottweiter, and Dhole classes).

\begin{figure*}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{cvpr2023-author_kit-v1_1-1/latex/figures/dog_prototypes.pdf}

    \caption{Visual comparison between the support (a) and trivial (b) prototypes from full Stanford Dogs, where each row exhibits prototypes of the same class. In each pair, the first column shows the original image with a prototype indicated in a yellow bounding box, the second column demonstrates the prototype's corresponding activation map. }
    \label{fig:visualprototypesdog}
    \vspace{10pt}
\end{figure*}

\section{More Examples of Interpretable Reasoning}
\label{sec:reasoning}

We also provide typical examples of the interpretable reasoning process of our ST-ProtoPNet method on Stanford Cars (Dense161-based) and Stanford Dogs (ResNet152-based) datasets, as illustrated in Fig.~\ref{fig:reasoningcar} and Fig.~\ref{fig:reasoningdog}.
For each example, we display two support and trivial prototypes of the predicted class as well as the corresponding source training images where these prototypes come.
For a testing image, the activation maps and highest similarity scores with these prototypes are also provided.
For simplicity, we only demonstrate the prototypes, activation maps, and similarity scores contributing to the predicted class for each example.

\subsection{Stanford Cars}

Fig.~\ref{fig:reasoningcar} displays the interpretable reasoning of our ST-ProtoPNet method in classifying a testing car (Acura TL Type-S 2008) image.
In each ProtoPNet branch, our method compares the testing image with training prototypes and computes the corresponding similarity scores, which is then weighted and summed to produce the classification logits.
The final classification logits is the sum of logits from both branches.
In particular, the support prototypes are mostly active on the fog light and headlight of the car.
Meanwhile, the trivial prototypes have high activations on the car's window and wheel.
In this case, the support ProtoPNet branch generates a comparably larger similarity score (17.840) in comparison with the trivial branch (16.698), which means the testing car image is more similar to the support prototypes than the trivial ones.
We can see that the support and trivial prototypes make complementary predictions contributing to the final classification decision.

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{cvpr2023-author_kit-v1_1-1/latex/figures/car_reasoning.pdf}

    \caption{An example of the interpretable reasoning of our ST-ProtoPNet for classifying a testing car (Acura TL Type-S 2008) image. }
    \label{fig:reasoningcar}
    \vspace{10pt}
\end{figure}

\subsection{Stanford Dogs}

Fig.~\ref{fig:reasoningdog} reveals the interpretable reasoning of our ST-ProtoPNet method in classifying a testing dog (Rhodesian Ridgeback) image.
To be specific, when classifying the Rhodesian Ridgeback dog, its head is usually activated by the support prototypes while its neck and belly are often activated by the trivial prototypes.
We should notice that for the full Stanford Dogs dataset, we use cosine similarity \cite{donnelly2022deformable}, instead of projection metric \cite{wang2021interpretable} used in cropped Stanford Cars, to measure the similarity between the testing image's feature map and prototypes, so the similarity scores are within the range of [0, 1].
In this example, the support and trivial ProtoPNet branch obtains an accumulated similarity score of 35.902 and 35.583, respectively.
By using both support and trivial prototypes, ST-ProtoPNet can capture richer representations of the target object (dog in this example) from different perspectives, benefiting the final classification decision in a complementary way.

\begin{figure}[t!]
    \centering
    \includegraphics[width=1.00\linewidth]{cvpr2023-author_kit-v1_1-1/latex/figures/dog_reasoning.pdf}

    \caption{An example of the interpretable reasoning of our ST-ProtoPNet for classifying a testing dog (Rhodesian Ridgeback) image. }
    \label{fig:reasoningdog}
    \vspace{10pt}
\end{figure}

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv


text:
by selfpublishing.com
How to Make Money Writing
How to Become a Freelance Writer
COACHING + PUBLISHING
FORMATTING + DESIGN
FREELANCE COMMUNITY
How Freelance Writers Can Create a Killer Resume
Evan Tice
By Carol Tice
Resumes have been important to writers forever. Writing a strong one could really help you land better gigs.
I recently took a writer survey on my blog about what readers would like to learn, and I got a request from Catherine to talk about how to create a killer writer's resume. So I'm going to answer that now:
I don't think a resume is important for writers anymore.
Before you run around the room screaming and tearing your hair, let me explain why.
I don't believe anyone actually reads resumes anymore. I'll send my resume out on job ads that say a resume is an absolute requirement. Otherwise, I'd never think to include it in my pitches or marketing to prospective clients. I believe even the job-ad posters skip right over the resume and look at your pitch or your clips, anyway.
Resumes are boring. Seriously. "Joe freelanced for Modern Refrigeration Magazine from 2002-2009″…zzzzz. Is that really putting you in the best light?
Resumes don't tell much about how good of a writer you are. You might have worked as a staff writer somewhere for years, where you were always considered the weak link in the writing team.
Increasingly, markets don't care about your track record. If you've got a couple of solid clip links you can email, you're good, especially with online markets. Many editors and marketing managers don't have time to study your entire career — they read a couple clips and decide you're good for it.
So if resumes are obsolete, what's replacing them?
For now, a short bio. When I'm asked for a resume, unless it's a job-ad robot Web site where I can't progress without attaching my resume file, I direct the prospect to look at my short bio on my Writer site. I believe it is far more compelling and enlightening in describing my background. It's less than a page long, despite my having been at this for about 20 years, so it mercifully sums up a lot, fast.
Consider that a good query or job pitch has a one-line bio at the bottom of the pitch page. That ought to do it.
The bio format allows you to simply tell the story of your writer's journey — where you've written for, the type of work you do, the type of writing you enjoy.
People like to read stories way more than they like to read lists of jobs you've had previously. The bio format also makes it easier to throw any awards you've won up near the top. I find many prospects are easily impressed by awards, so getting them up high is a good move.
Also, the bio format allows you to top your story with the best credits you've got. There's no compulsion to put things in chronological order.
For instance, I once wrote a couple articles for the college edition of the Wall Street Journal (before the Internet, darnit). I'm going to say it was about a decade ago. But in a bio, I could put that in the first line, since it's such a smokin' hot credit. On my resume, it's so long ago it wouldn't even make the second page — which as we all know is a page nobody reads.
Beyond bios
In the future, both resumes and bios look to be headed for the scrap heap. New, cooler ways of acquainting people with what we do are emerging.
One I recently learned about is Labels.io. Still in beta, this site allows you to present your experience in a concise, nifty graphical package. You create a bunch of quick tabs labeled by past client. You give it a top paragraph to introduce the package, and you're set. Load in some key words on jobs you'd like to be found for and presto — it's easy for prospects to locate you and verify you have the experience they want with a couple of quick clicks.
I'm confident Labels won't be the only graphical alternative-resume idea we'll see in the next couple of years.
As someone who's had to review resumes and hire writers myself (edited an alternative paper in San Pedro, Calif., for a year or so early in my career), I can say I look forward to the changes. Resumes are dull, and at this point the Internet ought to offer a better way for us to get hired.
This post originally appeared on the WM Freelance Writer's Connection.
Photo via Flickr user
How to Spy on Your Blog Using Google In-Page Analytics: Video
How One Writer's Blog Hit the Top 10
A brand new opportunity for writers has finally arrived!
It's a true blessing that these days there are so many ways to make a living through our writing work. From freelance writing through to editing and building a blog, you can make great money doing what you love. Sadly, some of the most rewarding ways of making...
Best Accounting Software for Freelancers: Freshbooks vs Quickbooks vs Wave (& More)
Let's be honest -- not every part of running a freelancing business is fun. There are some tedious, headache-inducing tasks that come with running a business that are simply unavoidable...like bookkeeping. Keeping track of all business-related income and...
WordGigs Review — Is It Worth It? (2022)
When it comes to finding writing gigs, there are a million places to choose from. You might be looking for a WordGigs review and trying to figure out whether you should go through the application process to become a freelance writer for their site. This WordGigs...
COMMUNITY FOR FREELANCE WRITERS
Copyright 2008 – 2023 MAKEALIVINGWRITING

redpajama_set_name: RedPajamaCommonCrawl


breastfeeding nursing cover bamboo style ideas for mums sewing pattern.
breastfeeding nursing cover baby mum up udder covers shawl cotton blanket bard peg tube feeding formula tubes from nz.
breastfeeding nursing cover baby for wide hooter hider breathable co sewing pattern.
breastfeeding nursing cover cotton covers shawl breast feeding flower printed for baby sewing pattern.
breastfeeding nursing cover learn how to make your own with this fantastic tutorial shawl.
breastfeeding nursing cover china best covers.
breastfeeding nursing cover 1 burp cloth by baby car seat covers shopping cart high chair stroller and canopy stretchy.
breastfeeding nursing cover privacy full coverage baby diy.
breastfeeding nursing cover covers baby car seat canopy scarf up apron shawl cape udder.
breastfeeding nursing cover covers cotton pregnant breast feeding flower printed for baby in from mother udder n.
breastfeeding nursing cover breast feeding up.
breastfeeding nursing cover baby in style with scarf diy.
breastfeeding nursing cover women mum udder covers privacy wrap baby infant cotton blanket shawl poncho cloth small feeding brea.
breastfeeding nursing cover scarf black shawl.
breastfeeding nursing cover muslin cotton china simplicity poncho up.
breastfeeding nursing cover maternity apron covers baby poncho cotton diy.
breastfeeding nursing cover maternity apron covers baby poncho cotton from.
breastfeeding nursing cover lightweight breathable cotton apron for rigid neckline udder covers grace.
breastfeeding nursing cover simplicity poncho up.
breastfeeding nursing cover udder covers best.
breastfeeding nursing cover how to sew a tutorial shawl.
breastfeeding nursing cover prints available best covers.
breastfeeding nursing cover baby infant breathable cotton floral style wrap cove for pregnant women in belly bands support from covers.
breastfeeding nursing cover mum scarf shawl swaddle blanket rayon covers.
breastfeeding nursing cover discreet udder covers.
breastfeeding nursing cover covers diy.
breastfeeding nursing cover scarf baby car seat canopy diagonal stripe shawl covers.
breastfeeding nursing cover wholesale organic special baby mum breathable multi functional fashion poncho scarf udder covers.
breastfeeding nursing cover great covers for in public sewing pattern.
breastfeeding nursing cover new design women outdoor scarf baby car seat blankets free shipping up.
breastfeeding nursing cover the feeder frock baby signature pocket style pick your color via covers.
breastfeeding nursing cover scarf up.


text:
Entrance to Wood
by Neftalí Ricardo Reyes Basoalto
Themes and Meanings
"Entrance to Wood" is essentially a mystical poem. In spite of its simple subject—wood (celery and wine are the subjects of the other two of the "Three Material Cantos")—"Entrance to Wood" is a poem written in the manner of the best religious poetry of the Spanish golden age. In this...
Start your 48-hour free trial to unlock this Entrance to Wood study guide. You'll get access to all of the Entrance to Wood content, as well as access to more than 30,000 additional guides and more than 350,000 Homework Help questions answered by our experts.
"Entrance to Wood" is essentially a mystical poem. In spite of its simple subject—wood (celery and wine are the subjects of the other two of the "Three Material Cantos")—"Entrance to Wood" is a poem written in the manner of the best religious poetry of the Spanish golden age. In this poetry, the speaker, through a total renunciation of the senses, usually rises to a new intellectual level and, ultimately, to spiritual communion with God. In Neruda's poem, through a structured reversal of this procedure, the spiritual aspect is minimized and sensory perceptions are maximized as the speaker bodily falls down toward a physical union with earthly things. What at first appears to be a fall toward death is actually a fall toward life. Neruda uses familiar religious symbols to represent the vital union, not with God, but with matter, "sweet matter."
The mystical theme is underscored by the speaker's material/spiritual journey. The poem's rhythms move forward ritually. The speaker's sinking through time and space creates the potential for soaring above them. Having reached the dark core of the wood (an inverted dark night of the soul), having exposed himself to matter at this depth, the speaker (and the poet) can join in its essential dialectic of death and life.
At the end of the poem, the speaker asks the wood to act along with him. Linking his "fallen soul" to his own night falling like "broken" water, he makes the wood's life and death his own. More precisely, it is he who breathes life into the wood by bringing his own spirit to its "crushed materials." It is the poet and the wood, human and nonhuman, that unite, ignite, and burn with pure passion at the poem's end. In the act of writing, the poet and his "matter," his poetry, are one.
Download Entrance to Wood Study Guide
Ode to an Artichoke
The Heights of Macchu Picchu
A Lemon
Canto general
Dead Gallop
A Few Things Explained
The Hunter in the Forest
The Poetry of Neruda
A Smell of Cordwood


This traditional worship service is a slightly shorter version of the late service. While there is typically no choral music at this service, there is often special music offered by our church musicians or other soloists. Nursery care is provided, and worship bags are available for preschool and elementary aged children.
You are invited to join us for Meet and Greet in Kirk Hall beginning at 9:30am.
The late service features traditional music from our Chancel Choir, as well as a Time for Children, and occasional musical offerings by other choirs and special guests. Children ages 4k – 1st grade are invited to attend Children, Worship, and Wonder following the Time for Children. Nursery care is provided, and worship bags are available for preschool and elementary aged children.


text:
Duke Dennis Net Worth 2022: Income, Facts, Family & More
December 31, 2021 September 28, 2021 by Amelia Warner
Duke Dennis net worth
Duke Dennis is a famous American YouTube gamer and video streamer. He uploads videos related to basketball simulating games NBA 2K. His YouTube channel has been growing rapidly and attracted a massive number of fans over the years. Duke Dennis's net worth is around $10 million.
Name Duke Dennis
Date of Birth February 26, 1994
Birthplace United States of America
Marital Status Not known
Girlfriend Not known
Weight 62 kg/ 136.6lbs
Eyes Color Dark brown
Hair Color Brownish
Zodiac Sign Pisces
Annual Income $17.6 K to $282 K
Monthly Income $1.6K to $23.5K
Daily Income $900
Tattoo Yes
Hobbies Basketball
How Much Money Does Dennis Make Per Day?
In 2020 Duke Dennis's channel had 1.2 million subscribers and so far has approximately 140 million views. His channel has an average of 180,000 views per day. You tubers get $2 to 7$ for 1000 views on a channel. So Duke Dennis should get paid 900$ per day according to the calculations.
How Much Money Does Dennis Make Per Month?
Youtubers also get paid monthly for their content. The viewers pay a monthly fee to watch premium content on YouTube without ads. It allows the YouTubers to get paid based on the watch time on their videos. Dennis's monthly income is $1.6K to $23.5K.
Duke Dennis /Photo
How Much Money Does Dennis Make Annually?
The YouTube channel of Duke Dennis is quite famous for the Basketball simulation games. It has attracted many subscribers to the channel. Duke Dennis's yearly income is $17.6 K to $282 K.
Main Source of Income
Duke Dennis's income comes mainly from his YouTube channel, where he uploads videos of basketball simulation games. He also has his Instagram and Twitter accounts. Duke Dennis's net worth from the Youtube channel is approximately $10 million.
Early Life of Duke Dennis
Duke tennis was born on February 26, 1994, in the United States of America. He is 17 years old and is a Pisces. One of the best qualities of Duke Dennis is that he works passionately to be the best in his field. His fans are very impressed with him and help him to achieve his goals.
He was enrolled in a catholic school for his early education. His personal life is not known yet as he hasn't talked about it yet.
Duke Dennis net worth & family
Duke Dennis took birth in an African-American family. He's a Christian by birth. There is no other information regarding Duke Dennis's family. He likes to keep his personal life separate from his
professional life. However, he has one sibling named Darius Dennis.
Duke Dennis Girlfriend
There has never been a rumor of Duke Dennis being in a relationship with someone. Information about his previous relationships is not known.
Duke Dennis Hobbies
Duke tennis is a fan of basketball and loves to play it as a sport. He has made his career in the basketball simulation game. He also likes to take part in online competitions.
Duke Dennis Height
Duke Dennis has an attractive height of 5 feet 8 inches (1.72 meters or 172 centimeters).
Duke Dennis Weight
Duke Dennis's weight is perfect for someone of his height. His eight is 62 Kgs or 136.6 lbs.
Read Also: Techno Gamerz Net Worth
Some Facts About Duke Dennis
Duke Dennis likes to have his body full of tattoos.
He likes to keep his love life separate. He has not disclosed any information regarding his girlfriend or any other relationships.
According to the information known about Duke Dennis, one can say that he is single and keep his attention solely on his career.
He has imitated professional basketball in his many videos.
He got the golden YouTube button on May 3, 2020. He has also shared pictures with the said button.
He like boxing
Duke Dennis is a huge fan of the NBA and Lakers.
He has 410 K followers on his Instagram account.
Duke uploaded his first video on YouTube on October 29, 2016. The video's title was "NBA 2K17 Dropped on Superstar Head Best Jumpshot in NBA 2k17."
Duke Dennis is the best in his career. His personal life never dominates his professional life. Duke Dennis's primary source of income is from YouTube. Duke Dennis's net worth is approximately $10 million.
Categories Games, Net Worth Tags Duke Dennis, Net Worth Post navigation
Holderness Family Net Worth 2022: Income, Facts, Family, Bio
Roger Hunt Died: What Was His Cause Of Death?
Coryxkenshin Net Worth 2022: Age, Height, Dating, Earnings
Cody Ko Net Worth 2022: Height, Age, Career, Girlfriend & More


text:
Pre-Christian Slavic writing is a hypothesized writing system that may have been used by the Slavs prior to Christianization and the introduction of the Glagolitic and Cyrillic alphabets. No extant evidence of pre-Christian Slavic writing exists, but early Slavic forms of writing or proto-writing may have been mentioned in several early medieval sources.

Evidence from early historiography
The 9th-century Bulgarian writer Chernorizets Hrabar, in his work An Account of Letters (, O pismenĭhŭ), briefly mentions that, before becoming Christian, Slavs used a system he had dubbed  "strokes and incisions" or "tallies and sketches" in some translations (Old Church Slavonic: , črŭty i rězy). He also provided information critical to Slavonic palaeography with his book.

Another contemporaneous source, Thietmar of Merseburg, describing a Rethra temple remarked that the idols there had their names carved out on them ("singulis nominibus insculptis," Chronicon 6:23).

Evidence from archaeology

In 1949, a Kerch amphora was found in Gnezdovo in Smolensk Oblast, Russia, containing what may be the earliest inscription in Old East Slavic. The amphora was found in the grave of a Scandinavian merchant who traded with the Orient. The excavator read the one-word inscription as Cyrillic "гороухща" (goruhšča), inferring that it designated mustard that was kept there. This explanation has not been universally accepted and the inscription seems to be open to different interpretations. Other scholars have read the apparent "хщ" letter combination as an "N", resulting in a transcription as "гороуна" which could be either Cyrillic or Greek and which might represent an Arabic name Hārūn with possessive suffix (Härün's amphora). The inscription is dated to the early 10th century, which suggests a hitherto unsuspected popularity of the Cyrillic script in pre-Christian Rus. Different sources prove that the Slavic and Norse population of Russia had trade connections to the Muslim and Asian worlds.

 In 2021 new archaeological evidence of an early instance of writing among a Slavic population was published. In 2017 a cattle bone, dated 585–640 AD, with runes of the Germanic Elder Futhark was discovered in a Prague culture settlement near Břeclav in the Czech Republic. The inscription contains no recognizable words in either Slavic or any other language but seems to represent an attempt at an abecedary. It may have been incised by people of Germanic origin that remained in the region after the departure of the Lombards, or the runes may have been engraved by a Slav. In either case, the find attests to a direct interaction between the Slavic and Germanic ethnolinguistic groups.  If runic knowledge was transferred from Germanic peoples to Slavs, it must have happened in Central Europe as judged by the rune shapes or it may have persisted in the region as a result of population continuity between Lombards and Slavs.

Evidence against
In the Vita Cyrilli, Rastislav, the duke of Moravia, sent an embassy to Constantinople asking Emperor Michael III to send learned men to the Slavs of Great Moravia, who being already baptised, wished to have the liturgy in their own language, and not Latin and Greek. Emperor called for Constantine and asked him if he would do this task, even though being in poor health. Constantine replied that he would gladly travel to Great Moravia and teach them, as long as the Slavs had their own alphabet to write their own language in, to which the Emperor replied that not even his grandfather and father and let alone he could find any evidence of such an alphabet. Constantine was distraught, and was worried that if he invents an alphabet for them he'll be labelled a heretic.

According to Alexey Karpov, this text is a later insertion in the chronicle, and its authenticity is questioned.

Footnotes

See also
Proto-writing
Christianization of the Slavs

References

Slavic culture
Palaeography
Slavonic inscriptions
Proto-writing

redpajama_set_name: RedPajamaWikipedia


text:
La  des , décernés par l', a eu lieu le , et a récompensé les programmes télévisés internationaux diffusés au cours de la saison 2011-2012.

Palmarès

Meilleure performance d'acteur
 Darío Grandinetti dans Televisión por la Inclusión
 Arthur Acuña dans The Kitchen Musical
 Jason Isaacs dans Case Histories
 Stein Winge dans Koselig med peis
 Zhu Yawen dans Yuan Qu De Fei Ying

Meilleure performance d'actrice
 Cristina Banegas dans Televisión por la Inclusión
 Sidse Babett Knudsen dans Borgen
 Rina Sa dans Zhong Guo Di
 Joanna Vanderham dans The Runaway

Meilleure série dramatique
 Braquo
 ICAC Investigators
 The Kitchen Musical
 The Slap
 The Social Leader

Meilleure comédie
 Absolutely Fabulous  A Mulher Invisível
 Spy
 Wat Als?

 Meilleure mini-série ou meilleur téléfilm
 Black Mirror  Early Autumn
 The Good Men
 L'Infiltré

 Meilleure telenovela
 O Astro  Rosa Fogo
 Remédio Santo
 Iron Daughters-in-Law

 Meilleur documentaire
 Terry Pratchett: Choosing to Die  Across Land, Across Sea
 Hitler's Escape
 Wettlauf Zum Südpol

 Meilleur programme artistique
 Songs of War  All My Life
 Queen: Days of Our Lives
 Blue Man

 Meilleur programme de divertissement non-scénarisé
 The Amazing Race Australia''
 The Challenger Muaythai
 El Hormiguero
 Planeta Extremo''

Directorate Award
 Kim In-Kyu

Founders Award
 Alan Alda et Norman Lear
 Ryan Murphy

Notes et références

Annexes

Articles connexes

  des
  des

Liens externes

  Les  par année sur l'

40
International Emmy Awards
Novembre 2012 aux États-Unis


text:
Part of HuffPost Food & Drink. ©2023 BuzzFeed, Inc. All rights reserved.
Food & DrinkFoodgordon ramsaygordon ramsay sued
Gordon Ramsay's Unpaid Bills Rack Up: Now Meat Supplier Says 200K Owed
The saga of "Hell's Kitchen" star Gordon Ramsay's unpaid restaurant bills continues. Last week, the New York Post reported that two wine suppliers have filed suit against the chef for tens of thousands of dollars of wine delivered to Ramsay's restaurant at the London Hotel in New York's midtown, but were never paid for. Wine importer V.O.S. Selections is suing Ramsay for $38,000 in unpaid bills, and Wineberry America for $40,900.
Other lawsuits abound, including $89,000 Ramsay is said to owe for a cleaners' bill at his restaurant at the London Hotel in West Hollywood.
And today, the New York Post's Page Six reports that Bronx-based meat supplier Endicott meats is claiming Gordon Ramsay owes them $190,000 for various products delivered to his restaurant in the London Hotel in New York. Accounting for the figure are deliveries of everything from duck breasts to racks of lamb to sweetbreads and bacon, according to Michael Braunschweiger, owner of Endicott Meats.
"Just like with the wine guys, he basically stopped responding to me in November," Braunschweiger told the Post, adding, "He skipped out on the bill and doesn't answer calls... For a guy who goes on TV and tells people how they should be running their business, maybe he should first learn how to run his own. It seems like he spends more time on TV and scouring for women, or whatever it is he does these days." Braunschweiger was alluding to an additional scandal surrounding Ramsay -- accusations, denied by the chef, that he cheated on his wife with at least three other women in 2008.
The New York Post has additional comments from Braunschweiger, and notes that Ramsay's reps at William Morris Endeavor have not responded to requests from comment.
Foodgordon ramsaygordon ramsay suedHell's Kitchen
More In Food & Drink
Your Body On Alcohol: How It Affects Your Heart, Liver, Weight And Cancer Risk
Everyone's Freaking Out About This Discontinued Pasta Shape. Here's Why It's A Big Deal.
Chefs Share The One Kitchen Tool That Helps Them Eat Healthy
The Best Instant Oatmeal At The Grocery Store, According To Nutritionists
Pro Chefs Share Secrets To Making Microwaved Food Taste Amazing
Nordstrom Has So Many Bras With Five-Star Ratings
Some Of Allbirds' Bestselling Shoes And Sneakers Are On Sale
8 Home Goods From Etsy That Could Be From A High-End Design Store
The Best And Worst 'Healthy' Drinks At The Grocery Store, Ranked By Nutritionists


text:
Last year's Class A runner-up Rockford knocked out of the boys basketball tournament on the first night
By: Jeff Chaney, March 5, 2013, 9:00 am
LOWELL, MI – Last year the Rockford boys basketball team had a magical run to the Breslin Center.
The Rams advanced all the way to the Class A state title game at the Breslin Center on the campus of Michigan State University.
Rockford lost to Saginaw in the championship game, but it was a season of memories for the Rams.
The run this year was a lot shorter, as Rockford fell to Forest Hills Northern on the first night of the MHSAA state tournament Monday night at Lowell High School, 58-55.
"Any tough is loss, if you lose on Monday night of the tournament or Saturday night at the Breslin Center," said Rockford coach Nick Allen, whose team ends its season with a 14-6 record. "It didn't work out for us tonight, it was a tough night."
Prospects looked good for Rockford early on, as the Rams forced an up-tempo game and hit six 3-pointers in the first 16 minutes of the game to take a 31-27 lead to halftime.
But in the second half, and more specifically in the third quarter, Forest Hills Northern began to take advantage of its size in the paint to turn the game around.
With Rockford still up by four with six minutes to play in the third quarter, the Huskies began to move the ball in the paint and let their beg men, Jamal White, Jayvon Bennett and Trey Moore take over.
And that trio scored 10 points in a 12-3 run to end the quarter that gave Forest Hills Northern a 44-39 lead heading into the final quarter.
Forest Hills Northern also out-rebounded the Rams in the third, 8-2.
From there, the Huskies were able keep Rockford in check the final eight minutes, thanks in part to the Rams going a bit cold from behind the 3-point arch.
Rockford connected on just two of 12 3-pointers in the second half.
Forest Hills Northern coach Joe Soules was thrilled to open his 2013 state tournament with a win against the defending runner-ups.
"(Rockford assistant coach Ryan DeKuiper) and Nick do such a fabulous job there," said Soules, whose team improved to 15-6 with the win. "They are so well coached with tough players, they really get after it. This is a big win for the team."
Forest Hills Northern had only two players in double figures, as senior point guard Michael McLaughlin led the team with 13 points, while White chipped in with 10.
The Rams were paced by senior guard Chase Fairchild's game-high 18 points.
Allen said it was a tough post-game speech to his team, especially to the six seniors on the team.
"Life is full of learning experiences," Allen said. "I told the team I hope we can learn from this night and use it in life."