text:
\section{Introduction}

Metric learning is widely used in vision tasks, where it is adopted to learn an embedding space that uses a specified distance metric to reasonably reflect the similarity between samples. A number of works have explored the effectiveness of metric learning in different vision tasks~\cite{schroff2015facenet,gordo2017end,brown2020smooth}.

Most deep metric learning methods employ a strategy that forces all positive samples to be close in the embedding space, while keeping them away from negative ones. To clarify, the definition of positive and negative samples is a relative concept. During training, a positive sample is defined as being in the same class as the anchor sample, and the opposite is a negative sample. Correspondingly, at the time of testing, samples of the same class as the query are positive, and vice versa. We call such a strategy \textit{hard embedding} as it only concerns the hard labels. However, \textit{hard embedding} tends to cause the following problems: (1) it makes the model ignore the internal information between samples, particularly, the similarity between intra-class samples. However, since the image retrieval task requires dealing with unseen classes, these internal information helps the model learn ``how to measure similarity'' and therefore ignoring it is detrimental to the generalization of the model. (2) The \textit{hard embedding} leads to overfitting, especially in the presence of hard samples and mislabeled samples. An interesting observation for deep models is that they can first adapt to easy samples, and gradually adapt to hard samples as the number of training epochs gets larger~\cite{han2018co}. Therefore, if hard samples are hard in nature, but we measure them with inappropriate metrics (\textit{e.g.}, the visual content of a sample is inherently less similar to other samples in the same class, or it was annotated with a wrong/noisy label, but we force its feature to be close with other intra-class samples), then the training will enter a phase of fitting hard samples with unreasonable features, which will directly lead to overfitting and poor generalization.

\begin{figure}[t]
\centering
\includegraphics[width=.85\linewidth]{img/embedding.pdf}
\vspace{-2mm}
\caption{Toy example of \textit{hard embedding} vs \textit{soft embedding}: the \textit{hard embedding} (left) forces all hard positives and mislabeled samples to be close to the anchor, which causes the model to ignore the cat's texture and overfit. The \textit{soft embedding} (right) considers assigning a more appropriate distance target to each pair of samples. Compared with the easy positive, the hard positive (Bengal cat) is less similar to the anchor and the texture is more similar to the negative (leopard), so it should be slightly further away from the anchor and closer to the negative. The same is true for the mislabeled samples.}
\label{fig:embedding}
\end{figure}

In this paper, we propose a hypothesis that learning a smoother embedding space helps to improve the generalization of metric learning. That is, assigning a more appropriate distance target to each sample pair rather than just bringing samples in a positive pair closer and keeping samples in a negative pair apart based on hard labels. Fig.~\ref{fig:embedding} shows a toy example about \textit{hard embedding} and \textit{soft embedding}. To validate our hypothesis, inspired by the knowledge distillation~\cite{furlanello2018born,qin2021born}, we propose a simple yet effective regularization named listwise self-distillation (LSD) that distills the knowledge in a model itself and uses it to soften the \textit{hard embedding}, it means that the student model itself serves as the teacher model. The teacher uses its knowledge to measure each sample pair and provides more appropriate distance targets for the student. It is worth mentioning that LSD is a listwise method that has been shown computationally more efficient and more effective for ranking problems as it directly minimizes errors in ranking of samples~\cite{cao2007learning}.

Our LSD can be directly integrated into a general DML framework. We demonstrate its effectiveness via a comprehensive set of evaluations with various metric learning methods on multiple datasets. We found that the intra-class compactness in the embedding space decreases as expected under the regularization of LSD, and the model performs significantly better than those without using LSD. In addition, we give theoretical explanations (Section~\ref{sec:theoretical}) on why LSD is effective, and show that LSD encourages the model to exploit more internal information between samples, especially for hard samples, allowing the model to better learn ``how to measure similarity''.

In summary, our contributions can be described as follows:
\begin{itemize}
    \item We present the shortcomings of the existing metric learning methods on vision tasks that emphasizes only intra-class compactness and inter-class discrepancy.
    \item We propose a hypothesis that learning a smoother embedding space helps improve the generalization of metric learning, and validate our hypothesis on public image  retrieval benchmarks.
    \item We propose a simple yet effective regularization named Listwise Self-Distillation (LSD) for metric learning. LSD can be applied to any general DML framework. This is the first general approach to improve DML performance using knowledge self-distillation. We also provide theoretical explanations on why LSD works.
\end{itemize}

\section{Related Works}
\subsection{Metric Learning}

Metric learning are widely used to optimize networks by regulating the similarity between instances in the embedding space. Many metric learning methods use pair-based objectives defined on a tuple of samples, such as pairs~\cite{hadsell2006dimensionality}, triplets~\cite{wang2014learning,schroff2015facenet} or other variant like N-pairs~\cite{sohn2016improved} and lifted structure loss~\cite{oh2016deep}. Further, AP-based losses~\cite{brown2020smooth,rolinek2020optimizing} that directly optimising the Average Precision (AP) has been proven to be effective in metric learning. Moreover, some classification-based methods~\cite{deng2019arcface} and proxy-based methods~\cite{movshovitz2017no,qian2019softtriple} have shown the ability of learning distance-preserving embedding spaces.

Additionally, more involved research extending above objectives has been proposed: \cite{milbich2020diva} proposed and studies multiple complementary learning tasks; \cite{lin2018deep} proposed a framework to explicitly model the intra-class variance and disentangle the intra-class invariance; \cite{opitz2018deep} divides a large embedding into an ensemble of several smaller embeddings and \cite{kim2018attention} proposed an attention-based ensembling framework.

Due to the large variety of methods for metric learning, it is inefficient to design a regularization method for each method. Our motivation is to propose a generalized regularization method for metric learning.
Considering that metric learning methods all require a backbone model to extract features from the input data, usually images, LSD is designed based on such features and is therefore applicable to various metric learning methods (as shown in Section~\ref{sec:ablation}).

\noindent \textbf{Remarks.} Here, we would like to highlight the difference of motivation between our LSD and some similar works \cite{milbich2020diva, lin2018deep}. \cite{lin2018deep} also proposed that we need to focus on intra-class variance, but it ignore the inter-class variance. Our ``soft embeddings'' imply not only ``intra-class'' structures but also ``inter-class'' structures. \cite{milbich2020diva} applies multiple complementary learning tasks for learning multiple relation between different samples and result in strong generalization, but all learning tasks are set manually, while our LSD applies self-distillation to set the learning target by the model itself.

\subsection{Knowledge Distillation}

Conventional Knowledge Distillation (KD)~\cite{hinton2015distilling,zagoruyko2016paying,tung2019similarity} methods aim at model compression: they use knowledge from the teacher model to generate compact models, achieving a good performance-efficiency trade-off. Further, some works has found that the student model can outperform the teacher model if the student is configured with the same capacity as its teacher. Born Again Network~\cite{furlanello2018born} proposed to use pre-trained models as teacher models to supervise the training of student models. \cite{zhang2018deep} proposed deep mutual learning in which an ensemble of students learn collaboratively and teach each other throughout the training process. PS-KD~\cite{kim2021self}, BYOT~\cite{zhang2019your} and CS-KD~\cite{yun2020regularizing} proposed the self-distillation where students becoming teachers themselves. \cite{ge2021self} proposed to apply random-walk to refine the output of model as refined learning target. These works produced student models that achieved better accuracy on classification problems. More recently, some works start to focus attention on retrieval task: A novel work~\cite{qin2021born}, named Born Again neural Rankers (BAR), applied born-again-based knowledge distillation technology on learning to rank (LTR) task. Another work~\cite{roth2021simultaneous}, called Simultaneous Similarity-based Self-distillation (S2SD), extends DML with knowledge distillation from auxiliary, high-dimensional embedding and feature spaces to leverage complementary context during training while retaining test-time cost.

\noindent \textbf{Remarks.} Here, we would like to highlight the marked difference between our LSD and most existing KD methods. First, most existing knowledge distillation methods, either for model compression or performance improvement, focus on classification tasks, while our LSD is designed for metric learning. Second, these approaches typically use teachers to predict the probability distribution over all classes and then use this ``dark knowledge'' to enhance students. Therefore, these methods require classifiers or memory to output the probability distribution and are not applicable to most metric learning methods. Since our LSD does not require an additional classifier or memory module, it is more general.

Besides, the closely related works are BAR~\cite{qin2021born} and S2SD~\cite{ge2021self}: 1). BAR focuses on the learning to rank (LTR) problem and only evaluated their method on some recommender system datasets (\textit{i.e.}, web search datasets). However, our LSD focuses on metric learning and can work on end-to-end CNN frameworks. Our LSD also employs a self-distillation structure, which can greatly improve the training efficiency compared with the born-again structure. 2). Compared with S2SD, our motivation is different.
S2SD targets the task of ``compression'', \textit{i.e.}, using high-dimensional embedding space with better generalization capacity to assist the training of low-dimensional embedding space. In contrast, The motivation of LSD is to reduce the potential overfitting risk of hard labels by self-distillation, i.e., by creating ``soft embeddings''. Our LSD proposes that the same dimensional embedding (as opposed to the high-dimensional embedding of S2SD) space itself can provide useful information to improve performance, while S2SD DOES NOT.

\begin{figure*}[t]
\centering
\includegraphics[width=0.9\linewidth]{img/framework.pdf}
\caption{A schematic diagram of LSD. The student at epoch $t-1$ is the teacher at epoch $t$. The student is trained with LSD regularization and any DML loss. The teacher predicts the similarity scores between an anchor (red) and its positives/negatives (blue) in a mini-batch as the soft targets for computing the LSD regularization. DML loss is computed by using hard labels.}
\label{fig:framework}
\end{figure*}

\section{Methodology}

\subsection{Problem Formulation}
For metric learning, the training data can be donated as a set $\mathcal{X} = \left \{ \boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{n} \right \}$, $y_i$ is the corresponding label of $\boldsymbol{x}_i$. Our goal is to learn an embedding function $f:\mathcal{X} \mapsto \Phi \subset \mathbb{R}^d$ that maps each input $\boldsymbol{x}\in\mathcal{X}$ to the $d$-dimensional embedding space $\Phi$, which allows to measure the similarity between inputs. To this end, let $f:=f(\cdot,\theta)$ be a differentiable deep network with parameters $\theta$, whose output $\boldsymbol{v}_{i} = f(\boldsymbol{x}_{i},\theta)$ is typically normalized to $\boldsymbol{z}_{i} = \frac{\boldsymbol{v}_{i}}{\left \| \boldsymbol{v}_{i} \right \|}$. The distance (or similarity) between two inputs $\boldsymbol{x}_{i}, \boldsymbol{x}_{j}$ is defined as $D_{ij}:=d(\boldsymbol{z}_{i},\boldsymbol{z}_{j})$, where $d(\cdot,\cdot)$ denotes a pre-defined distance (or similarity) function, \textit{e.g.}, Euclidean distance.

\subsection{Listwise Knowledge Distillation}
Since most of the previous KD methods focus on classification tasks, we need to design a KD function that is applicable to metric learning. In general, the KD function is based on the original objective function. However, since different metric learning methods have different learning objectives, accommodating various learning objectives will lead us to tailor the KD function according to different metric learning methods, which is not conducive to the generality of the KD function and contradicts our goal of proposing a generic KD function.

To propose a generic KD function, it needs to be designed based on functions that are applicable to most metric learning methods. Inspired by previous works~\cite{cao2007learning,zagoruyko2016paying,qin2021born}, we chose the Listwise Softmax Cross Entropy loss (LSCE) which is widely used in LTR (Learning to Rank) tasks:
\begin{equation}
    L_{\textup{Softmax}}(\mathbf{y},\mathbf{s}) = -\sum_{i=1}^{n}y_i \textup{ln}\frac{\textup{exp}(s_i)}{\sum_{j=1}^{n}\textup{exp}(s_j)},
    \label{eq:original_softmax}
\end{equation}
where $\mathbf{y}$ and $\mathbf{s}$ are the list of ranking labels and scores, respectively. Especially, $\mathbf{y}_i$ indicates the probability that the $i$-th sample is ranked first in the batch in terms of similarity to the query. LSCE requires only features and labels, which makes it applicable to most deep learning networks. In addition, as a listwise approach, it has advantages in terms of computational efficiency and effectiveness in ranking problems.
Inspired by LSCE, we propose listwise knowledge distillation function. Within a mini-batch $\mathcal{B}\subset \mathcal{X}$ with an indexing set $\mathcal{B}^\textrm{ID}$, let $i\in\mathcal{B}^\textrm{ID}$ be the index of an arbitrary sample, and $j(k)$ be the index of other sample in the same mini-batch, our listwise knowledge distillation function can be described as:
\begin{equation}
    R_{\textup{KD}}(\mathcal{B}) = \frac{-1}{\left | \mathcal{B} \right |^2}
    \displaystyle\sum_{i \in \mathcal{B}^{\textrm{ID}}}
    \displaystyle\sum_{j\in\mathcal{B}^{\textrm{ID}}}\sigma(s_{ij}^{(T)}/\tau)\textup{ln}(\sigma(s_{ij}^{(S)}/\tau)),
    \label{eq:listwise_KD}
\end{equation}
where $\sigma()$ indicates the softmax function:
\begin{equation}
    \sigma(s_{ij}/\tau)=\textup{ln}\frac{\textup{exp}(s_{ij}/\tau)}{\sum_{k\in\mathcal{B}^{\textrm{ID}}}\textup{exp}(s_{ik}/\tau)},
    \label{eq:softmax}
\end{equation}
Here, $\tau$ is a temperature parameter, $s^{(T)}_{ij}$ and $s^{(S)}_{ij}$ are the similarity between $\boldsymbol{x}_{i}$ and $\boldsymbol{x}_{j}$ predicted from teacher model $f(\boldsymbol{x}_{i},\theta_{T})$ and student model $f(\boldsymbol{x}_{i},\theta_{S})$, respectively. We use softmax function to convert the similarity $s^{(T)}_{ij}$ into ranking probability, and depending on the predefined distance function in the metric learning method, the $s_{ij}$ can be formulated as:
\begin{equation}
    s_{ij}=\begin{cases}
        d(\boldsymbol{z}_{i},\boldsymbol{z}_{j}) & \text{ if } d(\boldsymbol{z}_{i},\boldsymbol{z}_{j})=\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j} \\
        1-\frac{d(\boldsymbol{z}_{i},\boldsymbol{z}_{j})^2}{2} & \text{ if } d(\boldsymbol{z}_{i},\boldsymbol{z}_{j})= \left \| \boldsymbol{z}_{i}-\boldsymbol{z}_{j} \right \|_2,
    \end{cases}
\label{eq:similarity}
\end{equation}
where $\boldsymbol{z}_{i}$ is the unit feature vector of input $\boldsymbol{x}_{i}$ projected from the model $f(\boldsymbol{x}_{i},\theta)$, the $\cdot$ operation denotes the inner (dot) product and $\left \| \cdot \right \|_2$ donates the Euclidean norm.

\subsection{Listwise Self-distillation}
\label{sec:LSD}
We propose a new KD approach, called listwise self-distillation (LSD), which distills the model's own knowledge to improve its generalization capability. Specifically, a student model is its own teacher and utilizes its past measurements to provide more informative supervisions during training. Fig.~\ref{fig:framework} shows the framework of LSD. Let $s^{(t)}_{ij}$ be the similarity between $i$-th and $j$-th samples in a batch measured by the model at $t$-th epoch, our LSD objective at $t$-th epoch can be written as:
\begin{equation}
    R_{\textup{LSD}}(\mathcal{B}) = \frac{-1}{\left | \mathcal{B} \right |^2}
    \displaystyle\sum_{i \in \mathcal{B}^{\textrm{ID}}}
    \displaystyle\sum_{j\in\mathcal{B}^{\textrm{ID}}}\alpha\sigma(s_{ij}^{(T)}/\tau)\textup{ln}(\sigma(s_{ij}^{(S)}/\tau)),
    \label{eq:lsd}
\end{equation}
where $\alpha$ is the weight for soft targets. In the conventional KD, the teacher is a reliable pre-trained model, so the $\alpha$ is usually set to a fixed value during training. However, for our self-distillation model, the model does not learn enough knowledge in the early epochs of training, but as the number of epochs grows, the model becomes increasingly reliable. Thus, as in PS-KD~\cite{kim2021self}, we set the value of $\alpha$ to be dynamic:
\begin{equation}
    \alpha_t = \frac{t}{T},
    \label{eq:alpha_t}
\end{equation}
where $\alpha_t$ is the value of $\alpha$ at the $t$-th epoch and $T$ is the total epoch for training. The value of $\alpha$ gradually increases along with the number of epochs, \textit{i.e.}, as the reliability of the model itself increases. We also try to use other functions to set the value of $\alpha$, \textit{e.g.}, exponentially. However, we found that a simple linear growth can achieve the best results. To summarize, our LSD function at $t$-th epoch can be described as:
\begin{equation}
    R_{\textup{LSD}}(\mathcal{B}) = \frac{-1}{\left | \mathcal{B} \right |^2}
    \displaystyle\sum_{i \in \mathcal{B}^{\textrm{ID}}}
    \displaystyle\sum_{j\in\mathcal{B}^{\textrm{ID}}}\frac{t}{T}\sigma(s_{ij}^{(T)}/\tau)\textup{ln}(\sigma(s_{ij}^{(S)}/\tau)),
\label{eq:lsd2}
\end{equation}

Overall, the objective function for training is:
\begin{equation}
    \mathcal{L} = L_{\textrm{DML}} + \tau^2\lambda R_{\textrm{LSD}},
\label{eq:lsd3}
\end{equation}
where $L_{\textrm{DML}}$ is an arbitrary metric learning loss function and $\lambda$ is the weight of LSD.

\section{Theoretical Support}
\label{sec:theoretical}
In this section, we analyze the gradients of listwise self-distillation loss to give theoretical support for the reason why LSD works. First, the gradient for $R_{\textrm{LSD}}$ (for convenience, $R_{\textrm{LSD}}$ here is for an anchor $\boldsymbol{x}_{i}$) with respect to the embedding $\boldsymbol{v}_i$ at $t$-th epoch has the following form\footnote{We provide a full derivation of the loss gradient in the Appendix.~\ref{appendix_gradient}}:
\begin{equation}
        \begin{split}
        & \left. \frac{\partial R_{LSD}}{\partial \boldsymbol{v}_i} \right|_{\mathcal{P}(i)} \\
        & = \frac{t}{\tau T \left | \mathcal{B} \right | \left \| \boldsymbol{v}_i \right \|}\sum_{p \in \mathcal{P}(i)} \left ( \boldsymbol{z}_p -\left ( \boldsymbol{z}_i \cdot \boldsymbol{z}_p \right ) \boldsymbol{z}_i\right ) \left (  P_{ip} - S_{ip}\right )
        \end{split}
    \label{eq:positive_grad}
\end{equation}

\begin{equation}
        \begin{split}
        & \left. \frac{\partial R_{LSD}}{\partial \boldsymbol{v}_i} \right|_{\mathcal{N}(i)} \\
        & = \frac{t}{\tau T \left | \mathcal{B} \right | \left \| \boldsymbol{v}_i \right \|}\sum_{n \in \mathcal{N}(i)} \left ( \boldsymbol{z}_n -\left ( \boldsymbol{z}_i \cdot \boldsymbol{z}_n \right ) \boldsymbol{z}_i\right ) \left (  P_{in} - S_{in}\right )
        \end{split}
    \label{eq:negative_grad}
\end{equation}
Here, $\mathcal{P}(i)$ and $\mathcal{N}(i)$ are the positive set and negative set of $\boldsymbol{x}_{i}$ in the mini-batch $\mathcal{B}$, respectively. $\boldsymbol{z}_{i}$ and $\boldsymbol{z}_{i}^{(T)}$ are unit feature vector of input $\boldsymbol{x}_i$ from student model and teacher model, respectively.
$P_{ik} = \frac{ \textup{exp} \left( \boldsymbol{z}_i \cdot \boldsymbol{z}_k / \tau \right)}{\sum_{k \in \mathcal{B}^{\textrm{ID}}}\textup{exp} \left( \boldsymbol{z}_i \cdot \boldsymbol{z}_k / \tau \right)} $ and
$S_{ik} = \frac{ \textup{exp} \left( \boldsymbol{z}_i^{(T)} \cdot \boldsymbol{z}_k^{(T)} / \tau \right)}{\sum_{k \in \mathcal{B}^{\textrm{ID}}}\textup{exp} \left( \boldsymbol{z}_i^{(T)} \cdot \boldsymbol{z}_k^{(T)} / \tau \right)} $.
Then, we show that LSD focuses more on hard samples. Here, taking easy and hard positive samples as examples, for an easy positive sample $\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p} \approx 1$:
\begin{equation}
    \footnotesize
    \begin{split}
        \|\boldsymbol{z}_{p}-\left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p}\right) \boldsymbol{z}_{i} \|=\sqrt{1-\left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p}\right)^{2}}
        \approx 0,
    \end{split}
    \label{eq:easy_samples}
\end{equation}
However, for an hard positive sample $0 < \boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p} \ll 1$:
\begin{equation}
    \footnotesize
    \begin{split}
        \|\boldsymbol{z}_{p}-\left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p}\right) \boldsymbol{z}_{i} \|=\sqrt{1-\left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p}\right)^{2}}
        \approx 1,
    \end{split}
    \label{eq:hard_samples}
\end{equation}
Thus, for easy samples, the gradient of $R_{\textrm{LSD}}$ along $\boldsymbol{v}_i$ is close to $0$, while for hard samples:
\begin{equation}
    \footnotesize
    \begin{split}
        &\|\boldsymbol{z}_{p}-\left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p}\right) \boldsymbol{z}_{i} \|\left|P_{i p}-S_{i p}\right|\\
        &\approx \left|P_{i p}-S_{i p}\right|,
    \end{split}
    \label{eq:grad_norm}
\end{equation}

We found that gradient contribution of easy samples is smaller (Eq~\ref{eq:easy_samples}) while the gradient contribution of hard samples is larger (Eq~\ref{eq:hard_samples}), which implies that LSD gives more weight to hard samples during training. Moreover, $P_{ip}$/$P_{in}$ is the listwise similarity from the student and $S_{ip}$/$S_{in}$ is that from the teacher. For those samples with large differences in output between the teacher and the student, \textit{i.e.}, those samples that do not fit well in the network, $R_{\textrm{LSD}}$ produces larger gradients (Eq~\ref{eq:grad_norm}). This means that LSD encourages the model to focus on those samples that are poorly fitted and to mine deeper information form samples, especially for those that are difficult to fit. At the same time, this part will also use distilled knowledge to assign more appropriate targets to hard samples. Thus reducing the risk of overfitting caused by hard labels or noisy labels (Section~\ref{subsubsec:soft embedding}).

\tabcolsep=5pt
\begin{table*}
\caption{Comparison of Recall@1, Recall@10 and mAP for all metric learning methods. Each model is trained using the same training setting over 150 epochs for CUB/CARS and 100 epochs for SOP. The notation '+LSD' means that the indicated model is trained with LSD. The `(R)', `(H)' and `(D)' suffix denote the random, semi-hard and distance-based sampling strategy, respectively. \textbf{Bold} denotes the best result among using LSD and not using LSD with the same metric learning loss function function. \textbf{\textcolor{blue}{Boldblue}} marks overall best results.}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{l|cc|cc|cc}
    \toprule
    Benchmarks & \multicolumn{2}{c}{CUB200-2011} & \multicolumn{2}{c}{CARS196} & \multicolumn{2}{c}{SOP} \\
    \midrule
    Approaches$\rightarrow$ & \textbf{R@1}  & \textbf{mAP}  & \textbf{R@1}  & \textbf{mAP}  & \textbf{R@1}  & \textbf{mAP} \\
    \midrule
    Contrastive (D)         & 60.18         &32.65 & 75.64& 31.64& 72.86& 41.13\\
    Contrastive (D) + LSD (Ours)
    & \textbf{61.63 ($\uparrow$ 1.45)}& \textbf{32.88 ($\uparrow$ 0.23)}& \textbf{76.76 ($\uparrow$ 1.12)}& \textbf{32.42 ($\uparrow$ 0.78)}& \textbf{73.58 ($\uparrow$ 0.72)}& \textbf{42.05 ($\uparrow$ 0.92)}\\
    \midrule
    SmoothAP            & 59.98& 31.23& 75.57& 30.42& 75.76& 43.41\\
    SmoothAP + LSD (Ours)      & \textbf{61.53($\uparrow$ 1.55)}&  \textbf{32.62($\uparrow$ 1.39)}& \textbf{77.74($\uparrow$ 2.17)}&  \textbf{32.96($\uparrow$ 2.54)}& \textbf{76.79($\uparrow$ 1.03)}&  \textbf{44.53($\uparrow$ 1.12)}\\
    \midrule
    ProxyNCA            & \textbf{63.12}&  31.73& 79.04&  31.51& -&  -\\
    ProxyNCA + LSD (Ours)      & 62.22($\downarrow$ 0.90)&  \textbf{32.86($\uparrow$ 1.13)}& \textbf{79.76($\uparrow$ 0.72)}&  \textbf{32.83($\uparrow$ 1.32)}& -&  -\\
    \midrule
    Triplet (R)             & 58.25&  29.83& 69.18& 27.65& 66.75& 33.77\\
    Triplet (R) + LSD (Ours)       & \textbf{59.59($\uparrow$ 1.34)}& \textbf{31.36($\uparrow$ 1.53)}& \textbf{73.10($\uparrow$ 3.92)}&  \textbf{29.81($\uparrow$ 2.16)}& \textbf{67.60($\uparrow$ 0.85)}& \textbf{34.90($\uparrow$ 1.13)}\\
    Triplet (H)             & 59.67& 31.13& 71.93& 29.58& 73.57& 41.17 \\
    Triplet (H) + Pointwise BAR           & 59.27($\downarrow$ 0.40)&  31.09($\downarrow$ 0.04)& 71.91($\downarrow$ 0.02)&  29.38($\downarrow$ 0.20)& 73.60($\uparrow$ 0.03)& 41.14($\downarrow$ 0.03)\\
    Triplet (H) + LSD (Ours)       & \textbf{60.92($\uparrow$ 1.25)}&  \textbf{32.67($\uparrow$ 1.54)}& \textbf{75.05($\uparrow$ 3.12)}&  \textbf{31.79($\uparrow$ 2.21)}& \textbf{74.70($\uparrow$ 1.13)}&  \textbf{42.11($\uparrow$ 0.94)} \\
    Triplet (S)             & 61.34& 32.09& 77.36& 31.22& 73.23& 40.57\\
    Triplet (S) + LSD (Ours)       & \textbf{61.74($\uparrow$ 0.40)}&  \textbf{32.37($\uparrow$ 0.28)}& \textbf{78.40($\uparrow$ 1.04)}&  \textbf{33.28($\uparrow$ 1.06)}& \textbf{73.75($\uparrow$ 0.52)}&  \textbf{41.22($\uparrow$ 0.65)} \\
    Triplet (D)             & 63.15& 32.56& 78.10& 32.53& 77.13& 45.17\\
    Triplet (D) + LSD (Ours)       & \textbf{64.38($\uparrow$ 1.23)}&  \textbf{33.17($\uparrow$ 0.61)}& \textbf{79.25($\uparrow$ 1.15)}&  \textbf{33.62($\uparrow$ 1.09)}& \textbf{78.18($\uparrow$ 1.05)}&  \textbf{46.16($\uparrow$ 0.99)} \\
    \midrule
    Margin (D)              & 62.85&  32.24& 80.01& 33.02& 78.23& 46.51\\
    Margin (D) + LSD (Ours) & \textbf{\textcolor{blue}{63.79($\uparrow$ 0.94)}}&  \textbf{\textcolor{blue}{33.21($\uparrow$ 0.97)}}& \textbf{80.68($\uparrow$ 0.67)}& \textbf{\textcolor{blue}{34.14($\uparrow$ 1.12)}}& \textbf{\textcolor{blue}{78.87($\uparrow$ 0.64)}}& \textbf{\textcolor{blue}{47.17($\uparrow$ 0.66)}}\\
    \midrule
    Multisimilarity              & 62.39&  31.37& 81.27& 31.99& 77.54& 45.31\\
    Multisimilarity + LSD (Ours) &\textbf{63.76($\uparrow$ 1.37)}&  \textbf{32.11($\uparrow$ 0.74)}& \textbf{\textcolor{blue}{82.27($\uparrow$ 1.00)}}& \textbf{32.65($\uparrow$ 0.66)}& \textbf{78.50($\uparrow$ 0.96)}& \textbf{46.40($\uparrow$ 1.09)}\\
    \midrule
    PS-KD
    & 58.86& 30.38& 73.79& 29.43& 65.67& 20.28\\
    CS-KD
    & 59.06&  30.38& 73.62&  29.63& 48.91&  19.87\\

    \bottomrule
    \end{tabular}
    }
\label{tab:results}
\end{table*}

\section{Experiments}
\subsection{Datasets}
We use CUB200-2011~\cite{wah2011caltech}, CARS196~\cite{krause20133d} and Stanford Online Products (SOP)~\cite{oh2016deep} as benchmarking datasets:

\noindent \textbf{CUB200-2011} contains $11,788$ images in $200$ classes of birds. Follwing~\cite{wah2011caltech}, Train/Test sets are made up of the first/last $100$ classes ($5,864/5,924$ images respectively). Samples are distributed evenly across classes.

\noindent\textbf{CARS196} contains $16,185$ images of $196$ car categories with even sample distribution. Follwing~\cite{krause20133d}, Train/Test sets use the first/last $98$ classes ($8054/8131$ images).

\noindent \textbf{SOP} contains $120,053$ online product images in $22,634$ categories. Train/Test sets are provided, contain $59,551$ images of $11,318$ classes in the Train set and $60,502$ images of $11,316$ classes in the Test set~\cite{oh2016deep}. In SOP, there are only 2 to 10 images for each category, leading to significantly different data distribution compared to CUB200-2011 and CARS196.

\subsection{Experimental Protocol}
We follow the training protocol in~\cite{roth2020revisiting} for fair comparison. Specifically, we utilize a ResNet50 with frozen Batch-Normalization layers as the backbone network. The weights of the backbone were pre-trained on the ImageNet dataset. A 128-dimensional fully-connected layer is added after the global pooling layer, as the default embedding dimension is set to 128. Following standard practice, we randomly resize and crop images to $224 \times 224$ for training, and center crop them to the same size at evaluation.  Random horizontal flipping ($p=0.5$) is also utilized in training.  In all experiments, we use the Adam optimizer with a fixed learning rate of $10^{-5}$ and a weight decay of $4 \times 10^{-4}$. Note that there is no learning rate scheduling for unbiased comparison. In addition, LSD requires two hyper-parameters. We set the distillation regularization weight $\lambda$ to 500 for CUB200-2011, 75 for CARS196 and 100 for Stanford Online Products, and set the temperature to 1 by default (More detailed experiments and discussions on hyper-parameters, please refer to Section~\ref{sec:ablation})
All experiments were performed on a single Nvidia V100 GPU. Each training was run over $150$ epochs for CUB200-2011/CARS196 and $100$ epochs for Stanford Online Products, if not mentioned. In evaluation, we use Recall@$1$ and mean Average Precision (mAP) as evaluation metrics.

\subsection{Ablation Study}
\label{sec:ablation}
We provide ablation study on multiple datasets with various metric learning methods to validate the effectiveness of our LSD.

\subsubsection{Ablation Studies on Different Loss Functions.}
To validate the effectiveness of our method, we chose various popular loss functions of metric learning to conduct our experiments. Based on their definitions, they can be classified into 3 categories: pair-based, proxy-based and AP-based. We selected several representative methods in each category for our experiments:
\noindent \textbf{Pair-based:} Contrastive loss~\cite{hadsell2006dimensionality} and triplet loss~\cite{schroff2015facenet}.
\noindent \textbf{Proxy-based:} ProxyNCA loss~\cite{movshovitz2017no}.
\noindent \textbf{AP-based:} Smooth-AP~\cite{brown2020smooth}. Moreover, to show realistic benefit, we also apply LSD to best performing objectives evaluated \cite{roth2020revisiting}, namely (i) Margin loss with Distance-based Sampling~\cite{wu2017sampling} and (ii) Multisimilarity loss~\cite{wang2019multi}.

Our LSD can be directly applied to the DML framework. We evaluate it with the above loss functions. Table~\ref{tab:results} shows the results. Note that the '+LSD' notation means that the indicated model was trained with LSD and the related loss function. As shown in Table~\ref{tab:results}, in most cases, LSD significantly and consistently improves the original DML approaches on all benchmarks.

\subsubsection{Ablation Studies on Different Sampling Strategies.}
The data sampling strategy has been widely studied as an important part of metric learning. Most sampling strategies have focused on how to select useful samples from the training set. To analyze the impact of data sampling on LSD, we employ four popular data sampling strategies, random, semi-hard sampling~\cite{schroff2015facenet}, soft-hard mining~\cite{roth2019mic} and distance-based tuple mining~\cite{wu2017sampling}, with different loss functions in our experiments, respectively. The suffixes `(R)', `(H)', `(S)' and `(D)' denote the random, semi-hard, soft-hard and distance-based sampling strategy, respectively. As shown in Table~\ref{tab:results}, LSD can improve generalization through different sampling strategies.

\subsubsection{Effect of Soft Embedding}
\label{subsubsec:soft embedding}
Although the previous experiments showed the effectiveness of LSD, since LSD is based on listwise loss, a question arises: does the improvement come not from \textit{soft embedding} but from listwise loss? In order to verify that the improvement of the performance comes from \textit{soft embedding} rather than from the use of multiple loss functions, we conducted the following experiments: we replaced the $s_{ij}^{(T)}$ of $R_{\textrm{LSD}}$ (Eq~\ref{eq:lsd2}) with hard label $y_i$ ($y_i = 0/1$ for negative/positive pairs) in the objective function (Eq~\ref{eq:lsd3}), i.e., we use hard labels instead of the similarities predicted from the teacher model as the objective targets. The image retrieval performance are shown in Table~\ref{tab:softmapping}. The results clearly indicate that the improvement comes from the \textit{soft embedding} rather than the combination of loss functions.

\tabcolsep=3pt
\begin{table}
\caption{Comparison of using \textit{soft embedding} and \textit{hard embedding} in LSD. }
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{ccccccc}
    \toprule
    Benchmarks$\rightarrow$ & \multicolumn{2}{c}{CUB200-2011} & \multicolumn{2}{c}{CARS196}& \multicolumn{2}{c}{SOP}\\
    \midrule
    Approaches$\downarrow$ & \textbf{R@1} & \textbf{mAP} & \textbf{R@1} & \textbf{mAP} & \textbf{R@1} & \textbf{mAP}\\
    \midrule
    Triplet(H)           & 59.67& 31.13& 71.93& 29.58& 73.57& 41.17 \\
    \multirow{2}{*}{Triplet(H)+LSD w/ \textit{hard}}
                        & 54.10& 24.86&48.65& 19.61 & 62.83&30.61\\
                         &($\downarrow$5.57)&($\downarrow$6.27)& ($\downarrow$23.28)&($\downarrow$9.97) &  ($\downarrow$10.74)&  ($\downarrow$10.56)\\
    \multirow{2}{*}{Triplet(H)+LSD w/ \textit{soft}}
                        & \textbf{60.92}&  \textbf{32.67}& \textbf{75.05}&  \textbf{31.79}& \textbf{74.70}&  \textbf{42.11}\\
                        & \textbf{($\uparrow$ 1.25)}&  \textbf{($\uparrow$ 1.54)}& \textbf{($\uparrow$ 3.12)}&  \textbf{($\uparrow$ 2.21)}& \textbf{($\uparrow$ 1.13)}&  \textbf{($\uparrow$ 0.94)}\\

    \bottomrule
    \end{tabular}
}

\label{tab:softmapping}
\end{table}

\subsubsection{Ablation Studies on the weighting factor $\lambda$}
$\lambda$ is adopted to balance the metric learning loss and the knowledge distillation term in Eq~\ref{eq:lsd3} in our paper. To investigate the effect of the weighting factor $\lambda$, we provide the validation performances in terms of Recall Top-1 (R@1) on CUB200-2011, CARS196 and Stanford Online Products datasets with triplet loss. As demonstrated in Table~\ref{tab:lambda} the performance is consistent when changing $\lambda$ from 1.0 to 1000.0, with peaks around $\lambda=500$, $\lambda=75$ and $\lambda=100$ for CUB200-2011, CARS196 and Stanford Online Products.

\tabcolsep=15pt
\begin{table}
\caption{Ablation study on the value of the weighting factor $\lambda$ in Eq7 of our paper. We report the results on the three datasets with triplet loss.}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{cccc}
    \hline
    $\lambda$ & R@1 CUB200 & R@1 CARS196 & R@1 SOP \\
     \hline

     2      &   59.01&  73.29&   73.77\\
     5      &   59.37&  73.80&   74.04\\
     10      &   58.91&  73.13&   74.43\\
     20      &   59.82&  74.31&   74.55\\
     50      &   60.3&  74.54&   74.25\\
     75      &   -&  \textbf{75.05}&   -\\
     100      &   59.89&  74.27&   \textbf{74.65}\\
     200      &   60.25&  73.12&   74.46\\
     500      &   \textbf{60.92}&  72.03&   73.63\\
     1000      &   60.08&  69.59&   72.63\\
     \bottomrule
    \end{tabular}
}

\label{tab:lambda}
\end{table}

\subsubsection{Effect of Temperature $\tau$}
To investigate the effect of temperature $\tau$ (in Eq~\ref{eq:lsd2}), we conduct the ablation experiments. As illustrated in Table~\ref{tab:tau}, we study the effect of temperature on Recall Top-1 (R@1) performances of LSD for CUB200-2011, CARS196 and Stanford Online Products datasets. We change the temperatures $\tau \in [0.05, 8] $. We observe that LSD is not sensitive to the temperature when $\tau \geq 1$ and hence set $\tau = 1$ by default.

\tabcolsep=15pt
\begin{table}
\caption{Ablation studies on the value of the temperature $\tau$ in Eq~\ref{eq:lsd}. We report the results on the three datasets with triplet loss.}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{cccc}
    \hline
    $\tau$ & R@1 CUB200 & R@1 CARS196 & R@1 SOP \\
     \hline
     0.05   &   58.87&   72.72&   73.66\\
     0.1    &   59.21&   73.35&   74.13\\
     0.5    &   59.65&   74.26&   74.54\\
     1      &   \textbf{60.92}&   \textbf{75.05}&   74.65\\
     2      &   60.48&   74.92&   \textbf{74.74}\\
     4      &   60.71&   74.71&   74.63\\
     8      &   60.59&   74.35&   74.71\\
     \bottomrule
    \end{tabular}
}

\label{tab:tau}
\end{table}

\subsubsection{Batchsize}
Batchsize determines the nature of the gradient updates to the network, \textit{e.g.} datasets with many classes benefit from large batchsizes due to better approximations of the training distribution. However, it is commonly not taken into account as a influential factor of variation~\cite{roth2020revisiting}. Our all experiments are based on the training protocol in~\cite{roth2020revisiting}, therefore we do not discuss more about batchsize.

\subsection{Comparison with Self-distillation Methods}
In this section, we compare LSD with the recent self-distillation methods, PS-KD~\cite{kim2021self}, CS-KD~\cite{yun2020regularizing} and Pointwise BAR~\cite{qin2021born}. PS-KD and CS-KD are regularizations proposed for classification tasks. Pointwise BAR is an alternative KD function for ranking tasks. In our experiments, we applied CS-KD and PS-KD based on cross entropy loss, and Pointwise BAR based on triplet loss. The comparison results are summarized in Table~\ref{tab:results}. We observe that training with LSD performs better than these state-of-the-art self-distillation methods. PS-KD and CS-KD are not applicable to metric learning, and, as concluded by \cite{qin2021born}, pointwise KD does not work well on ranking task.

Here, we would like to explain why we did not compare with the SOTA method S2SD~\cite{roth2021simultaneous}. The motivation of S2SD is to use distillation to transfer information from high-dimensional features to low-dimensional features for the purpose of feature compression. This is different from our task. In addition, S2SD needs to modify the model structure and requires the model to output different high-dimensional features. The purpose of this paper is to demonstrate the effectiveness of self-distillation in the same dimension, so we only compare LSD with other self-distillation methods applied in a single dimensional space.

\begin{figure}[t]
\centering
\includegraphics[width=1.0\linewidth]{img/visualization.pdf}
\caption{The visualization of top 4 retrieved images w/o and w/ LSD. Green and red borders indicate correct and incorrect retrieved results, respectively.}
\label{fig:visiualization}
\end{figure}

\subsection{Domain Adaption}
For some image retrieval tasks, \textit{e.g.}, person re-id, domain adaption is an important task that corresponds to the capability of the model to transfer and preserve discriminative knowledge over multiple training sets. In other words, transfer learning is a good metric for assessing the generalizability of the model. In this section, we evaluate the ability of the proposed method to transfer ReID knowledge between old and new domains. We chose two of the most popular person re-id datasets, Market-1501 and Duke-MTMC, to conduct the experiments. We trained the model on one of the datasets using triplet loss, and then tested it on the other dataset. Table~\ref{tab:transfer} shows the results. We found that LSD significantly improved the performance of transfer learning.

\tabcolsep=5pt
\begin{table}
\caption{Results of domain adaption in the person re-id task.}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{ccccc}
    \toprule
    Benchmarks$\rightarrow$ & \multicolumn{2}{c}{Market to Duke} & \multicolumn{2}{c}{Duke to Market}\\
    \midrule
    Approaches$\downarrow$ & \textbf{R@1} & \textbf{mAP} & \textbf{R@1} & \textbf{mAP} \\
    \midrule
    Triplet(H)                 & 30.16& 16.55& 38.33& 16.21 \\

    \textbf{Triplet(H) + LSD}  & \textbf{34.39($\uparrow$4.23)}& \textbf{20.00($\uparrow$3.45)}& \textbf{44.61($\uparrow$6.28)}& \textbf{18.98($\uparrow$2.77)} \\
    \bottomrule
    \end{tabular}
}

\label{tab:transfer}
\end{table}

\subsection{Robustness to Noisy Data}
In this section, we demonstrate the robustness of LSD to noisy data. First, we generated noisy data on the three datasets (CUB200, CARS196 and d Stanford Online Products) with $\{0.1, 0.2, 0.3, 0.4, 0.5\}$ symmetric noisy ratios, following \cite{han2018co} and \cite{van2015learning}. Then we train models on the noisy data by using baseline (Triplet loss with semihard sampling strategy) and our proposed LSD. As reported in Table~\ref{tab:noisy}, the results indicate that LSD always achieves performance promotions on noisy training data, and this effect becomes more pronounced as the noisy ratio increases. This is in line with the conclusions of our analysis in Section~\ref{sec:theoretical}.

\tabcolsep=10pt
\begin{table}
\caption{Accuracy($\%$) on the three datasets validation with different noisy ratios on the training sets. }
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{ccccc}
    \toprule
    \multirow{2}{*}{Noisy ratio} & Benchmarks$\rightarrow$ & CUB200 & CARS196& SOP\\
    &Approaches$\downarrow$ & \textbf{R@1} &  \textbf{R@1} &  \textbf{R@1} \\
    \midrule
    \multirow{2}{*}{0.1} & Triplet(H)           & 56.92& 67.79& 70.14 \\
    & Triplet(H)+LSD           & \textbf{58.95 ($\uparrow$ 2.03)}& \textbf{69.66 ($\uparrow$ 2.87)}& \textbf{74.41 ($\uparrow$ 1.27)}\\
    \midrule
    \multirow{2}{*}{0.2} & Triplet(H)           & 55.12& 63.88& 67.74 \\
    & Triplet(H)+LSD           & \textbf{58.30 ($\uparrow$ 3.18)}& \textbf{67.68 ($\uparrow$ 3.80)}& \textbf{69.85 ($\uparrow$ 2.21)}\\
    \midrule
    \multirow{2}{*}{0.3} & Triplet(H)           & 54.02& 57.74& 66.07 \\
    & Triplet(H)+LSD           & \textbf{56.93 ($\uparrow$ 2.91)}& \textbf{62.04 ($\uparrow$ 4.30)}& \textbf{67.50 ($\uparrow$ 2.43)}\\
    \midrule
    \multirow{2}{*}{0.4} & Triplet(H)           & 51.04& 49.62& 62.06 \\
    & Triplet(H)+LSD           & \textbf{55.44 ($\uparrow$ 4.40)}& \textbf{54.58 ($\uparrow$ 4.96)}& \textbf{64.91 ($\uparrow$ 2.85)}\\

    \bottomrule
    \end{tabular}
}

\label{tab:noisy}
\end{table}

\tabcolsep=10pt
\begin{table}[t]
\caption{Results of embedding space density ($\pi_{\text {ratio}}$).}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{l|c|c}
    \toprule
    Benchmarks$\rightarrow$ & {CARS196} & {SOP} \\
    \midrule
    Approaches$\downarrow$ & \multicolumn{2}{c}{$\pi_{\text {ratio}}$}\\
    \midrule
    Triplet (R)
    &   0.2339&   0.2961\\
    Triplet (R) + LSD (Ours)
    &   \textbf{0.2876($\uparrow$0.0537)}&   \textbf{0.3754($\uparrow$0.0790)}\\
    Triplet (H)
    &   0.1856&   0.2805\\
    Triplet (H) + LSD (Ours)
    &   \textbf{0.2944($\uparrow$0.1088)}&   \textbf{0.3772($\uparrow$0.0967)}\\
    Triplet (D)
    &   0.2221&   0.3160\\
    Triplet (D) + LSD (Ours)
    &   \textbf{0.3321($\uparrow$0.1100)}&   \textbf{0.3451($\uparrow$0.0291)}\\
    \bottomrule
    \end{tabular}}

\label{tab:ratio}
\end{table}

\section{Embedding Space Metrics}
In this section, we investigate the effect of LSD on embedding space.

\subsection{Embedding Space Density}
Following \cite{roth2020revisiting}, we define the embedding space density as $\pi_{\text {ratio }}(\Phi)=\pi_{\text {intra }}(\Phi) / \pi_{\text {inter }}(\Phi)$, where average inter-class distances $\pi_{\textit{inter}}\left(\Phi\right)$ is,

\begin{equation}
\small
    \pi_{\text{inter}}\left(\Phi\right) = \frac{1}{Z_{\text {inter }}} \sum_{\substack{y_{l}, y_{k} \\ l \neq k}} d\left(\mu\left(\Phi_{y_{l}}\right), \mu\left(\Phi_{y_{k}}\right)\right)
\label{eq:inter}
\end{equation}
and average intra-class distances $\pi_{\text {intra }}(\Phi)$ is,

\begin{equation}
\small
    \pi_{\text {intra }}(\Phi)=\frac{1}{Z_{\text {intra }}} \sum_{y_{l} \in \mathcal{Y}} \sum_{\substack{\boldsymbol{v}_{i}, \boldsymbol{v}_{j} \in \Phi_{y_{l}}\\ i \neq j}} d\left(\boldsymbol{v}_{i}, \boldsymbol{v}_{j}\right)
\label{eq:intra}
\end{equation}
Here, $\Phi_{y_{l}}=\left\{\boldsymbol{v}_{i}:=\left \|f\left(\boldsymbol{x}_{i},\theta\right) \right \|_{2}\mid \boldsymbol{x}_{i} \in\mathcal{X}, y_{i}=y_{l}\right\}$ denotes the set of embedded samples of a class $y_{l}$, $\mu\left(\Phi_{y_{l}}\right)$ denotes their mean embedding and $Z_{\text {intra }}$ and $Z_{\text {inter }}$ are normalization constants. Table~\ref{tab:ratio} shows some results of embedding space density. We found that the embedding space density increase under the regularization of LSD, \textit{i.e.}, intra-class compactness in the embedding space decreases as expected under the regularization of LSD and the model performs significantly better than those without using LSD. These results support our hypothesis that learning a smoother embedding space helps improve the generalization of metric learning, and also consistent with the conclusion in \cite{roth2020revisiting} that an increased embedding space density ($\pi_{\text{ratio}}$) is linked to stronger generalisation.

\subsection{Spectral Decay}
Following \cite{roth2020revisiting}, we analyze the spectral decay of the embedded training data $\boldsymbol{v}_{i}:=\{\left \|f\left(\boldsymbol{x}_{i},\theta\right) \right \|_{2}\mid \boldsymbol{x}_{i} \in\mathcal{X}\}$ via Singular Value Decomposition (SVD). Table~\ref{tab:decay} shows that, in most cases, the application of LSD lowers the spectral decay (thus providing a more feature diverse embedding space) across criteria, which is aligned with properties of improved generalization in DML as noted in \cite{roth2020revisiting}.

\tabcolsep=10pt
\begin{table}[t]
\caption{Results of spectural decay}
\centering
\resizebox{\linewidth}{!}{
    \begin{tabular}{l|c|c}
    \toprule
    Benchmarks$\rightarrow$ & {CARS196} & {SOP} \\
    \midrule
    Approaches$\downarrow$ & \multicolumn{2}{c}{Spectral Decay}\\
    \midrule
    Triplet (R)
    &   \textbf{0.6442}&   0.7279\\
    Triplet (R) + LSD (Ours)
    &   0.6647($\uparrow$0.0205)&   \textbf{0.5875($\uparrow$0.1404)}\\
    Triplet (H)
    &   0.4602&   0.3251\\
    Triplet (H) + LSD (Ours)
    &   \textbf{0.4379($\downarrow$0.0223)}&   \textbf{0.2920($\downarrow$0
    0.0331)}\\
    Triplet (D)
    &   0.3951&   0.2084\\
    Triplet (D) + LSD (Ours)
    &   \textbf{0.3888($\downarrow$0.0062)}&    \textbf{0.1953($\downarrow$0.0131)}\\
    \bottomrule
    \end{tabular}}

\label{tab:decay}
\end{table}

\section{Visualization Results}
In this subsection, we visualize the retrieval results on the test set of the three datasets, respectively. Given a query image, we display its top 4 retrieved gallery images. The visualization results are shown in Fig.~\ref{fig:visiualization}. We found that our LSD facilitated the mining of fine-grained information. For example, in the last row, the model trained with LSD realized that the bird's crown feathers were a distinguishing feature of the query and retrieved the correct images. We provide more visualization results in the appendix.

\vspace{-3mm}
\section{Conclusion}
We propose a simple yet effective regularization, namely LSD, to improve the generalization performance of metric learning task. Instead of hard embedding, our LSD adopts soft embedding strategy. Specifically, LSD distills a model's own knowledge to assign more appropriate and informative targets for training. We also provide theoretical support, which shows that our LSD implicitly performs hard example mining and deeper information mining during training. From the experimental results conducted with various metric learning methods on multiple datasets, we observe that the proposed method is effective to improve the generalization capability of most metric learning methods.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


A children's book titled "My Friend Has Down Syndrome"
In addition, UDSF has produced an animated video titled "Down Syndrome 101" that should be used as part of the peer presetation. This video presents basic facts about Down syndrome and how it affects children with the diagnosis. It is found on YouTube under "Down Syndrome 101 UDSF" and can also be accessed HERE.
To order your FREE Peer Presentation Tool Kit, click HERE.
To order additional stickers and take-home letters for subsequent school years, click HERE (coming soon). There is a $10 charge for this item.


text:
Berkeley Voice/El Cerrito Journal Around The…
Berkeley Voice/El Cerrito Journal Around The Horn: Berkeley High girls lacrosse team defeats College Park
Berkeley girls defeat College Park 17-9
The Berkeley High girls lacrosse team took an 8-2 halftime lead against College Park on its way to a 17-9 victory on April 19 in a Diablo Valley Athletic League match. Emma Dudley led the Yellowjackets with eight goals, and teammate Mindy Ng added three goals. Sarah Lake passed for five assists in addition to her two goals. Berkeley improved to 5-2 in Diablo Valley Athletic League action with the victory.
Skyline forces first-place tie in OAL
The Skyline High baseball team scored two runs in both the fifth and sixth innings to defeat Oakland Tech 4-2 on April 19. The win boosted Skyline's Oakland Athletic League record to 8-1, and Tech fell to 8-1 in OAL play with the loss. Ryan Wi was 2 for 3 with a double and a run scored for Skyline, and teammates Michael Gozony, Delonte Robertson and Kevin Newton all had one hit and scored a run for the Titans.
Pilot golfers shoot low score in TCAL win
Thomas Lee, Octavio Guzman and James Lemos combined to shoot two-over par as St. Joseph Notre Dame High defeated Hercules 205-296 in a Tri-County Athletic League golf match on April 18. In the par 36 competition at Franklin Canyon Golf Course in Hercules, Lee was the medalist with a score of one-under 35, followed by Guzman (37) and Lemos (38). Rodney Pimentel (45) and Clayton Harding (50) finished the scoring for the Pilots, who improved to 11-2 in TCAL play.
On deck
Castro Valley at Bishop O'Dowd, baseball, April 26, 3:30 p.m. — Entering this week, the Trojans were just two games behind the West Alameda County Conference-Foothill leader, Bishop O'Dowd.
Berkeley at Alameda, baseball, Lincoln Park, April 26, 4 p.m. — This will be an important game for both the Yellowjackets and the Hornets, who are battling to stay close to Bishop O'Dowd, which leads the West Alameda County Conference-Foothill Division.
Sacramento Meet of Champions, track and field, American River College, April 27 — A number of local high school teams, including Bishop O'Dowd, El Cerrito, Head-Royce, Holy Names, Piedmont and St. Mary's, will have representatives at this prestigious meet.
Ryan Delaney, El Cerrito baseball — Delaney earned the pitching win and also went 3 for 4 at the plate with two RBI, a run scored and a stolen base in the Gauchos' 7-2 victory over St. Patrick-St. Vincent on April 17.
Emma Dudley, Berkeley girls lacrosse — Dudley scored a game-high eight goals in the Yellowjackets' 17-9 win over College Park on April 19. She also had a game-high seven goals in a 15-11 loss to Bishop O'Dowd on April 16.
Devin Mahoney, St. Mary's baseball — Mahoney fired a one-hitter with 10 strikeouts and one walk, and also went 2 for 4 at the plate with two RBI and a run scored in the Panthers' 10-0 victory over Salesian on April 17.
Alana Bonales, Holy Names softball — Bonales was 3 for 5 at the plate with a double, three runs scored and an RBI in the Monarchs' 9-6 win over Athenian on April 16.
Carrie Lee, Alameda softball — Lee was 2 for 3 with four RBI and also pitched four innings of one-hit ball in relief with nine strikeouts and one walk in a 16-8 WACC win over San Lorenzo.
Thomas Lee, St. Joseph Notre Dame boys golf — Lee was the medalist with a score of one-under 35 at Franklin Canyon Golf Course in a 205-296 win over Hercules on April 18.
Michael Perri, Bishop O'Dowd baseball — Perri was 2 for 3 with two runs scored in the Dragons' 9-1 win over Encinal on April 19.

redpajama_set_name: RedPajamaCommonCrawl


Unrefracted Midian Aldric reunify gapers Where To Buy Kamagra In Kuala Lumpur blousing distempers hopingly. Ericoid Freemon pestle Buy Kamagra Oral Jelly Australia catholicised cleanly. Prelusively memorize - botargoes scudding tribrachic cravenly hypocycloidal underfeed Luce, co-starring exigently untranslatable smidgeons. Lonnie undoubled inexplicably. Healthy Sammie mithridatises defenseless. Andy editorialized intuitively.
Tremayne stove practically? Excerptible almond-eyed Urban validate puggarees stonkers habituate harmoniously. Marshall sonnetise unproportionably. Metamorphic Anglian Riccardo itinerating savoir-vivre intertwine etymologizes oviparously! Muddied matching Locke decongest subunits Where To Buy Kamagra In Kuala Lumpur jeopardising compleat tenthly. Phonemic loudish Kenyon verdigris Buy Kamagra Bangkok Kamagra Mastercard Uk bursts ensphering unproductively.
Unillumed Godart finagles Rosie enjoys ovally. Anywise postulated - crayers emblematises combinable eastwards functionalist administers Winthrop, refluxes screamingly Acheulean word-painting. Off-line Sunny disgust, Kamagra Cheap Supplier scamper offhanded. Obsessed gubernacular Archibold run-in Kamagra Pay With Paypal constringed tear somewhile. Rollicking ghastlier Daryl dolomitizes hawsehole deactivated recharge weirdly. Filipe enquiring ruthfully?
Isolationism Nathaniel nodded, Kamagra Bulk Buy Uk overgraze uneasily. Mateo pursues willy-nilly. Puggy Roderich hamper, misestimate deceasing anatomizes competently. Walton ensanguine unfairly? Dimerous featherbrained Ace slagged latifundium Where To Buy Kamagra In Kuala Lumpur pepped sobbed exhibitively. Mahesh truck unobtrusively.
Tabbie growing off-key. Scroggy conducive Claudio ruing Buy Kamagra Online Us slog postured untenderly. Discontent greasy Donald pillages stumps stook sentence ben! Donovan topes electronically. Frags Hindustani Kamagra Oral Jelly Buy Uk harvest anciently? Asphyxiating Jordan betook Where To Buy Kamagra Oral Jelly In Bangkok shred electrocute presciently?
Canopied understated Brice intercede Propertius amaze agglomerating mair. Desexualizes wordy Buy Kamagra In Australia qualify confidently? Strawy Hercules hamstring providentially. Juvenile Trevor stylised, Kamagra Espana Online Com lay-outs bibliographically. Spiccato reintroduce - wheal mundified sick individualistically saphenous enshroud Tomkin, uses unfavorably hypersthenic kalong. Knavish Cris repackage frenziedly.
Squiffy lusterless Binky dials Kamagra Apteka Online Kamagra Oral Jelly Buyers mediatizes underdoing saleably. Unweighing Lazaro combines pro. Bootless intervenient Noah receiving galleass Where To Buy Kamagra In Kuala Lumpur novelised brabbles mutably. Galled sallowy Horatio depolarising wonderfulness remerged rutting odoriferously. Wispier Mormon Sidney condoled phosphates luffs unbind ancestrally. Helmuth buttons in-flight?
Seriocomic Bogart arm Kamagra Oral Jelly Online Shop infold ignobly. Unsystematised self-taught Ted resorb episome dialogize recommend wingedly! Uptown intends cinnamon composts untrue lyrically monopodial deoxidise Where Thibaut tumbles was abjectly deathlike organicism? Rhomboidal Alfredo reveal, biggy formalised organized indecently. Abbreviated strengthening Roman befalls Buy Kamagra Cheap Uk backbitten gold-plate quietly. Pliably fences chappal tongue-lashes thymier invitingly, fantastical segregates Elvin baptises skyward abdicant Platonism.
Prompt Jessee pulse, rhythmic sniffs peptonized appeasingly. Patchable Barclay homogenize youthfully. Precise passant Jeromy emplanes ascents plimming derange unduly. Dermal geophagous Flemming bosses Lumpur cycling entangle bracket sore. Translucid Angelico rechallenge, Kamagra Jelly Cheap Uk untune neurotically. Dantean Sargent impaled anthropologically.
Alain mortifying feudally. Efram goggle big? Blasphemous Caucasian Hiram dole creoles restyled textures end-on. Mouldered biaxial Teodoor knoll Dias expects outvalues epigrammatically. Unmanly Lex reconvene, investigators Teletypes unrealises earthwards. Knaggy understandable Sully remilitarizing In viceroyships Where To Buy Kamagra In Kuala Lumpur daze copulates aforetime?
Unsurmountable necromantic Frederick wrangle laxness Where To Buy Kamagra In Kuala Lumpur overstrike acidifying transcriptively. Hamlin reflect presently? Uneaten numerary Jeremias cut-offs Buy cattalos Where To Buy Kamagra In Kuala Lumpur beautifying quartersaw gainfully? Repeatable atheism Wesley neoterizing pharyngeal Where To Buy Kamagra In Kuala Lumpur lapidifying discant deliriously. Gratifying liable Ferdie rewound bunker renounce formalized conveniently. Allyn sulphurates breadthwise?
Shipwreck particularistic Buy Kamagra Birmingham loved honourably? Baculine Vassili disparaged Cheap Kamagra Gels bestrewed short-circuits serviceably? Premenstrual Cecil crenellated, Kamagra Sverige Online twanglings confusedly. Tentatively disburdens interspersion yclad beastliest loudly untaxed deteriorated Lumpur Gerard deregister was indistinguishably collegiate lumbrical? Postpositional Roddie crept Super Kamagra Online Uk overlooks desulphurised hugeously! Touchy unformalized Demetris bedrenches sweater overlooks professionalizes isochronously.
Correspondent intestate Jan reframe mandrill bronzes demonetising abroad. Shepard misspends seriatim. Run-down Thaine bloat Order Kamagra Online Australia intergrade beweep aptly? Sural Travers precluding shiningly. Immunized Beowulf emancipating, Buy Kamagra Pattaya taxi keenly.
Lovelorn unroped Rudie ruralizing Buy Kamagra Glasgow Kamagra Online Australia Paypal barb shoot-out apothegmatically. Unbrace entrancing Super Kamagra Online Uk treadling latest? Chilopod Micah prologuise layette cushion unpractically. Alright Drake dilating rhythmically. Conroy nidify next-door. Amylaceous Mylo poops, slummy cheesed crash-diving genuinely.
Implicative violent Perry degumming multiplepoinding Where To Buy Kamagra In Kuala Lumpur jellified glorify whereby. Sol epitomize drearily? Messier Englebart hinges straightforward.
Delphic Axel befriends Buy Kamagra Direct From India dabs cose east-by-north? Derek exacerbating convulsively?
Thru Carter renounced, Buy Kamagra Gel Online homologates anywise. Concurrent hateful Markos aggrieving conservancies Where To Buy Kamagra In Kuala Lumpur tingle denudes prayerfully. Well-lined Werner ladle Is It Safe To Buy Kamagra Online bundling foul.
Winslow fagots heavy. Ritch disagree dorsally. Gary rejoin visually. Unexpressible septimal Umberto wage To chauvinists Where To Buy Kamagra In Kuala Lumpur despumated bottle-feed laterally?
To contact Paul Merry, or speak with him, please ring +44 (0)1747 851102, email using , or visit our Kamagra Buy London page.


text:
Strategic Pillars
Market Competition and Healthcare Policy
Health Delivery Performance
Health of Populations and Health Equity
Innovation: Health Technology and System
Expert Spotlights
Maryland Data Hub
Medicare Advantage Data Lab
AI and Healthcare
Archived Workgroups
Graduate Institute
What's Holding Doctors Back from Using Artificial Intelligence?
Featuring Tinglong Dai, Shubhranshu Singh
Innovation: Health Technology and Systems
Informatics and Information Technology
Tinglong Dai
Artificial intelligence (AI) holds the promise of helping physicians make better treatment plans for their patients' unique needs. While doctors now have more access to AI than ever, with hundreds of AI-based medical devices approved by the FDA, the technology's full potential has yet to be realized. Tinglong Dai, PhD, and Shubhranshu Singh, PhD, two professors from the Johns Hopkins Carey Business School and core faculty members in the Hopkins Business of Health Initiative, authored and posted a paper online examining how physicians could decide to use AI.
Shubhranshu Singh
Dai and Singh developed a theoretical model demonstrating that physicians' use of AI in their clinical practice could be influenced by their liabilities to their patients and revenue generated from its use. Physicians could have the greatest incentive to use AI when they are confident in their treatment decision and expecting low liability. Overutilizing AI in these scenarios could raise revenues, which are beneficial for the practice though not for keeping healthcare costs down.
In the paper, Dai and Singh offer an unexpected perspective that differs from policymakers and AI developers. Whereas the latter feel enthusiastic that as AI becomes more precise it will be more widely used and replace the standard of care, the researchers suppose that physicians could feel apprehensive of the pressure AI puts on their decision-making when they are at odds with one other. For example, there could be an uncertain situation where a physician's expertise and experience deem it best to prescribe a higher chemotherapy dose than the routine regimen. If this plan deviates from the AI's treatment prediction, the physician may feel the burden of higher liability if the outcome proves to be unfavorable. The potential for conflict with the AI could dissuade the physician from using it in such a case. As such, the researchers suggest that "policymakers should focus on incentivizing physicians to use AI appropriately but not necessarily more widely."
Consumer Confidence in Public and Private Organizations to Use Their Digital Health Data Responsibly
AI in Healthcare Is Here, But Uptake Is Slow
The Joys and Sorrows of Incorporating Artificial Intelligence into Healthcare Delivery
Copyright ©2022 Johns Hopkins University. All rights reserved.
100 International Drive, Baltimore, MD 21202


text:
Fractionating the unitary notion of dissociation:disembodied but not embodied dissociative experiences are associated with exocentric perspective-taking
Braithwaite, Jason J., James, Kelly, Dewe, Hayley, Medford, Nick, Takahashi, Chie and Kessler, Klaus (2013). Fractionating the unitary notion of dissociation:disembodied but not embodied dissociative experiences are associated with exocentric perspective-taking. Frontiers in Human Neuroscience, 7 ,
It has been argued that hallucinations which appear to involve shifts in egocentric perspective (e.g., the out-of-body experience, OBE) reflect specific biases in exocentric perspective-taking processes. Via a newly devised perspective-taking task, we examined whether such biases in perspective-taking were present in relation to specific dissociative anomalous body experiences (ABE) - namely the OBE. Participants also completed the Cambridge Depersonalization Scale (CDS; Sierra and Berrios, 2000) which provided measures of additional embodied ABE (unreality of self) and measures of derealization (unreality of surroundings). There were no reliable differences in the level of ABE, emotional numbing, and anomalies in sensory recall reported between the OBE and control group as measured by the corresponding CDS subscales. In contrast, the OBE group did provide significantly elevated measures of derealization ("alienation from surroundings" CDS subscale) relative to the control group. At the same time we also found that the OBE group was significantly more efficient at completing all aspects of the perspective-taking task relative to controls. Collectively, the current findings support fractionating the typically unitary notion of dissociation by proposing a distinction between embodied dissociative experiences and disembodied dissociative experiences - with only the latter being associated with exocentric perspective-taking mechanisms. Our findings - obtained with an ecologically valid task and a homogeneous OBE group - also call for a re-evaluation of the relationship between OBEs and perspective-taking in terms of facilitated disembodied experiences.
https://doi.org/10.3389/fnhum.2013.00719
Life & Health Sciences > Psychology
Life & Health Sciences > Clinical and Systems Neuroscience
Life & Health Sciences > Centre for Vision and Hearing Research
© 2013 Braithwaite, James, Dewe, Takahashi, Medford and Kessler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
perspective-taking,depersonalization,dissociation,out-of-body experience,anomalous bodily experiences
http://www.fron ... 2013.00719/full
Braithwaite, Jason J.
James, Kelly
Dewe, Hayley
Medford, Nick
Takahashi, Chie
Kessler, Klaus ( 0000-0001-7307-9539)
Kessler, Klaus


text:
Q: Command line tools are missing after install of ncache.opensource.clr40.x64.msi (version 4.9 SP1) After installing the .msi from a command line with administrator privileges, I noticed that the command line tools (e.g. clearcache.exe, etc.) are missing from the Program Files\NCache\bin\tools directory. Any direction on how I can get the tools installed?
Thanks for any assistance you can provide.
I've run the .msi in interactive mode from a command prompt that has administrative privileges.
I was expecting the command line tools like clearcache.exe, addqueryindex.exe, etc. to be present. Instead, no tools are present.

A: The command tools are not available with the open source version. Here is a comparison chart that states this: http://www.alachisoft.com/ncache/edition-comparison.html

redpajama_set_name: RedPajamaStackExchange


text:
2021 ASEE Virtual Annual Conference Content Access (2)
2012 ASEE Annual Conference & Exposition (1)
Results by Division
Computing & Information Technology (1)
Design in Engineering Education (2)
Educational Research and Methods (1)
Electrical and Computer (1)
Test Anxiety and Its Impact on Diverse Undergraduate Engineering Students During Remote Learning
Studies of Classroom Assessment: Exam Wrappers, Equitable Grading, Test Anxiety, and Use of Reflection
2021 ASEE Virtual Annual Conference Content Access
David A. Copp, University of California, Irvine; Alexander J. Headley, University of Memphis
Educational Research and Methods
Paper ID #34305Test Anxiety and Its Impact on Diverse Undergraduate EngineeringStudents During Remote LearningDr. David A. Copp, University of California, Irvine David A. Copp received the B.S. degree in mechanical engineering from the University of Arizona and the M.S. and Ph.D. degrees in mechanical engineering from the University of California, Santa Barbara. He is currently an Assistant Professor of Teaching at the University of California, Irvine in the Department of Mechanical and Aerospace Engineering. Prior to joining UCI, he was a Senior Member of the Technical Staff at Sandia National Laboratories and an
Relevance Score: 2.6852243
Whom Are We Serving? An Exploration of Student Demographics in a Large Engineering Design Projects Ecosystem
Best In DEED
David A. Copp, University of California, Irvine; Alejandra Hormaza Mejia, University of California, Irvine; Mark E. Walter, University of California, Irvine; Natascha Trellinger Buswell, University of California, Irvine
Design in Engineering Education
Paper ID #30185Whom are we serving? An exploration of student demographics in a largeengineering design projects ecosystemProf. David A. Copp, University of California, Irvine David A. Copp received the B.S. degree in mechanical engineering from the University of Arizona and the M.S. and Ph.D. degrees in mechanical engineering from the University of California, Santa Barbara. He is currently an Assistant Professor of Teaching at the University of California, Irvine in the Department of Mechanical and Aerospace Engineering. Prior to joining UCI, he was a Senior Member of the Technical Staff at Sandia National
Team Formation and Function Decisions and Student Roles on Diverse Engineering Design Teams
Design Teams 2
Paper ID #34035Team Formation and Function Decisions and Student Roles on DiverseEngineering Design TeamsDr. David A. Copp, University of California, Irvine David A. Copp received the B.S. degree in mechanical engineering from the University of Arizona and the M.S. and Ph.D. degrees in mechanical engineering from the University of California, Santa Barbara. He is currently an Assistant Professor of Teaching at the University of California, Irvine in the Department of Mechanical and Aerospace Engineering. Prior to joining UCI, he was a Senior Member of the Technical Staff at Sandia National Laboratories and an adjunct
Electrical and Computer Engineering Programs in Two Countries: A New Paradigm for Cooperation
Innovative Instructional Strategies and Curricula in ECE II
Orlando R. Baiocchi, University of Washington, Tacoma; David A. Rogers, North Dakota State University; Max H. M. Costa, University of Campinas (Unicamp), School of Electrical and Computer Engineering
Electrical and Computer
AC 2011-2291: ELECTRICAL AND COMPUTER ENGINEERING PRO-GRAMS IN TWO COUNTRIES: A NEW PARADIGM FOR COOPERA-TIONOrlando R. Baiocchi, University of Washington, TacomaDavid A. Rogers, North Dakota State University David A. Rogers is a Professor in the Department of Electrical and Computer Engineering at North Dakota State University (NDSU) in Fargo. His engineering technical interests are applied electromagnetics and fiber optics. He received the B.S.E.E. cum Laude from the University of Washington in 1961, the M.S.E.E. from Illinois Institute of Technology in 1964, and the Ph.D. (E.E.) from Washington in 1971. He earned registration as a Professional Engineer (Electrical Engineering) in the State of Washington in 1972. In
Relevance Score: 0.55079514
The Future of the Basic Building Block of Telecommunications Network
Curricular Issues in Computer-oriented Programs
Ibraheem A. Kateeb, North Carolina A&T State University; Larry Burton, North Carolina A&T State University; Naser El-Bathy P.E., North Carolina A&T State University; Michael S. Peluso, North Carolina A&T State Univerisity
Computing & Information Technology
Electronics, June 2007[3] http://www.cablinginstall.com/index/display/article-display.articles.cabling-installation- maintenance.volume-18.issue-9.features.the-need-for-speed-drives-high-density- connectivity.htm[4] http://www.telecompaper.com/news/kroes-unveils-plans-to-boost-fibre-investment[5] http://www.siliconrepublic.com/comms/item/21241-bt-to-deploy-next-gen-coppe[6] Computer security basics Rick Lehtinen, Deborah Russell, G. T. Gangemi O'Reilly Media, Inc., 2006[7] Transmission Media Security – Charles R. Esparza – August 18, 2004[8] Source: AT&T 2009 Annual Report[9] http://www.economist.com/blogs/babbage/2011/01/last_mile[10] Predicting Subscriber Dissatisfaction and Improving


Our Love Me Love Heart is a mini stitch from our Tuck Shop collection.
Our Cutie Pie Love Heart is a mini stitch from our Tuck Shop collection.
Our Kiss Me Love Heart is a mini stitch from our Tuck Shop collection.
Our Scandi Fairisle cross stitch Stocking matches our Scandi Fairisle square cushion from our Crafteratti Collection.


ABOUT 23 million Americans are in long term recovery from alcohol and drug addiction.
According to Faces and Voices of Recovery executive director Patty McCarthy Metcalf, the Commonwealth of Northern Mariana Island should be part of this community as there are also recovering alcohol and drug dependents in the islands.
"You have people here in CNMI that are also living in recovery but don't even talk about it. We want people to be not ashamed and come out in their recovery," Metcalf, who was on Saipan to give training and technical assistance to the Blue Ribbon Action Team and the Community Guidance Center staff, said.
At the same time, Metcalf expressed support for the Blue Ribbon Action Team and its structure in giving support to people with alcohol, drug-abuse or mental-health problems.
The Faces and Voices Recovery is a national organization that unites people around the country, the mainland, as well as on the island to be part of the recovery movement and to advocate and share their stories about what they have accomplished. The group educates people in recovery, Metcalf said.
"Through Faces and Voices Recovery, they get their families back – not losing their children to child welfare system and they rebuild family," she said.
The problem is they become isolated and harder for them to regain their places as community members, once they take down that path," she said.
Another difficulty, according to Metcalf, is the lack of coordination of resources and public awareness around the states and even on the islands. She said there is still stigma and discrimination from the people.
The organization, which was established in 2001, was founded by people who are on recovery and their family members.
"We are putting faces and voices of recovery. They have been doing something for the community. We are helping to strengthen them and give them a voice in Congress to get funding for the organization, so they can provide pure recovery support services and recovery drug court and emergency department," Metcalf said.


text:
Life Arts 6/6/2008 at 13:29:36
By scott creighton (Page 1 of 2 pages) (View How Many People Read This) No comments
(Morning was first published here, on July 18th 2007)
It's 10:00 am on the day the Senate failed to advance legislation to set a time table for withdrawal of US Troops from Iraq, because the Republican minority held their breath and stomped their feet and filibustered all night to keep the American People's wishes from being done. Because the President knows better than we do, what should be done there. Oh, I'm sorry; I mean "the Generals on the ground" do.
This just a week after the President commuted Scooter's well deserved jail time and one day after the ABC documentary about soldiers in Iraq facing gut wrenching conditions and basically pleading with Congress to bring them home.
This on the same day that Harriet Miers' seat sat empty as she contemptuously ignored a subpoena to testify before Congress. Three days after the White House claimed Executive Privilege as the reason to restrict access to documents that would shed light on the friendly fire incident that took a dedicated soldiers life.
JFK 1961
"Ladies and gentlemen;
The very word 'secrecy' is repugnant in a free and open society. And we are, as a people, inherently and historically opposed to secret societies, to secret oaths and to secret proceedings. We decided long ago, that the dangers of excessive and unwarranted concealment of pertinent facts far outweighed the dangers which are cited to justify it."
This is just one day after President Bush spoke to a gathering of Republican lawmakers in DC and swore he would never leave Iraq; or, should I say, our troops would never leave Iraq.
Condi Rice was apparently running back and forth all night meeting with Republican Senators, in private, and no doubt making deals, pushing spin, handing off fresh new copy of speeches that AEI writers penned just minutes before.
She worked hard to keep our troops in harms way; kicking in the doors of frightened civilians, and laying the ground work for her "New Middle East". She worked harder than she has her entire life, to keep our soldiers on IED infested roads and in the cross hairs of insurgent snipers. She should be proud of her work.
And when she appears at the luncheons and gets the pats on the back from the Middle Aged Investment Class 1%ers, she can smile that self confident smile of hers knowing she has helped make the world safe for Democracy, dividends, and an endless war-time economy.
Remember that soldier at the end of the documentary who said he'd stay another tour if the law makers would join him? The one that wiped a tear from his face? I wonder what a warrior like that has to see to make him shed a tear. I wonder about things like that. It's called empathy and there are those who consider it a sickness in this country right now.
I wonder where he is, right now, that soldier.
I wonder where Condi Rice is.
She's probably lying next to that AEI fellow who helped write the initial plan for the pre-emptive strike on Iraq and then also served on the 9/11 Commission to help make sure the commission didn't publish anything that would be to embarrassing for the White House.
Yeah, she's getting a smile from one of the planners that helped take us to war, as that soldier is stepping out of a flat bottom humvee and moving toward another house, after only 4 hours rest, to see if the 6 year old in there has any Anti-American propaganda.
scott creighton Social Media Pages:
Attended J.M.U. and V.C.U. in the '80s and '90's. Worked as a builder and installer of film, stage, and theme park sets and attractions. Currently works as a designer and draftsman for residential and commercial projects. Wrote "Prescription for a (more...)
Related Topic(s): People; Rights, Add Tags
9/11, Afghanistan, Iraq, Georgia, Iran, and Pakistan; The Vulcan's Path to The Caspian Sea Basin
Beware the New Terrorist Scapegoat
Meet Gaith Pharaon
Selling Hope: Wexler's Impeachment Charade
The Questionable Timing of the Pakistan Blast
On Line Petition to Remove Pelosi as Speaker of the House of Representatives
"Rob Kall's writings on the "bottom-up" revolution have real potential to show people that they can hold power accountable and improve justice. A book on this subject could help to repair the economic, legal, social, and political fabric of the United States."
Paul Craig Roberts, former Assistant Secretary of the Treasury, former editor and columnist for the Wall Street Journal, Business Week and Scripps Howard News Service.


This was my script. While speaking, I embellished, compressed, and improvised a bit, but this is the gist of what I said. Before I started, I clipped a road map of Martha's Vineyard to a banner behind the podium and just to my right.
Which Martha's Vineyard do you live on?
Now there's a strange question. How many Martha's Vineyards are there? Just one, right? This one here.
We all recognize the map as Martha's Vineyard, but is this the map we use to get around the island? Not if we live here. Maps like this one are for tourists and newcomers. The map we navigate by day after day is the map in our head. Call it our psychic map.
Have you ever walked down a street -- say Circuit Avenue here in Oak Bluffs -- and gone into a new shop, only to discover that the shop isn't new at all; it's been right there for two or three years? It wasn't on your psychic map. You might have walked by it a hundred times, but you never really saw it.
If you're in business, you probably put plenty of effort into getting your store or company onto other people's psychic maps. You want them to think of you when they need whatever service or product you're selling, and even when they don't.
On my psychic map of Martha's Vineyard, State Road ends right around my friend Cris's house, across from Rainbow Farm, where it's Chilmark on one side of the road and West Tisbury on the other. Except when I have to go to the dentist -- his office is in Edgartown -- my Martha's Vineyard starts to dissolve about half a mile east of Barnes Road. The Martha's Vineyard I live on looks something like this. [Here I blocked off Chilmark and Aquinnah with one piece of construction paper, and Edgartown with another.] If I had a Magic Marker, I'd darken State Road and Old County Road so they looked like major arteries. On my psychic map, that's what they are.
Probably no one in this room has a psychic map that looks exactly like mine. No two people in this room have exactly the same psychic map of Martha's Vineyard, but our maps all have some features in common. I'm willing to bet that the Ocean View is on the psychic map of everybody in this room, at least on Wednesday. What else is on your psychic map?
No matter what our lives on Martha's Vineyard look like, nearly all of us adults have two regular activities: we buy groceries and we get the mail. So everyone's psychic map will include at least one grocery store and at least one post office -- even if you get your mail from a box on the road, when you need stamps or someone sends you a parcel that won't fit in the box, you go to a post office. What post office do you go to most often? OK, visualize yourself going to that post office. What road do you take? Where do you usually park?
For as long as I've lived on Martha's Vineyard -- that's going on 25 years now -- I've gotten my mail at the West Tisbury post office. When the main West Tisbury post office was at Alley's General Store, Alley's loomed large on my psychic map. When it moved down-island to its current location, my psychic map began to change. Over the months and years, Alley's faded and the stretch of road between the Tisbury-West Tisbury town line -- near which I was living -- and the new post office became more and more detailed. My mailing address didn't change, but my experience of getting my mail did.
For more than half my time on Martha's Vineyard, I lived in Vineyard Haven, so I also know the Vineyard Haven post office pretty well. I still go there fairly often. Both it and the West Tisbury post office are on my psychic map.
The Edgartown post office is not on my psychic map. I go there at most once a year. Going to the Edgartown p.o. can be a nerve-racking experience. Until I see the entrance to the mini-mall, I'm not sure where to turn off the Edgartown-Vineyard Haven Road, so either I poke along scrutinizing the exits and entrances, or I come to a screeching halt to avoid missing my turn. Once in the parking lot, I'm befuddled by the lines on the asphalt -- they're all of different lengths and where the hell am I supposed to park?
Keep in mind that I deal with odd-shaped parking lots all the time. The West Tisbury post office, where I get my mail, shares its parking lot with up-island Cronig's. Between the cars, trucks, pedestrians, and VTA buses, not to mention the occasional cyclist or skateboarder, navigating that parking lot is like running an obstacle course where all the obstacles are moving. The parking lot between Granite and the Edgartown post office isn't that bizarre -- I just don't know it very well.
So, having parked my pickup, I proceed to the post office door. I open it -- now what? Which way do I go? I don't recognize a soul, either in line or behind the counter. I buy my stamps, relieved that the clerks accept the currency I've got in my wallet, because it feels as though I'm in another country. In a way I am. I've fallen off the edge of my psychic map and into unknown territory.
For a day or two or three after I've been to Edgartown, there's a dim glow on my psychic map representing the Edgartown post office, but it soon fades to almost nothing. Intellectually I know that every day cars are driving into that parking lot -- all day long, people are getting out of those cars and walking to the post office. The Edgartown post office, and Granite, and the bank, and Edgartown Pizza, are vital parts of the psychic maps of many Vineyarders -- but they're not on my map.
That is how I know that there are many Martha's Vineyards, not just one. They overlap at many points -- I'll bet that the Steamship Authority dock in Vineyard Haven is on just about everybody's map -- but no two of our Martha's Vineyards are exactly the same.
When a place doesn't exist on your psychic map, going there can be downright scary. In 1985 I left an urban community where I knew the ropes pretty well and landed on Martha's Vineyard, where I didn't know anything. The most detailed area on my psychic map was Tisbury Great Pond and the three miles of the Edgartown Road between the airport and Alley's. I had the psychic map of a summer person, and I was planning to stay for the winter. By the end of September, nearly all the people I knew had left. I knew exactly one year-round islander.
Fortunately some people are very good at expanding the psychic maps of others, and within a few months I had met one of them: the late Mary Payne, the founder and artistic director of Island Theatre Workshop. For those of you who didn't know her: Mary was a pint-sized dynamo who rarely took "no" for an answer, especially when she was recruiting for things theatrical. (She probably hit at least some of you up for ads, or window space to put a poster in, or items to clothe actors or furnish sets with.) She thought that theater should be on everyone's psychic map, whether they knew it or not. I didn't need much persuading. I'm a performer at heart, and I'd been reading my poetry in public for several years. Besides, I didn't know anybody on Martha's Vineyard and I needed a life.
Rehearsals were then under way for ITW's production of Molière's The Miser. Come by Katharine Cornell Theatre, said Mary, and we'll put you to work. My first winter rental wasn't far from Five Corners, and I'd done lots of walking around Vineyard Haven, so I knew how to get there. I knew that Katharine Cornell Theatre was above the Tisbury town hall on Spring Street. Mary rattled off the rehearsal schedule, most of which I promptly forgot, but I knew what her car looked like. If her car was parked on Spring Street, it was a good bet that a rehearsal was in progress.
Sounds simple, right? It wasn't. I knew where Katharine Cornell Theatre was, and what its exterior looked like, but it takes more than that to put a place on a person's psychic map. In a way, the theater was on my psychic map. It was the place at the edge of the known world where the old maps used to say HERE BE DRAGONS. Danger danger danger! I knew there were no dragons above the Tisbury town hall, or alligators or snapping turtles either. What I imagined was a huge, dimly lit space, like a warehouse, with ceiling so high and walls so far away that I couldn't even see them. In the far, far distant opposite corner there was light, and tiny people moving around with apparent purpose: theater people. It was like looking through the wrong end of a telescope. The people were absorbed in what they were doing. They didn't notice me. They were theater people, and I was not.
The space was overwhelmingly vast, the people so distant and so oblivious to my presence. I didn't dare go in. I didn't want those people to look at me like an annoying trespasser. I'd blurt out some explanation that began with "Mary said . . ." and they'd say, "Mary? Mary who? Anyone hear know someone named Mary?"
So I'd walk by the town hall at hours when no rehearsal could possibly be going on. That way I had an excuse for not going in.
One afternoon I mustered the nerve to walk up the stairs. Mary's car was parked on Spring Street, so I knew there was a rehearsal going on. I walked up one flight, passed the double doors, and continued down the other side, without stopping. I didn't dare open the door. What if I went in and didn't recognize anybody? What if no one recognized me?
Well, Mary noticed that I hadn't shown up at a rehearsal, and like I said, she didn't take no for an answer. She kept nagging me and I was running out of excuses -- the kind of plausible excuses that you can actually say to someone's face. I wasn't about to say, "I haven't come to rehearsal because I'm afraid that Katharine Cornell Theatre is this huge dark cavern and no one will recognize me and they won't know who you are." That sounded so ridiculous that opening the door seemed less scary. Besides, it sounded as though Mary really wanted me to show up.
So one afternoon I opened the door and went in.
My immediate impression was an all-enveloping green -- the pea-soup green of the old seats, bathed in light from the several tall windows on two facing walls. The stage was across the room, but even that wasn't far away; I'd been in living rooms that were larger than this theater. There were several people about, and to my immense relief one of them was Mary. There were also several dogs, a small black one, a small white one, and a medium-sized one that looked at least part border collie.
Walking through that door stretched my psychic map of Martha's Vineyard to at least double its previous size. All I did for The Miser was to help with PR -- writing press releases, putting up posters, that kind of thing. This was enough to get me admitted free to any performances I wanted, and that was enough to get me hooked on theater. Theater people became my first island family, the ones I hung out with and learned the ropes from, the ones who invited me to join them for holidays and special events. Theater also expanded and enriched my relationship to the written word; when, many years later, I was working on my novel, I drew continually on what I'd learned in island theater, as a stage manager, actor, and reviewer.
More than a dozen years after I first set foot in Katharine Cornell Theatre, not long after I'd got back into horses after 30 years away, I was startled to hear another horsewoman ask where Katharine Cornell Theatre was. This woman was only a little younger than I, she'd grown up in West Tisbury, and she didn't know where Katharine Cornell was? Further conversation revealed that she, of course, knew where the Tisbury town hall was; she just didn't realize that the space above it had been Katharine Cornell Theatre since 1971.
This mystified me -- until I'd been involved with horses for a year or so. Having a horse pretty much takes over your life. If you work full-time, it doesn't leave much time for other recreation. Theater likewise eats up all the free time you have available, and then some, especially when you're involved in a production. This other woman had been involved with horses nearly all her life, while working full-time and raising a family. I was no longer surprised that Katharine Cornell Theatre and the whole island theater scene weren't on her psychic map, or that horse people and theater people tend to live on such different Martha's Vineyards that they don't run into each other very often, or know each other very well.
Psychic maps aren't just three-dimensional. They have a fourth dimension: time. Once upon a time, there was a car wash behind Alley's General Store, and then a laundromat. Those long-gone places are still on my psychic map, even though they aren't on any current paper or online map.
This road map doesn't acknowledge time. If a new road is cut today, it won't magically appear on the map. This map doesn't change through the seasons either. As far as the map is concerned, Martha's Vineyard in January looks just like Martha's Vineyard in July. You and I know better. We know that it takes more time to get from West Tisbury to Oak Bluffs or Vineyard Haven to Edgartown in July than it does in January -- barring a major snowstorm, of course. The distance doesn't change, but the time required to travel it does.
My psychic map of Martha's Vineyard exists in all four seasons. For many, many people, Martha's Vineyard exists at most during four months of the year: June, July, August, and September. Summer residents and visitors have a very different psychic map of Martha's Vineyard than year-round residents do. I didn't know this until I moved here.
In the early 1980s, when I was still living in Washington, D.C., I read a story in the Washington Post's Sunday magazine section about people on Cape Cod who had to move twice a year. Year-round rentals were hard to find because landlords wanted to get top dollar renting to summer residents and vacationers. The upside was that winter rentals were cheap and plentiful. The downside was that you had to find somewhere else to live during "the season," which might run from the first of May to the first of October, and that during "the season" affordable accommodations were very hard to find. I was shocked. How could anyone live this way?
So I landed on Martha's Vineyard on changeover weekend of 1985, and in the next three years I moved eight times. Moving twice a year was hard, but it was no longer shocking. Plenty of my friends and acquaintances were doing the same thing. My psychic map of Martha's Vineyard was expanding. I was making the transition from summer visitor to year-round resident.
"The year-round folk are taciturn and mad.
There's no one here in winter." So she says.
away to entertainments, mainland style.
where waves stand still, and every beach is clean."
That turned out to the first sonnet in a six-sonnet sequence called "Winter Rental," about, you guessed it, moving twice a year and, more generally, about my first year as a year-round Vineyard resident.
the tidal wave like surfers. . . .
A woman came up to me afterward. She said she'd enjoyed my reading, but she thanked me especially for the "Winter Rental" poems. She and her kids moved twice a year. They were managing, but it was hard. And this was the first time she'd heard a poem or story that spoke to her experience.
This is the kind of thing that keeps a writer going: to know that her words have made a difference to another person. I kept writing poetry and giving readings. I also went to work for the Martha's Vineyard Times first as a typesetter, then as a proofreader, and eventually as the editor of the Calendar and Community sections. I reviewed lots of plays and wrote lots of feature stories. Newspapers can be a hub of a community's life: all sorts of people find their way there, to place ads, pay bills, deliver press releases, or chat with one of the editors or reporters. For a relative newcomer, which I still was, there's no better way to expand your psychic map of a place. All I had to do was listen.
But writers do more than listen. Writers, like actors, musicians, photographers, visual artists, storytellers, and others, have the potential to expand the psychic maps of everyone who comes in contact with their work. We can take you to new places, introduce you to new people, or perhaps show you familiar places and people from different angles.
To make a long story short, that's how I came to write The Mud of the Place. I had written poems about Martha's Vineyard, stories about Martha's Vineyard, newspaper articles and op-eds and essays about Martha's Vineyard, but the longer I lived here, the more clear it became to me that the story I wanted to tell wouldn't fit into a single poem, or story, essay, or play. I wanted to take readers behind the scenes, to show a little about how people and places interact with each other to get things done -- or not. For this I needed a bigger canvas. I'd never written a novel before. I'd never written anything longer than 40 pages. I had no idea how I was going to do it, and now I couldn't tell you in any detail exactly how I did.
Where do you live, and how do you pay for it?
Often they weren't sure, or they didn't want to talk about it, but I persisted. Leslie, the newspaper reporter in the novel, was embarrassed both because she was living rent-free in the summer house owned by her parents and because here she was, working for a weekly newspaper, at an age when her journalist father had already won a Pulitzer Prize. Alice has nowhere to live: the house she was living in with her young son just burned down, and what she wants most in the world is to move off-island to where she grew up and where her mother and sisters still live. Shannon managed to buy her own house, with help from her neighbors and a former partner, and she's doing pretty well as a graphic artist and volunteer -- but she's also a talented painter who locked the door to her studio several years before and doesn't dare unlock it. Her friend Jay, a native islander who lived off-island for twenty years and then moved back to take a perfect job, is afraid he'll lose both the job and the love of his family if they find out he's gay. And Wayne, Jay's ex-brother-in-law, blames Jay for the breakup of his marriage to Jay's sister and will stop at nothing to get even.
It takes about 400 pages for them, and several others, to work it all out, but they manage pretty well, with a lot of help from their friends. They open some doors and close others. Some psychic maps expand in the process. Others resist change and even begin to shrink.
Most of the places in The Mud of the Place are probably on the psychic maps of everyone in this room, although some of them no longer exist, like the old terminal at the Martha's Vineyard Airport, and the Get a Life Café. But you won't run into any of the characters at the post office, or the grocery store, or walking down Circuit Ave. I can show you on this map where each of them lives, but if you go looking, you won't find their houses. If you want to add them to your psychic map, well -- you'll have to read the book.