\section{Introduction}

\begin{figure}[thb]
    \centering
    \begin{subfigure}[b]{0.98\linewidth}
        \centering
        \includegraphics[width=0.9\linewidth]{intro/caption1.pdf}
    \end{subfigure}
    \begin{subfigure}{0.96\linewidth}
        \includegraphics[width=0.32\linewidth]{intro/girl100.jpg}
        \includegraphics[width=0.32\linewidth]{intro/girl110.jpg}
        \includegraphics[width=0.32\linewidth]{intro/girl138.jpg}
        \caption{{\it girl}: total occlusion}
    \end{subfigure}
    \begin{subfigure}{0.96\linewidth}
        \includegraphics[width=0.32\linewidth]{intro/hel006.jpg}
        \includegraphics[width=0.32\linewidth]{intro/hel495.jpg}
        \includegraphics[width=0.32\linewidth]{intro/hel640.jpg}
        \caption{{\it helicopter}: abrupt changes in target appearance}
    \end{subfigure}
    \begin{subfigure}{0.96\linewidth}
        \includegraphics[width=0.32\linewidth]{intro/bas465.jpg}
        \includegraphics[width=0.32\linewidth]{intro/bas478.jpg}
        \includegraphics[width=0.32\linewidth]{intro/bas492.jpg}
        \caption{{\it basketball}: multiple object instances}
    \end{subfigure}
    \begin{subfigure}{0.96\linewidth}
        \includegraphics[width=0.32\linewidth]{intro/fish068.jpg}
        \includegraphics[width=0.32\linewidth]{intro/fish137.jpg}
        \includegraphics[width=0.32\linewidth]{intro/fish209.jpg}
        \caption{{\it fish}: similar foreground and background features}
    \end{subfigure}
\caption{DAWN can handle various difficult situations in unsupervised video object tracking whereas existing methods (ECO~\cite{eco}, ADNet~\cite{adnet}, DaSiamRPN~\cite{dasiamrpn}) fail in at least one of them.}
    \label{fig:teaser}
\vspace{-0.1in}
\end{figure}

Despite recent progress in deep learning and pertinent research on video object tracking, as exemplified in VOT challenges~\cite{vot2016,vot2017} and OTB benchmarks~\cite{otb2013,otb2015}, top contenders in these challenges and benchmarks still have problem in unsupervised object tracking where for a given video only the bounding box in the first frame is given and no reset is allowed, in the presence of total occlusion, abrupt changes in target appearance, multiple object instances, and similar foreground and background features, see Figure~\ref{fig:teaser}.

In this paper, we introduce a deep neural network augmented with both foreground and background memory blocks which are updated by an attention LSTM as a memory controller for unsupervised video object tracking.  Central to our network architecture is an external memory module to remember (and forget) evolving target object and scene structures which are inspired by~\cite{mavot,memtrack}, thus setting such memory-augmented tracking methods apart from top tracking methods such as~\cite{mdnet,deeptrack} (online learning),~\cite{adnet,DRL,rdt} (deep reinforcement learning), and \cite{eco,ccot} (coupling deep features with traditional features).

The central issue in unsupervised video object tracking is that target appearance given in the first frame is likely to dramatically change during the course of tracking, which can be complicated by occlusion and image degradation such as motion blurs.  Online tracking is a typical method to adjust a given model to adapt to changes during tracking.  However, updating network parameters through online backpropagation can be quite computationally expensive, as opposed to updating an external memory block where a lightweight read and write strategy can be adopted.

Such memory updating strategy for unsupervised object tracking, on the other hand, is susceptible to problems such as total occlusion and severe motion blurs which can contaminate memory blocks. To protect the memory from erroneous update due to the above problems, we utilize an attention LSTM as a memory controller.  The attentional module attenuates the effect of noise by focusing on the target despite partial occlusion or similar background features, thus making the memory I/O less vulnerable in our method.

By adding an identical memory module to remember (and forget) the evolving {\em background} as well, our dual memory augmented network (DAWN) can suppress confusing background that may look similar to the tracking target, and thus can seamlessly deal with the above difficult tracking situations which are problematic to conventional trackers. Such confusing background can distract these previous models and make them start tracking the occluder and thus the wrong object.

To  focus on dual memory and attention LSTM, we fix the aspect ratio of our bounding boxes on DAWN to make us comparable to previous memory-based methods~\cite{memtrack,mavot}. To  eliminate contributing factors other than the two technical contributions, we do not have sophisticated engineering other than restarting DAWN after total occlusion, so we evaluate our model in VOT2016~\cite{vot2016} and VOT2017~\cite{vot2017} competitions including comparison with the VOT2018~\cite{vot2018} real-time champion, showing DAWN performs significantly better than state-of-the-art trackers while running fps $>$ 10.

\section{Related work}

\subsection{Deep Feature in Tracking}
Over the past decade, we have witnessed the significant development of deep learning on many important computer vision tasks, particularly object classification and detection~\cite{vgg,resnet,faster_rcnn}. Recently deep learning has been extensively employed in tracking thanks to its outstanding feature representations. A number of trackers, coupling deep features with traditional features as correlation filters such as C-COT~\cite{ccot} and ECO~\cite{eco} have shown their effectiveness on various tracking benchmarks~\cite{otb2013,otb2015} and challenges~\cite{vot2016,vot2017}.

Tracking methods based on Tracking-by-Detection~\cite{survey,survey2,deeptrack} build a classifier that separates the target from the enclosing background. However, training data of these online-only approaches is the input video itself which fundamentally limits their learned models. The MDNet~\cite{mdnet} solves this problem by end-to-end learning of an offline deep feature extractor, where the classifier will be refined online. Although it has achieved competitive performance, its speed is restricted by online training and updating.

Deep feature matching based methods are gaining much attention because of its excellent speed and tracking performance. The GOTURN~\cite{goturn} learns target tracking states by comparing and matching feature pairs in consecutive frames, so it cannot handle target occlusion in principle. The SiamFC~\cite{SiamFC} uses fully convolutional Siamese networks to match features between template frame and target frame. However, this method only utilizes the first frame as the template which will not work well when the tracking object undergoes notable deformation in the long term.

\begin{figure*}[tb]
    \centering
    \includegraphics[width=0.96\linewidth]{graph/DAWN_model.pdf}
    \caption{General structure of our network, where {\bf attention LSTM} (Figure~\ref{fig:attention_comparison}) and {\bf dual memory} (Figure~\ref{fig:dual_mem}) will be detailed. Blue rectangle in the frame image is the Region Of Interest (ROI) where ROI features are extracted. ROI features are then fed into attention LSTM together with foreground memory read in the last frame, which are also fed into a conventional LSTM. The outputs from two LSTMs are used to read memories from foreground and background memory blocks respectively. These memories are then used to generate a heat map to predict the bounding box which divides ROI into two parts, a foreground image, and a background image. Features extracted from the two images are then used to write into foreground and background memory blocks respectively.}
    \label{fig:model}
\vspace{-0.2in}
\end{figure*}

\subsection{Memory Structure in Tracking}
To solve the static template frame problem in GOTURN and SiamFC mentioned above, memory blocks are adopted in a number of contemporary trackers. The RDT~\cite{rdt} uses a template pool to store the target's different appearances. Even though its template pool updating is trained by deep reinforcement learning~\cite{DRL}, the target appearances written and read are discrete. MemTrack~\cite{memtrack} uses NTM (Neural Turing Machine)~\cite{NTM} as its memory block to store continuous feature of the target, and uses LSTM~\cite{lstm} to dynamically control memory reading and writing. They used gated residual template learning to manage the quantity of retrieved memory which is used to mix with the starting template. This memory architecture is commonly used in many fields of deep learning, especially in one-shot learning, such as~\cite{mmnet,santoro2016one}. On the other hand, MAVOT~\cite{mavot} retrieves and memorizes the information of both foreground and background to support background suppression. A similar idea was adopted in DSiam~\cite{DSiam} where a dynamic Siamese network was used for target image variation and background suppression. Our improved attention LSTM memory controller is closely related to~\cite{memtrack} and background memory to~\cite{mavot}, which will be explained in the next section.

\subsection{Attention Mechanism in Tracking}
Attention models, initially applied in image recognition tasks, was then combined with a recurrent model to form Recurrent Attention Model (RAM)~\cite{RAM}. The RAM soon became popular in multiple areas such as image captioning~\cite{Xu2015ShowAA}, image classification~\cite{Wang_2017}, pose estimation~\cite{Chu_2017_pose}, etc. In multiple objects tracking, STAM~\cite{Chu_2017_attention} uses the attention model to deal with occlusions and interactions among targets. For single object tracking, RASNet~\cite{RASNet} introduces general attention, residual attention and channel attention. MemTrack~\cite{memtrack} uses a common RAM structure similar to those in NLP tasks to roughly locate the object, and provide the memory block a correct retrieval key. However, we found that this structure can give bad location estimation and consequently, it tracks the background and memorizes the relative location of the object context. In this paper, we modify the RAM structure to produce more accurate estimations of the target's location. While a single object is tracked in unsupervised, it may be partially or totally occluded during tracking, which will cause the model especially one with memory blocks to track another object instead of the original target. Our attention mechanism is designed to solve the occlusion problem, which demonstrates more robust behavior than memory structures without attention mechanism such as~\cite{mavot}.

\section{Dual Augmented Memory Network}
We present our DAWN model in detail. Figure~\ref{fig:model} shows the overall framework: our model consists of a dual memory structure for remembering the evolving foreground and background features, an improved attention LSTM and a conventional LSTM for controlling the read/write operations of the foreground and background memory respectively. Note that {\em no} backpropagation is run, online or offline during the tracking process. All feature extractors in DAWN share the same weights. In this section, $*$, $\cdot$ and $\odot$ denote element-wise multiplication, matrix multiplication and convolution operation respectively.

\subsection{Feature Extraction}
We follow the Fully Convolutional Neural Networks structure from SiamFC~\cite{SiamFC} to extract deep features within the ROI which include the deep features for both target and background. The output will then be fed into three branches, namely, ROI branch, foreground branch, and background branch. The ROI branch receives an ROI patch from the newest frame and bounding box predicted in the last frame. The output of the ROI branch is an $n \times n \times c$ matrix $\mathbf{F}$,  which consists of vectors $\mathbf{f}_{i,j}$ of size $c$ where $0 \leq i,j < n$. Working with the memory I/O, $\mathbf{F}$ will be used to predict the new bounding box.

The foreground branch receives the newest target patch from the bounding box predicted in ROI branch. The output of the foreground branch is an $m \times m \times c$ matrix $\mathbf{F_{fore}}$, which will be written to foreground memory.

The background branch receives the background information in ROI and outputs the feature of the background branch, an $n \times n \times c$ matrix, which will be written in background memory after an additional average pooling layer.
\vspace{-0.18in}
\subsection{Attention Based on Foreground Memory}
\label{sec:attention}
\vspace{-0.07in}
In video object tracking, a bounding box is an axis-aligned rectangle, so it includes a lot of background regions as well. Thus features extracted from a given bounding box contain noises which adversely affect both the performance and robustness of the model. To extract object features with less noise, Yang and Chan (MemTrack)~\cite{memtrack} introduced an attention LSTM as a memory controller, but their attention module acts more like a simple protocol for memory reading and writing. Often times, their attention seems to be random and consequently the model tracks some part of the background and remembers the target's relative position with respect to the tracked background regions.  Thus, once the target's relative position starts to change, MemTrack will lose track, see Figures~\ref{fig:handnbook} and~\ref{fig:butt}.

\begin{figure}[htb]
    \centering
    \begin{subfigure}{0.70\linewidth}
        \includegraphics[width=\linewidth]{graph/Memtrack_attention.pdf}
        \caption{MemTrack}
    \end{subfigure} \\
    \begin{subfigure}{0.70\linewidth}
        \includegraphics[width=\linewidth]{graph/DAWN_attention.pdf}
        \caption{DAWN}
    \end{subfigure}
    \caption{Attention modules of MemTrack~\cite{memtrack} and DAWN.}
    \label{fig:attention_comparison}
\vspace{-0.05in}
\end{figure}

Since attention can be roughly interpreted as an estimation of the target object's position, the concept of memory augmentation can also be applied here. Specifically, to achieve better estimation, we introduce a memory augmented attention block whose difference from the attention block in MemTrack~\cite{memtrack} is shown in Figure~\ref{fig:attention_comparison}. In DAWN, we utilize the memory read from the last frame, i.e., a $m \times m \times c$ matrix $\mathbf{M}$ consists of vectors $\mathbf{m}_{i,j}$ of size $c$, where $0 \leq i,j < m$.

First, to generate attention score, $\mathbf{M}$ is convolved with $\mathbf{F}$ which outputs an $a \times a$ matrix $\mathbf{A}$, as weights of attention scalars $a_{i,j}$, where $0 \leq i,j < a$ and $a = n - m + 1$, whose value is given by
\begin{equation}
    a_{i,j} = \frac{\exp(r_{i,j})}{\displaystyle\sum_{s=0}^{a-1}\displaystyle\sum_{t=0}^{a-1} \exp(r_{s,t})},
\end{equation}
where
\begin{equation}
    r_{i,j} = \displaystyle\sum_{s = 0}^{m-1} \displaystyle\sum_{t = 0}^{m-1} \mathbf{f}^{\top}_{i + s,j + t} \cdot \mathbf{m}_{s,t}.
\end{equation}

Then, we use average pooling on $\mathbf{F}$ with filter size $m \times m$ and stride size 1, and obtain $a \times a \times c$ matrix $\mathbf{F_{avg}}$:
\begin{equation}
    \mathbf{F_{avg}} = \mathit{AveragePooling}_{m \times m}(\mathbf{F}).
\end{equation}

Finally, we use the attention score $\mathbf{A}$ and averaged feature $\mathbf{F_{avg}}$ to produce attention feature $\mathbf{F_{att}}$:
\begin{equation}
\mathbf{F_{att}} = \mathbf{A} * \mathbf{F_{avg}}.
\end{equation}

Comparing with MemTrack~\cite{memtrack}, our attention mechanism has the following advantages:
\begin{enumerate}
\setlength{\itemsep}{0.0ex}
\item We use the last frame to extract the memory for generating attention, more inter-frame information being incorporated in DAWN during tracking. This is a more direct approach than MemTrack~\cite{memtrack}, which uses the LSTM hidden layer and current ROI feature to generate attention.
\item Our mechanism allows us to use the first frame target feature to initialize the memory, which is more stable and robust than LSTM hidden layer initialization. See the first frame in Figure~\ref{fig:butt}.
\item The uncompressed memory extracted from the last frame has less information loss than the compressed LSTM hidden layer.
\end{enumerate}

\subsection{Memory Controller}
\label{sec:controller}
DAWN uses standard LSTM with layer normalization~\cite{layer} and dropout regularization~\cite{dropout} in both foreground and background memory controllers, see Figure~\ref{fig:dual_mem}. The input to foreground and background memory controller are $\mathbf{F_{att}}$ and $\mathbf{F_{avg}}$ respectively, computed in section~\ref{sec:attention}. With the input feature and the previous hidden state $\mathbf{h_{t-1}}$, the hidden state updates to $\mathbf{h_{t}}$ in both controllers. Our memory reading and writing process are inspired by MemN2N~\cite{memn2n} and MemTrack~\cite{memtrack}. Here, we describe foreground memory reading and writing process. The background memory read/write is similar.

During a reading process, a template is retrieved from all the memory slots as a weighted sum, where the read weights are the \textit{softmax} result of the cosine similarity between the given read key $\mathbf{r_{t}}$ and all the memory keys. The read key $\mathbf{r_{t}}$ is calculated by hidden states $\mathbf{h_{t}}$ of LSTM:
\begin{equation}
\mathbf{r_{t}} = \mathbf{W}^{r} \cdot \mathbf{h_{t}} + \mathbf{b}^{r},
\end{equation}
where $\mathbf{W}^{r}$ and $\mathbf{b}^{r}$ are weight matrix and bias.
The memory key is the output of $m \times m$ average pooling on the corresponding memory slot.

During a writing process, the write weight $\mathbf{w_{t}}$ controls whether to update the memory and which slots to update. Using hidden states $\mathbf{h_{t}}$, we apply the same method as MemTrack~\cite{memtrack} to generate $\mathbf{w_{t}}$. The writing process is defined by
\begin{equation}
\mathbf{M_{t}} = (1-\mathbf{w}_{t}) * \mathbf{M_{t-1}} + \mathbf{w}_{t} * \mathbf{F_{fore}},
\end{equation}
where $\mathbf{F_{fore}}$ is the output of the foreground branch.

For LSTM initialization, we use the initial target's feature with average pooling and~\textit{tanh} activation to generate hidden state $\mathbf{h_{0}}$ and cell state $\mathbf{c_{0}}$.  We apply the Residual Template Learning adopted in MemTrack~\cite{memtrack} in foreground memory but not in background memory.

\subsection{Dual Memory Structures} \label{sec:DMS}
\begin{figure}[htb]
    \centering
    \includegraphics[width=0.80\linewidth]{graph/DAWN_memory.pdf}
    \caption{Dual Memory Structure of DAWN.}
    \label{fig:dual_mem}
\vspace{-0.15in}
\end{figure}

Since memory structure with foreground-only features shows limitations in tackling problems such as occlusion, multiple objects, and similar background features, etc., DAWN uses dual memory blocks, see Figure~\ref{fig:dual_mem}. Foreground memory can remember the appearance of the target, while background memory can record surrounding variation.

Reading from foreground and background memory has been discussed in Section~\ref{sec:attention} and~\ref{sec:controller}. After reading foreground memory $\mathbf{M}$ and background memory $\mathbf{M_{back}}$,  both with dimension $m \times m \times c$, we obtain $\mathbf{\overline{M}}$ by non-maximal suppression using subtraction:
\begin{equation}
\mathbf{\overline{M}} = \mathbf{M} - \mathbf{M_{back}}
\end{equation}

$\mathbf{\overline{M}}$ is convolved with the $\mathbf{F}$ to obtain the heat map $\mathbf{H}$ with dimension $a \times a$, which is calculated by:
\begin{equation}
    \mathbf{H} = \mathbf{F} \odot \mathbf{\overline{M}}.
\end{equation}

We up-sample the heat map $\mathbf{H}$ as in SiamFC~\cite{SiamFC} to predict the bounding box. Then we extract features from the image cropped with the bounding box and write them to the corresponding memory blocks. An additional average pooling layer is applied before writing background features into background memory.

Comparing with memory blocks only involving foreground features~\cite{memtrack}:
\begin{enumerate}
\setlength{\itemsep}{0.0ex}
\item {\em Occlusion}. DAWN can suppress the adverse influence of possible memory contamination caused by occlusion. Previously, when the target is being occluded, features of the occluding object will be written into and thus contaminate foreground memory (see `girl' and `frisbee' in Figure~\ref{fig:mem}), consequently causing the model to track a false background object. But with DAWN's background memory block, since the memory of a background object has been recorded in previous frames, subtracting its memory can mitigate the above contamination while keeping track the target after occlusion. When detecting total occlusion, DAWN will not update bounding box until the target re-appears in a subsequent ROI.
\item {\em Similar Background}. DAWN can distinguish other similar background objects or features, which usually confuse trackers with only foreground memory.  Since background features have been memorized, although they look similar to the target in foreground memory, DAWN can still differentiate them after subtracting the relevant background memory. Using cosine window to penalize large displacements can suppress similar objects in the background. See `basketball', `fish' in Figure~\ref{fig:teaser}, and `godfather', `gymnastics' in Figure~\ref{fig:mem}.
\end{enumerate}

\subsection{DAWN-PRN: DAWN with Region Proposal Network
}
\begin{figure}[htb]
    \centering
    \includegraphics[width=0.96\linewidth]{graph/net.pdf}
    \caption{
    DAWN-PRN: $\overline{M}$ is same and defined by Eq.~(7).}
    \label{fig:model_rpn}
\end{figure}

We change our tracking backbone from SiamFC~\cite{SiamFC} to SiamRPN~\cite{SiamRPN}, and call it DAWN-RPN, which is shown in Figure~\ref{fig:model_rpn}. In DAWN-RPN, we adopt the techniques in SiamRPN~\cite{SiamRPN} to generate bounding boxes. The reading and writing operations of memory is unchanged.

\section{Experiments}
We will present the comparative results with other trackers using VOT toolkits on two challenging tracking competitions: VOT2016 and VOT2017, and then show the respective improvement due to the attention and memory module. Please see our videos at \emph{https://zhmeishi.github.io/DAWN/} .

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.10\linewidth]{vot/vot2016_rank_2.png}
    \includegraphics[width=0.34\linewidth]{vot/vot2016_rank_1_m.png}

    \includegraphics[width=0.10\linewidth]{vot/vot2017_rank_2.png}
    \includegraphics[width=0.34\linewidth]{vot/vot2017_rank_1_m.png}

    \caption{DAWN is ranked $3^{rd}$ in VOT2016 (left) and VOT2017 (right) among all the VOT fast trackers running at fps $>$ 10. }
    \label{fig:stat1617}
\vspace{-0.15in}
\end{figure*}

\subsection{Implementation Details}
We apply the same Alex-like~\cite{alexnet} CNN feature representation as SiamFC~\cite{SiamFC}. The input image sizes of the ROI, target and background branch are respectively $255 \times 255 \times 3$, $127 \times 127 \times 3$ and $255 \times 255 \times 3$. The output matrix size of the foreground branch is $6 \times 6 \times 256$, and, for the ROI branch and background branch, the size is $22 \times 22 \times 256$. Our DAWN is pre-trained offline on the video object detection dataset of ImageNet Large Scale Visual Recognition Challenge (ILSVRC15)~\cite{imagenet}. The offline training strategy is the same as MemTrack~\cite{memtrack}, where the training optimizer is Adam~\cite{adam} with initial learning rate $1e^{-4}$ and the weight decay set to be $0.02$ every $10,000$ iterations. The number of memory slots is 8. We suppress the heat map with a cosine window by an exponential factor of $0.27$. The target image size is $1.32$ times bounding box size. We update the target size with exponential smoothing  $s_{t} = (1 - \alpha )s_{t-1} + \alpha s_{new}$ to deal with scale changes, where $s$ is the target size and the exponential factor $\alpha$ is 0.6.

Our tracker is implemented in Python using the TensorFlow framework. All of the experiments were conducted on the following hardware specifications: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, 16 GB RAM, and NVIDIA GPU GTX1070.

\subsection{VOT2016 and VOT2017 Results}

Using the VOT toolkit, we compare DAWN with all fast trackers (fps $> 10$) in VOT2016 and VOT2017 competitions which respectively contains 60 video sequences. Figure~\ref{fig:stat1617} shows the comparison results. Only fine-tuned on the training dataset, DAWN has consistently good performance in VOT2016 and VOT2017 with same hyper-parameters. Though DAWN ranks the third in VOT2016 and VOT2017, where the rank is measured by the expected overlap ratio (EAO), but our bounding box aspect is fixed, DAWN still has an excellent success rate in both VOT2016 (Table~\ref{table:suc_rate_16}) and VOT2017 (Table~\ref{table:suc_rate_17}) with the least number of failures.
\vspace{-0.1in}
\begin{table}[!h]
  \centering
\begin{tabular}{l|ccc}
\hline
      Measures & DAWN  & STAPLEp & Staple\\
\hline
EAO ($\uparrow$)           & \textcolor[rgb]{0,1,0}{0.28} & \textcolor[rgb]{0,0,1}{0.29} & \textbf{\textcolor[rgb]{1,0,0}{0.30}} \\
Failures ($\downarrow$)      &
\textbf{\textcolor[rgb]{1,0,0}{18.58}} & \textcolor[rgb]{0,1,0}{24.32} & \textcolor[rgb]{0,0,1}{23.89} \\
\hline
\end{tabular}
\caption{Model Performance in VOT2016}
\label{table:suc_rate_16}
\vspace{-0.15in}
\end{table}
\vspace{-0.1in}
\begin{table}[!h]
  \centering
\begin{tabular}{l|ccc}
\hline
Measures & DAWN  & ECOhc & SiamDCF\\
\hline
EAO ($\uparrow$)           & \textcolor[rgb]{0,0,1}{0.24} & \textcolor[rgb]{0,0,1}{0.24} & \textbf{\textcolor[rgb]{1,0,0}{0.25}} \\
Failures ($\downarrow$)      &
\textbf{\textcolor[rgb]{1,0,0}{26.58}} & \textcolor[rgb]{0,0,1}{28.77} & \textcolor[rgb]{0,1,0}{29.41} \\
\hline
\end{tabular}
\caption{Model Performance in VOT2017}
\label{table:suc_rate_17}
\vspace{-0.1in}
\end{table}

\subsection{VOT unsupervised Tracking}\label{sec:unsupervised}

Here we choose videos that are very difficult to track in the unsupervised mode and provide brief information as well as major tracking difficulties. Then we will tabulate the tracking results of DAWN and current state-of-the-art trackers.

The following summarizes the major problems in unsupervised tracking with their abbreviations used in Table~\ref{result_table}:
\vspace{-0.2in}
\begin{table}[!h]
  \centering
\begin{tabular}{ll}
OC: & Occlusion (Total or Partial) \\
MB: & Motion Blur (Severe) \\
AC: & Abrupt Changes in Target Appearance \\
SB: & Similar Background Objects or Features \\
\end{tabular}
\vspace{-0.1in}
\end{table}

\begin{table*}[th]
  \centering
  \linespread{0.0}
\begin{tabular}{l|c|ccc|ccc|ccc}
\hline
Video       & DAWN & MemTrack & MAVOT & SiamFC & KCF & DSST & DaSiam & ADNet & C-COT & ECO \\
\hline
\footnotesize{wiper \hfill(OC, MB)}         & \checkmark & \checkmark &  & \checkmark & \checkmark & \checkmark &  & \checkmark & \checkmark & \checkmark \\
\footnotesize{bolt1 \hfill(SB)}          & \checkmark &  &  & \checkmark & \checkmark &  & \checkmark & \checkmark & \checkmark & \checkmark \\
\footnotesize{butterfly \hfill(AC, SB)}         & \checkmark &  & \checkmark & \checkmark &  &  & \checkmark & \checkmark & \checkmark & \checkmark \\
\hline
\footnotesize{helicopter \hfill(AC)}      & \checkmark & \checkmark &  & \checkmark & \checkmark & \checkmark &  & \checkmark &  & \checkmark \\
\footnotesize{godfather \hfill(OC, SB)}       & \checkmark &  & \checkmark &  & \checkmark &  &  & \checkmark & \checkmark & \checkmark \\
\footnotesize{soccer2 \hfill(OC, MB)}         & \checkmark & \checkmark &  & \checkmark &  &  & \checkmark & \checkmark & \checkmark &  \\
\hline
\footnotesize{ants1 \hfill(SB)}           & \checkmark & \checkmark &   & \checkmark &  &  &  &  & \checkmark & \checkmark \\
\footnotesize{basketball \hfill(OC, SB)}      & \checkmark &  &  & \checkmark & \checkmark &  &  &  & \checkmark & \checkmark \\
\footnotesize{motocross2 \hfill( MB,AC)}         & \checkmark &  & \checkmark &  &  &  & \checkmark &  & \checkmark & \checkmark \\
\hline
\footnotesize{ball2 \hfill(MB,SB)}         & \checkmark & \checkmark &  &  &  &  & \checkmark & \checkmark &  & \\
\footnotesize{gymnastics1 (MB, SB)}     & \checkmark &  &  &  &  &  & \checkmark &  & \checkmark & \checkmark \\
\footnotesize{girl \hfill(OC)}            & \checkmark &  & \checkmark &  &  &  &  & \checkmark &  &  \\
\hline
\footnotesize{zebrafish1 \hfill(AC, SB)}      & \checkmark &  &  &  &  &  & \checkmark &  &  & \\
\footnotesize{fish2 \hfill(AC, SB)}         & \checkmark &  &  &  &  &  &  &  &  & \\
\footnotesize{frisbee \hfill(OC, AC)}         & \checkmark &  &  &  &  &  &  &  &  & \\
\hline
fps                            & 15 & 27 & 7 & 36 & 64 & 41 & 60 & 4.5 & 0.27 & 4.6 \\
\hline
\end{tabular}
\caption{Results on state-of-the-art trackers, especially C-COT~\cite{ccot} the most precise tracker of VOT2016 and VOT2017, and DaSiam~\cite{dasiamrpn} the champion of VOT2018 real-time competition, in tracking unsupervised sequences with different technical difficulties. KCF~\cite{kcf}, DSST~\cite{dsst}, and DaSiam~\cite{dasiamrpn} are real-time trackers. \checkmark means the tracker can track the whole sequence from start to end successfully. Running speed in VOT toolkit are measured under GTX 1070. }
\label{result_table}
\vspace{-0.15in}
\end{table*}

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.48\linewidth]{vot/vot_2016_m.png}
    \includegraphics[width=0.48\linewidth]{vot/vot_2017_m.png}
    \caption{Comparison results on unsupervised tracking on VOT2016 (left) and VOT2017 (right). Fast trackers with fps $> 10$ in VOT toolkit are tested. In the above plots, $x$-axis indicates overlap threshold and $y$-axis accuracy.}
    \label{fig:curve1617u}
\vspace{-0.2in}
\end{figure}

Apart from a good success rate, DAWN also has excellent performance in unsupervised tracking regarding accuracy per frame as the standard. As shown in the Figure~\ref{fig:curve1617u}, DAWN enjoys an overwhelming success compared with fast trackers running at fps $>$ 10 in VOT2016, and a performance better than most fast trackers running at fps $>$ 10 in VOT2017. DAWN can focus on not losing target and the result shows that it can keep track of many hard sequences that current SOTA cannot.

\begin{figure}[!h]
    \centering
    \includegraphics[width=0.48\linewidth]{vot/unsupervised_2016_c.png}
    \includegraphics[width=0.48\linewidth]{vot/unsupervised_2017_c.png}
    \caption{Unsupervised result of VOT2016 (left) and VOT2017 (right), where $x$-axis is overlap threshold and $y$-axis is accuracy.}
    \label{fig:curve}
\vspace{-0.2in}
\end{figure}

Further, we used DAWN as the pretrained model and trained the RPN branch, which was similarly done in training  SiamRPN~\cite{SiamRPN}. Then we tested DAWN-RPN on VOT2016 and VOT2017 using the VOT toolkit, and compare DAWN-RPN with DAWN and SiamRPN-- (which is our own implementation since the training code is not released). Figure~\ref{fig:curve} shows the consistent improvement  of DAWN-RPN over SiamRPN-- (and the original DAWN as well) in the unsupervised competitions in VOT2016 and VOT2017.

\subsection{Attention}

This section reports the comparison results between the attention module of DAWN and MemTrack evaluated under the VOT2016 dataset. Both models are trained on the VID dataset of ILSVRC~\cite{imagenet}.

\begin{figure}[htb]
    \centering
    \begin{subfigure}{0.98\linewidth}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/DAWN_hand12.jpg}
            \includegraphics[width=\linewidth]{attention/DAWN_hand96.jpg}

        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/memtrack_hand12.jpg}
            \includegraphics[width=\linewidth]{attention/memtrack_hand96.jpg}

        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/mavot_hand12.jpg}
            \includegraphics[width=\linewidth]{attention/mavot_hand96.jpg}

        \end{subfigure}
    \end{subfigure}
    \begin{subfigure}{0.98\linewidth}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/DAWN_book38.jpg}
            \includegraphics[width=\linewidth]{attention/DAWN_book167.jpg}
            \caption{DAWN}
        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/memtrack_book38.jpg}
            \includegraphics[width=\linewidth]{attention/memtrack_book167.jpg}
            \caption{Memtrack}
        \end{subfigure}
        \begin{subfigure}{0.32\linewidth}
            \includegraphics[width=\linewidth]{attention/mavot_book38.jpg}
            \includegraphics[width=\linewidth]{attention/mavot_book167.jpg}
            \caption{MAVOT}
        \end{subfigure}
    \end{subfigure}
    \caption{Results of two sample frames selected in \textit{hand} and \textit{book} respectively. Time axis is running downward.}
    \label{fig:handnbook}
\vspace{-0.25in}
\end{figure}

Figure~\ref{fig:handnbook} shows the respective attention of DAWN and MemTrack: the blue box is the ROI where the attention is computed, and the inner box (red) is the bounding box produced by the respective tracker. Within a given ROI, brightness indicates the scale of the attention: the brighter the pixel, the higher the attention at that pixel.

Observing the results on the two videos `hand' and `book', when tracking starts, MemTrack's attention is quite random and mostly focused on the background, causing it to lose track of the object in subsequent frames due to a background object with similar features (in video `hand' under severe motion blur both the face and hand are dominated by flesh color), or changes in the object's relative position on the background (in video `book'). On the other hand, DAWN produces a more precise attention, thus giving a stronger focus on the object in the early stage of tracking and making it less vulnerable to losing track in later frames. A stronger focus helps the model to be more robust against similar objects in video `hand', and a precise attention prevents the model from tracking background features.

\begin{figure}[htb]
    \centering
    \begin{subfigure}[b]{0.98\linewidth}
        \includegraphics[width=0.19\linewidth]{attention/DAWN_but1.jpg}
        \includegraphics[width=0.19\linewidth]{attention/DAWN_but20.jpg}
        \includegraphics[width=0.19\linewidth]{attention/DAWN_but25.jpg}
        \includegraphics[width=0.19\linewidth]{attention/DAWN_but43.jpg}
        \includegraphics[width=0.19\linewidth]{attention/DAWN_but57.jpg}
        \caption{DAWN}
    \end{subfigure}
    \begin{subfigure}[b]{0.98\linewidth}
        \includegraphics[width=0.19\linewidth]{attention/memtrack_but1.jpg}
        \includegraphics[width=0.19\linewidth]{attention/memtrack_but20.jpg}
        \includegraphics[width=0.19\linewidth]{attention/memtrack_but25.jpg}
        \includegraphics[width=0.19\linewidth]{attention/memtrack_but43.jpg}
        \includegraphics[width=0.19\linewidth]{attention/memtrack_but57.jpg}
        \caption{MemTrack}
    \end{subfigure}
    \caption{Attention in \textit{butterfly}.}
\vspace{-0.15in}
    \label{fig:butt}
\end{figure}

In `butterfly' (Figure~\ref{fig:butt}) a false initial attention causes  MemTrack  not moving at all, whereas DAWN with the proper initialization can readily track the object all along.

\begin{figure*}[thb]
    \centering
    \begin{subfigure}[b]{0.90\linewidth}
        \includegraphics[width=0.19\linewidth]{memory/com_fri003_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fri011_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fri015_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fri042_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fri072_m.jpg}
    \end{subfigure}
    \begin{subfigure}[b]{0.90\linewidth}
        \includegraphics[width=0.19\linewidth]{memory/com_girl099.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_girl108.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_girl117.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_girl126.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_girl135.jpg}
    \end{subfigure}
    \begin{subfigure}[b]{0.90\linewidth}
        \includegraphics[width=0.19\linewidth]{memory/com_gym086.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_gym100.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_gym388.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_gym392.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_gym403.jpg}
    \end{subfigure}
    \begin{subfigure}[b]{0.90\linewidth}
        \includegraphics[width=0.19\linewidth]{memory/com_god170.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_god196.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_god204.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_god210.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_god230.jpg}
    \end{subfigure}
    \begin{subfigure}[b]{0.90\linewidth}
        \includegraphics[width=0.19\linewidth]{memory/com_fish003_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fish030_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fish036_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fish044_m.jpg}
        \includegraphics[width=0.19\linewidth]{memory/com_fish072_m.jpg}
    \end{subfigure}
    \begin{subfigure}[b]{0.90\linewidth}
        \centering
        \includegraphics[width=0.9\linewidth]{memory/caption.pdf}
    \end{subfigure}
    \caption{From top to bottom: \textit{frisbee, girl,  gymnastics1,  godfather, zebrafish1}. }
    \label{fig:mem}
\vspace{-0.2in}
\end{figure*}

\subsection{Memory}
We present a number of examples to show how background memory can effectively address occlusion, confusing background features, multiple object instances and abrupt changes of appearance (Figure~\ref{fig:mem}).

Both partial and total occlusion occur in `frisbee' and `girl'. In both videos, DAWN's background memory module can suppress the occluder (i.e., the guy), allowing the original target to be correctly tracked even after total occlusion. In `godfather', the white veil and white hat distract the trackers without background memory. But DAWN still works well thanks to its effective background memory. Notice that due to abrupt changes of target appearance and similar background features, tracking in `zebrafish1' becomes an almost impossible task for most of the tested trackers, especially those with a fixed aspect ratio of the bounding box. However, with attention and dual memory, DAWN can robustly track the zebrafish, despite the low overlap ratio due to its fixed aspect ratio.

\begin{figure}[!h]
    \centering
    \includegraphics[width=0.14\linewidth]{vot/mem.png}
    \includegraphics[width=0.42\linewidth]{vot/mem_2016_m.png}
    \includegraphics[width=0.42\linewidth]{vot/mem_2017_m.png}
    \caption{Comparison result of average overlap with ground-truth in VOT2016 (left) and VOT2017 (right).}
    \label{fig:aeo}
    \vspace{-0.20in}
\end{figure}

\begin{table}[h]
  \centering
\begin{adjustbox}{max width=0.8\linewidth}
\begin{tabular}{l|ccc}
\hline
Tracker       & \tabincell{c}{Foreground \\ Memory}  & \tabincell{c}{Background \\ Memory} & Attention \\
\hline
DAWN           & \checkmark & \checkmark & \checkmark \\
MemTrack       & \checkmark &  & \checkmark \\
MAVOT          & \checkmark & \checkmark &  \\
SiamFC         &  &  &  \\
\hline
\end{tabular}
\end{adjustbox}
\caption{Model architecture comparison.}
\label{table:model}
\vspace{-0.1in}
\end{table}

\begin{table}[!t]
  \centering
\begin{adjustbox}{max width=0.8\linewidth}
  \centering
\begin{tabular}{l|cccc}
\hline
      Measures & SiamFC  & MemTrack & DAWN- & DAWN \\
\hline
EAO ($\uparrow$)           & 0.19 & 0.23 & 0.24 & 0.24 \\
Failures ($\downarrow$)      &
34.03 & 30.53 & 27.93 & 26.58 \\
\hline
\end{tabular}
\end{adjustbox}
\caption{Performance in VOT 2017, SiamFC: no memory, MemTrack: target memory, DAWN-: dual memory, DAWN: dual memory with improved attention.}
\label{table:ablation}\vspace{-0.2in}
\end{table}

Table~\ref{table:model} compares the attention and memory modules among memory-based trackers. Figures~\ref{fig:curve} and~\ref{fig:aeo} show our clear improvement over others while still running very fast. Tested in the VOT toolkit, DAWN runs at 15~fps, SiamFC~\cite{SiamFC} at 36 fps,  MemTrack~\cite{memtrack} at 27 fps, and MAVOT~\cite{mavot} at 7 fps. Table~\ref{table:ablation} shows that after adding background memory block and using our new attention scheme the success rates are improved.

\vspace{-0.1in}
\section{Conclusion}

We have presented a simple and yet effective memory-based approach for unsupervised video object tracking. While strikingly simple in implementation, attention LSTM and dual memory structures enable DAWN to track video objects in unsupervised manner with excellent performance and robustness under challenging scenarios: partial and total occlusion, severe motion blur, abrupt changes in target appearance, multiple object instances, and similar foreground and background features. We have presented extensive quantitative and qualitative experimental comparison: DAWN is ranked third in both VOT2016 and VOT2017 challenges with an excellent success rate among all fast trackers running at fps $>$ 10 and has great performance in unsupervised tracking in both challenges. We further showed using DAWN-RPN that state-of-the-art models can immediately benefit by simply augmenting them with the proposed dual memory and attention LSTM.

 \section*{Acknowledgement}
This research is supported by Tencent and the Research Grant Council of Hong Kong SAR under grant no. 16201818.

{\small
\bibliographystyle{ieee}


\section{Introduction}
\label{sec:introduction}

Supervised deep learning techniques have revolutionized the field of medical image segmentation~\cite{litjens2017survey}, particularly with fully convolutional neural network architectures such as the U-Net~\cite{ronneberger2015u}.
To learn these networks, a loss function $L$ is optimized using gradient based approaches and backpropagation. This function is usually defined in terms of metrics that quantify the discrepancies between a trustworthy/ground truth labelling and the predicted segmentation.

In this typical framework a loss function is not explicitly tailored to aim for a specific feature in the target space. Hence, the network firstly learns the dominating characteristics of the target images in the training set, and its remaining capacity is gradually devoted to characterize other less prevalent target features. This becomes an issue when dealing with highly pathological data, where lesions or disease appearance might significantly differ between patients. To overcome this limitation, some authors proposed to train segmentation models using a linear combination of different losses such as cross-entropy and Dice~\cite{khened2019fully}. However, these metrics are still computed from the same target representation, so they do not enhance a specific target feature. In this paper we propose to extend this idea by using the framework of \textit{augmented target loss functions}, introduced in~\cite{ortho}. Rather than relying on a single or a linear combination of loss functions defined on the original prediction and target space, Breger \textit{et al.}~\cite{ortho} proposed to compute the loss on alternative representations of the predictions and targets, obtained by applying differentiable transformations $T$ that enhance specific characteristics.

This paper focuses on the application of an augmented target loss function for photoreceptor layer segmentation in retinal optical coherence tomography (OCT) scans of patients with macular diseases. OCT is the state-of-the-art technique for imaging the retina, as it brings volumetric information through a stack of 2D scans (B-scans) at a micrometric resolution~\cite{SE2018ai}. Ophthalmic disorders such as diabetic macular edema (DME), retinal vein occlusion (RVO) and age-related macular degeneration (AMD) gradually affect photoreceptors while progressing. The abnormal accumulation of fluid due to these diseases significantly alters the retina, eventually leading to photoreceptor cell death. This last characteristic can be noticed through OCT imaging: first as a pathological thinning of the photoreceptor layer, and more lately as complete disruptions on it (Fig.~\ref{fig:custom-weighting}, right). It has been observed that these abnormalities are highly correlated with focal vision impairment~\cite{takahashi2016photoreceptor} and visual acuity loss when located at the central area of the retina~\cite{gerendas2017oct}. Hence, the automated characterization of the morphology of the photoreceptor layer is relevant for efficient quantification of functional loss.

In this paper we build on top of the architectural innovations proposed in~\cite{orlando2019u2net} by training such a model using an augmented target loss function. Fitting the framework we introduce a novel amplified-target loss that induces further penalization to errors within the central area of the B-scans. As the most challenging pathologies are usually observed at the central area of fovea-centered OCT scans, our hypothesis is that incorporating this loss function as a kind of regularizer enforces the network to better characterize disease appearance. We validate our approach using a series of OCT scans of patients with AMD, DME and RVO. Our results empirically show that the proposed loss functions improve the performance within the central millimeters of the retina compared to using traditional losses without compromising the performance in the entire volume.

\begin{figure}
    \centering
    \includegraphics[width=0.99\textwidth]{figures/example-of-loss_new.pdf}

    \caption{Left: scanning laser ophthalmoscopy (SLO) of a patient with RVO. The square indicates the area captured by the OCT volume and the rings represent the central subfield (CSF) and the 3 and 6 central millimeters (3 CMM and 6 CMM). The blue line highlights the B-scan showed in the right side. Right: CSF B-scan with photoreceptor layer annotation (green) with (a) disruptions and (b) abnormal thinning. The red heat map represents the weighting strategy applied in our loss function. The central coordinate of the image is indicated with the yellow dotted line, and a profile of the weighting strategy is illustrated on top of the B-scan. }
    \label{fig:custom-weighting}
\end{figure}

\section{Methods}
\label{sec:methods}
\subsection{Augmented target loss functions for image segmentation}
\label{subsec:variational-losses}

In a supervised learning problem we aim to learn a function $f$ with $f_\theta(x) \approx y$, where $\theta$ denotes the free parameters and $S = \{(x,y)^{(i)} \}, 1 < i < N$ is a given training set with pairs of inputs $x$ and ground truth labels $y$. In the context of image segmentation, $x$ corresponds to an input image, $y$ and $\hat{y}$ are manual and predicted segmentations and $f_\theta$ is some segmentation model (e.g. a fully convolutional neural network such as the U-Net~\cite{ronneberger2015u}).

To adjust the weights $\theta$ from the chosen network structure $f_\theta$, a loss function $L$ is minimized using gradient based optimization. $L$ is a piecewise differentiable loss function, e.g. cross-entropy (CE) or mean squared error (MSE), that measures the pixel-wise differences between $\hat{y}$ and $y$. In standard settings no specific areas of the images are penalized more than others. Thus, the parameters $\theta$ are mostly adjusted to characterize those features from the training set that have the most impact on the overall loss. Although this might be helpful to segment healthy anatomy, in pathological scenarios the network will overfit the prevalent features unless explicit regularization is imposed during training.

Here, we propose to use the framework of augmented target (AT) loss functions, introduced in \cite{ortho}. These losses take into account prior knowledge of target characteristics via error estimation in transformed target spaces. The framework can be applied to any supervised learning problem based on loss optimization where additional information about the target data is available, provided it can be formulated as a transformation function $T$. The transformation may correspond to any piecewise differentiable function on the target space that yields a more beneficial representation of some known target characteristic.

Following \cite{ortho}, the AT loss functions $L_\text{AT}$ is a linear
combination of losses applied to transformed targets. Its general form is:
\begin{equation}\label{atloss}
L_\text{AT} = \sum_{j=1}^d \lambda_j \cdot L^j \big(\{T_j(y_i)\}_i^{},\{T_j(\hat{y}_i)\}_i^{} \big),
\end{equation}
where $\lambda_j > 0$ corresponds to some weight, $T_j$ to a specific transformation and $L^j$ to some loss function, for all $j \in \{1,\dots,d\}$.

Setting typically $T_1$ to the identity and $L^1$ to a standard loss, the additional terms in the $L_\text{AT}$ loss act as
 amplified target information, yielding a new optimization problem:
\begin{equation}\label{eq:optimization}
    \hat{\theta} = \argmin_\theta \{ \lambda_1 \cdot L_1\big(\{y_i\}_i^{},\{\hat{y}_i\}_i^{}\big) + \sum_{j=2}^d \lambda_j \cdot L^j \big(\{T_j(y_i)\}_i^{},\{T_j(\hat{y}_i)\}_i^{}\big),
\end{equation}
where the weights $\lambda_1$ and $\{ \lambda_j \}_{j=2}^d$ control the balance between the main loss and the regularization terms respectively.

\subsection{Amplified-target loss functions for photoreceptor layer segmentation}
\label{subsec:our-loss}

We experimentally study the AT loss function framework in the context of photoreceptor layer segmentation in pathological OCT scans. We tailor a so called \textit{amplified-target loss} in which a transformation $T$ is designed to bring an increased penalty to errors within the central area of the images. This loss is intended to incorporate the prior knowledge that abnormalities such as pathological thinnings and disruptions of the photoreceptor layer are more common in the central millimeters of the foveal area. To do so, we define a transformation $T(y_i) = W \odot y_i $, where $\odot$ denotes the Hadamard (entrywise) product, $y_i$ corresponds to the given binary targets and $W$ represents a weighting matrix that encodes a penalization weight for errors. This operation can analogously be applied to the predictions $\hat{y}_i$. Fig.~\ref{fig:custom-weighting} graphically illustrates the design of the weighting matrix $W$. Formally, we define $W = G_\sigma * V$, where $G_\sigma$ stands for a Gaussian filter with standard deviation $\sigma$. We define $V$ as:
\begin{equation} \label{eq:weighting}
    V_{j,i} := \left\{
        \begin{array}{ll}
            \omega & \qquad \text{for } i_0 < i < i_1 \text{ and all } j, \\
            1 & \qquad \text{otherwise,}
        \end{array}
        \right.
\end{equation}
 where $\omega$ denotes the maximum weight assigned to the central area and $[i_0, i_1]$ is the horizontal interval of the image that is amplified. The Gaussian filter $G_\sigma$ is used to smooth the penalization factor within the edges of the interval.

Following the formulation in (\ref{eq:optimization}), we can then redefine our empirical risk minimization problem as
\begin{equation}
\hat{\theta} = \argmin_\theta \{ \lambda_1 \cdot L^1  \big(\{y_i\}_i^{},\{\hat{y}_i\}_i^{}\big) + \lambda_2 \cdot L^2 \big(\{ W \odot y_i \}_i^{}, \{W \odot  \hat{y}_i \}_i^{}  \big) \},
\end{equation}
where we choose $\lambda_1, \lambda_2  \in \mathbb{R}$ and $L^1 = L^2$ as CE or MSE losses.

\section{Experimental setup}
\label{sec:experimental-setup}

\subsection{Materials}
\label{subsec:materials}

Our method was trained and tested on an in-house data set with 53 Spectralis OCT volumes of patients suffering from DME (16), RVO (27) and AMD (10). Each image comprises $496 \times 512$ pixels per B-scan, 49 B-scans per volume. All the B-scans were manually annotated by certified readers under the supervision of a retina expert, who modified the labels when necessary to obtain ground truth segmentations. The set was randomly divided into a training, a validation and a test set, each of them with 34, 4 and 15 scans, respectively, with approximately the same distribution of diseases and percentages of disrupted columns per B-scan (or A-scans).

\subsection{Network architecture and training setup}
\label{subsec:training-setup}

We used the photoreceptor segmentation network described in~\cite{orlando2019u2net} in our experiments (note that any other architecture could be applied within our framework). We used as baselines CE and MSE comparing it to the adapted AT loss.

Every configuration was trained at a B-scan level with a batch size of 2 images, using Adam optimization and early stopping. Hence, training was stopped if the validation loss did not improve for the last 45 epochs. The learning rate was set to $\eta = 0.0001$, and divided by 2 if the validation loss was not improved during the last 15 epochs. Data augmentation was used in the form of random horizontal flippings. Binary segmentations were retrieved as in~\cite{orlando2019u2net} by thresholding the softmax scores of the photoreceptors class using the Otsu algorithm.

\section{Results and Discussion}
\label{sec:results}

We evaluated the performance for segmenting the photoreceptor layer using the volume-wise Dice index, at the CSF, the 3 CMM, the 3-1 ring and the full volume (Fig.~\ref{fig:custom-weighting}, left). All the experiments with our AT loss functions were conducted using fixed values for $\sigma=\frac{1}{16}X$, $i_0=\frac{1}{4}X$ and $i_1=\frac{3}{4}X$ (with $X=512$ being the horizontal size of the B-scans, in pixels), without optimizing them on the validation set. Different configurations for $\omega = 2^k, k \in \{1, ..., 5\}$ and $\lambda_1$ and $\lambda_2 \in \{ 0.001, 0.01, 0.1, 1, 2, 4, 8$ \} were analyzed, and the best configuration according to Dice index on the validation set was then fixed to allow a fair comparison on the test set. From this model selection step, we observed that $\omega=8$, $\lambda_1=1$ and $\lambda_2=8$ reported the best performance for the AT loss with categorical cross-entropy (CE), and $\omega=32$, $\lambda_1=\lambda_2=1$ for the AT loss with mean square error (MSE).

\begin{figure}[t]
    \centering
    \subfigure[Cross entropy]{\includegraphics[width=0.45\textwidth]{figures/boxplot_cross-entropy_Dice.pdf}}
    \subfigure[MSE]{\includegraphics[width=0.45\textwidth]{figures/boxplot_mse_Dice.pdf}}
    \caption{Volume-wise Dice values for all the evaluated models and our proposed approach in each evaluation area. Circles indicate mean values. CSF: central subfield (1 central millimeter). 3 CMM: three central millimeter. 3-1 ring: area between CSF and 3 CMM.}
    \label{fig:quantitative-results}
\end{figure}

Fig.~\ref{fig:quantitative-results} depicts boxplots with the quantitative performance of each model on the test set, compared with their corresponding baselines trained only with CE and MSE, for each evaluation area. The mean and standard deviation values of the Dice index are presented in Table~\ref{tab:table-dice}. The incorporation of the AT loss allows to perform consistently better in all the cases, with the best results reported by the MSE loss. Statistical analysis using one-tail Wilcoxon sign-rank tests at a significance level $\alpha = 0.05$ showed that the model trained with MSE + AT loss reported significantly higher Dice values in the CSF area compared to using CE + AT loss or only MSE ($p < 0.0171$). These differences were not statistical significant with respect to the model trained with CE ($p = 0.1902$). When comparing the Dice values at the 3-1 ring, the MSE with AT loss model reported statistically significant better results than using only CE or MSE ($p < 0.0042$), which is consistent with its behavior in the 3 CMM ($p < 0.0416$). No statistically significant differences in performance were observed at the full volume level (two-tails test, $p > 0.0730$).

\begin{table*}[t]
\centering
\caption{Volume-wise mean $\pm$ standard deviation Dice values in the test set for each photoreceptor segmentation model and the different areas.}
\resizebox{\textwidth}{!}{
\begin{tabular}{C{4.5cm} | C{1.9cm} | C{1.9cm} | C{1.9cm} | C{1.9cm}}
\hline
\textbf{Method} & \textbf{CSF} & \textbf{3 CMM} & \textbf{3-1 ring} & \textbf{Full volume} \\
\hline
\hline
CE loss &	0.622 $\pm$~0.271	 &	0.691 $\pm$~0.242	 &	0.708 $\pm$~0.242 & 0.820 $\pm$~0.118 \\
\hline
CE + AT loss (CE,~$\omega=8$,~$\lambda_1=1$,~$\lambda_2=8$) & 	\textbf{0.656 $\pm$~0.256}		& \textbf{0.718 $\pm$~0.218}	&	\textbf{0.732 $\pm$~0.218}& \textbf{0.828 $\pm$~0.100} \\
\hline
\hline
MSE loss	& 0.560 $\pm$~0.303 &		0.707 $\pm$~0.223	 &	0.727 $\pm$~0.223 & \textbf{0.835 $\pm$~0.096}\\
\hline
MSE + AT loss (MSE,~$\omega=32$,~$\lambda_1=\lambda_2=1$)	& \textbf{0.708 $\pm$~0.254}	&	\textbf{0.749 $\pm$~0.215}	&	\textbf{0.760 $\pm$~0.213} & 0.821 $\pm$~0.102\\
\hline
\end{tabular}}
\label{tab:table-dice}
\end{table*}

We qualitatively analyzed the segmentation and score maps  using the CE and MSE combined with the AT loss. Fig.~\ref{fig:score-maps} depicts segmentation results in a central B-scan from the test set, with score maps represented as heatmaps. Using MSE produces noisy scores within the lateral areas of the B-scans, and therefore spurious elements in the segmentation. CE, on the contrary, results in smoother score maps, although with few false negatives in the vicinity of subretinal fluid. This behavior is linked to the one observed in Table~\ref{tab:table-dice}, where the MSE + AT loss model reported higher Dice in the central area than using CE, and smaller values in the full volume. The model trained with only MSE performs poorly in the CSF, the 3 CMM and the 3-1 ring, which indicate that it struggles to deal with pathologies. Similarly, the high performance at a volume level indicates that it can better characterize normal appearances. When using MSE + AT loss, a significant reduction in the amount of false negatives occurs at the central areas. However, as mentioned before, the score maps are noisy at the borders of the B-scans, which causes a drop in the full volume Dice. The model trained with CE + AT loss is less accurate at the center than the one trained with MSE + AT loss, but it still outperforms the baseline approaches. Moreover, at a volume basis the CE + AT loss remains competitive with respect to the one trained only with CE loss.

Finally, Fig.~\ref{fig:qualitative-results} presents qualitative results in exemplary central B-scans from our test set obtained both by the models trained with CE only and  with CE + AT loss. Our approach produced more anatomically plausible segmentations than the standard CE loss in pathological areas with subretinal fluid (Fig.~\ref{fig:qualitative-results} (a) and (b)) or large disruptions (Fig.~\ref{fig:qualitative-results} (c)).

\begin{figure}[t]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/score-maps.pdf}
    \caption{Qualitative effect of the loss selection in the pixel score values. From left to right: manual annotation (green), score map (orange) and binary segmentation (yellow) obtained with MSE + AT loss (MSE, $\omega=32, \lambda_1=\lambda_2=1$) and CE + AT loss (CE, $\omega=8$, $\lambda_1=1$, $\lambda_2=8$).}
    \label{fig:score-maps}
\end{figure}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.98\textwidth]{figures/qualitative.pdf}
    \caption{Qualitative results in central B-scans from the test set. From left to right: manual annotations (green), results with only CE loss (blue) and results with CE + AT loss (CE, $\omega=8$, $\lambda_1=1$, $\lambda_2=8$).}
    \label{fig:qualitative-results}
\end{figure}

\section{Conclusions}
\label{sec:conclusions}

In this paper we proposed to use the framework of augmented target loss functions for photoreceptor layer segmentation in pathological OCT scans. We define an amplified-target loss incorporating a transformation that weights the central area of the input B-scans to further penalize errors committed in this region. We experimentally observed that this straightforward approach allows to significantly improve performance within the central millimeters of fovea-centered OCT scans, without affecting the overall performance in the entire volume. These results indicate that the proposed AT loss function acts as a form of regularization, better characterizing photoreceptors appearance within highly pathological regions. We are currently exploring new alternatives to identify the regions to weight and to learn their corresponding weights. Further experiments are also performed to evaluate our approach in the context of other OCT based applications such as fluid segmentation and using OCT scans from other vendors.

\noindent\\
\textbf{Acknowledgements}\\
This work is funded by WWTF AugUniWien/FA7464A0249 (MedUniWien); VRG12-009 (UniWien). We thank NVIDIA Corporation for donating a GPU.


\section{INTRODUCTION}
\label{sec:intro}
The study of exceptional points (EPs) has revealed a variety of fundamental phenomena and spawned next generation technological developments \cite{FGG17,GMKMRC18,MA19}. Examples range from hypersensitive gyroscopes and bio-sensing \cite{HHWGGCK17,M17,COZWY17,W14} to lasing control \cite{HMHCK14,FWMWZ14,HMHHHCK16} and unidirectional invisibility \cite{L11,P14}. Most studies have been performed in the photonics framework \cite{FGG17,GMKMRC18,MA19}, and only few utilized other areas like electronic circuitry \cite{SLLREK12,AYF17,LCTEK18}, acoustics \cite{SDCCRWZ16,FSA16,AP17} and atomic physics \cite{PCSQWJX16, HH17}. Here we develop a platform where EPs are realized by coupling together, via an elastic plate that emulates a complex environment, two identical elastic resonators with differential loss. In contrast to previous studies, the formation of the EPs, are manipulated via a set of ``environmental" plate modes that indirectly control the coupling between the two resonators. Our work paves the way for exciting applications in micro-electromechanical device engineering, were the hypersensitive nature of EPs can be utilized for monitoring structural integrity of surfaces or for realizing a new family of double-cantilevered atomic force microscopes.

\section{EXCEPTIONAL POINTS FROM PARITY TIME SYMMETRY}
\label{sec:ptep}
Theoretically discussed more than fifty years ago \cite{kato66}, EPs are non-Hermitian degeneracies associated with the coalescence of two eigenvalues, and their corresponding eigenvectors \cite{heiss12}. The simplest example is a parity-time (${\cal PT}$)-symmetric dimer \cite{BB99} consisting of two identical oscillators of mass $m$ and spring constant $k_0$, having resonance frequency $\omega_0 =\sqrt{k_0/m}$. The oscillators are coupled together with a spring with Hooke's constant $k$, see Fig.~\ref{fig1}(a). Each oscillator is equipped with opposing power flow, a gain and loss of strength $\gamma = b/(m \omega_0)$ expressed in terms of an equivalent linear drag and anti-drag forces $f_\mp = \mp bv$ where $v$ is the speed of each oscillator and $b$ the drag coefficient. Such a system is not invariant under time-reversal (${\cal T}$) i.e. inversion of the flow of time $t\rightarrow -t$. It also violates parity symmetry, i.e., it is not invariant under spatial reflections $x \rightarrow -x$. Instead, its equations of motion are invariant under joint ${\cal PT}$-symmetry for all $\gamma$-values.
\begin{figure*}
  \includegraphics[width=\hsize,keepaspectratio]{Fig1}
\caption{(a) A simple ${\cal PT}$-symmetric dimer made by two identical coupled oscillators, with one of them (green) experiencing energy attenuation and the other (red) energy amplification. (b) COMSOL simulations of the parametric evolution of the eigenfrequency spectrum of an elastic dimer system consisting of two pillars formed in close proximity to a base. The two pillars are located symmetrically along the long axis of a $20 cm$ by $12.8 cm$ plate, $1 cm$ thick. Each pillar has a $4 cm$ height with a square cross section $2 cm$ on a side, with their centers displaced by $4 cm$. The varied parameter is the radius
$r_{\rm hole}$ of a semicircular indent at a position that preserves the mirror symmetry of the structure. The gain/loss parameter $\gamma$ is zero. The set of modes that are investigated in the presence of gain/loss contrast are highlighted with a green ellipse. At the insets we show the numerically evaluated (using COMSOL) corresponding eigenmodes. (c) The real and imaginary parts of the eigenfrequencies for the fundamental torsion modes, corresponding to $r_{\rm hole}=0$, are shown vs. the gain/loss parameter $\gamma$. (d) The same for the three nearby modes associated with $r_{\rm hole}\approx3.8cm$. (e) The same as previously but for the eigenmodes of the effective Hamiltonian Eq. (\ref{3by3}). The dimensionless parameters used are $\kappa=6.14 \times 10^{-3}, \lambda_1=-0.9\times10^{-3}=-\lambda_2, \nu=-2\times 10^{-3}$ while $\omega_0=2\times \pi\times16.290\times 10^3 Hz$.
}
\label{fig1}
\end{figure*}

This can be seen in the equations of motion, which in the frequency domain ($e^{i \omega t}$) are given by
\begin{equation}
\begin{pmatrix}
1 + \kappa - iu\gamma - u^2 & -\kappa \\
-\kappa & 1 + \kappa + iu\gamma - u^2
\end{pmatrix}
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix}
=0
\label{mpt_dimer}
\end{equation}
where $u=\omega/\omega_0$ is the frequency relative to the isolated oscillator frequency, and $\kappa = k/k_0$ measures the strength of the inter-oscillator coupling spring. The loss (drag) in Eq.(\ref{mpt_dimer}) is applied to particle 1 and the gain (an anti-drag) is applied to particle 2, evident by the signs of the respective terms involving the gain/loss parameter $\gamma$ defined earlier. The ${\cal PT}$-symmetry is evident from the parity interchange operation (switch the rows and columns) and the time reversal operation (change the sign of $i$) which together return to the same relations.

We make a connection to coupled mode theory (CMT) by casting the equations of motion for this ${\cal PT}$-dimer into a Liouvillian-like eigenvalue form
\begin{equation}
\Omega \psi = \omega \psi,\quad
\Omega = \omega_0
\begin{pmatrix}
1 +{\kappa - i \gamma\over 2}  & -{\kappa \over 2} \\
-{\kappa \over 2} & 1 +{\kappa + i \gamma \over 2}
\end{pmatrix}
\label{std_pt}
\end{equation}
where $\psi$ is a vector of oscillator amplitudes. This form assumes weak coupling so that $u \approx 1$ and greatly simplifies the analytic description, with the factor of two originating from the epansion of $1-u^2$. (see the Appendix)

The eigenfrequencies and eigenmodes of this system are
\begin{multline}
\hspace{5mm}
\omega_\pm=1+{\kappa\over 2}\mp {1\over 2}
\sqrt{\kappa^2-\gamma^2}
\\
\text{and }\hspace{5 mm} \psi_\pm=\left(\frac{i\gamma\pm \sqrt{\kappa^2-\gamma^2}}{\kappa}, 1\right)^T
\label{d_loss}
\end{multline}
respectively. In the {\it exact phase} $\gamma<\kappa\equiv\gamma_{\rm EP}$, the coupling between the spatially separated gain and loss elements is capable of exactly communicating a balanced flow of energy: the modes have real eigenvalues and identical, steady-state oscillatory magnitudes. In this phase the eigenvectors $\psi_{1,2}$ are also eigenvectors of the ${\cal PT}$-operator. In the
{\it broken phase} $\gamma>\gamma_{\rm EP}$, the flow of energy overwhelms the coupling and effectively decouples the two oscillators into modes that are ${\cal PT}$ images, $\psi_2={\cal PT}\psi_1$, with a single real frequency and conjugate imaginary parts. The $\psi_2 (\psi_1)$ mode, describing displacements predominantly on the gain (loss) side, grow (decay) in time. The exact and broken ${\cal PT}$-symmetry phases are separated by a sharp transition -- the exceptional point (EP) -- where the eigenvectors $\psi_{1,2}$ coalesce and the frequencies merge with a characteristic square-root singularity $\Delta\omega \sim \sqrt{\gamma_{\rm EP}-\gamma}$. The scenario is generic and applies to other anti-linear symmetries ${\cal LT}$, where ${\cal L}$ is not necessarily the parity ${\cal P}$ but any other linear operator such as mirror symmetry ${\cal M}$ or rotation symmetry ${\cal R}$ \cite{BB99,B07,M11}.

\section{ENVIRONMENTAL MODES}
\label{sec:env}
In our study, each oscillator of the dimer is taken as the fundamental quarter-wave torsion resonance of an elastic pillar with one free end and one fixed end, see insets of Fig.~\ref{fig1}(b). The torsion pillars are formed in close proximity on a base composed of the same elastodynamics medium (aluminum). It has to be stressed that the complexity of the actual system is far larger than the simple coupled oscillators discussed above. Even the one-dimensional approximation of elastic pillars at long wavelengths allows for three distinct modes: bending, compression, and torsion. In three-dimensions, any free surface invokes hybridized bulk modes \cite{graff91}. Here we focus our interest on the lowest torsional pillar modes to avoid the most obvious couplings to out-of-plane plate modes.

When two similar pillars are brought in the proximity of one another, communication of displacement fields through the not-perfectly-rigid base couples the two resonators, lifting their degeneracy. The frequency splitting is an indication of the coupling strength, $\kappa\approx\Delta f\approx 45Hz$ (see green highlight at the left side of Fig.~\ref{fig1}(b). The emerging pair is associated with a hard (out of phase) and soft (in phase) torsional supermodes respecting a ${\cal MT}$ symmetry of the total structure, seen highlighted in green on the left axis of Fig.~\ref{fig1}(b).

Unlike the simple coupled oscillator model, the pillar-pillar coupling involves {\it only} interactions with the modes of the base (environmental modes) leading to far more reaching consequences. In our simulations, we purposely manipulate the influence of these environmental levels to a dramatic degree by introducing a semicircular indent of radius $r_{\rm hole}$ at a position that preserves the mirror symmetry of the structure. We note that such configurations (known as Sinai billiards) lead to chaotic dynamics in the classical (particle) limit with direct consequences to the levels and eigenmodes of the wave system \cite{levels}. A typical signature of chaoticity in the wave mechanics framework is the formation of avoided crossings between nearby levels \cite{levels} (level repulsion) as $r_{\rm hole}$ increases, see Fig.~\ref{fig1}(b).

We use COMSOL Multiphysics\cite{comsol} to investigate of the system as a function of the indent size, and inspect the behavior at key values of $r_{\rm hole}$ as a function of a balanced gain/loss contrast $\gamma$ in the pillars by imposing opposite imaginary parts to their shear modulus i.e. $G=G_0 (1 \pm i\gamma)$ where $G_0=25 GPa$ is the shear modulus for aluminum. In the case of a simple two-level interaction (e.g. at $r_{\rm hole}=0$ -- see left level pair in Fig.~\ref{fig1}(b) -- the normal modes approach one another as $\gamma$ increases, and eventually coalesce at some $\gamma_{\rm EP}$ which is determined by the coupling strength $\kappa$ between the two torsional modes, see Fig.~\ref{fig1}(c). The symmetry-violation scenario is the one common to the ideal ${\cal PT}$-oscillator model, Fig.~\ref{fig1}(c) as well as the coupled mode theory model of Eq.~\ref{std_pt}.

\begin{figure*}
  \includegraphics[width=\hsize,keepaspectratio]{Fig2}
\caption
{(a) Photograph of the experimental set-up. The pillar dimer with its plate environment is machined from a single piece of 6061 aluminum. The two pillars are located symmetrically along the long axis of a $20 cm$ by $12.5 cm$ plate, $1 cm$ thick. Each pillar has a $4 cm$ height with a square cross section $2 cm$ on a side, with their centers displaced by $4 cm$. Weakly-coupled piezoelectric transducers allow elastic-wave transmission to be measured by a Vector Network Analyzer (VNA in the figure) through transformers. Plate-to-plate, or pillar-to-pillar transmission can be chosen to explore various modes with differing sensitivity. The lower right inset shows a detail of transducers placed to couple predominantly to the torsion modes. (b) Transmission spectrum of the two pillars structure for different transducer configurations. Of the many modes present, this frequency range spans the two coupled torsional modes at approximately 16.6 kHz and the two closest plate modes on either side. The relative surface displacements are also shown with violet to red spanning zero to a linear maximum. The black solid line shows the base-to-base transmission and the blue and red lines show the inter- and intra-pillar transmissions respectively. The symmetry of the nearest modes illustrates that the inter-pillar coupling must be mediated by other modes, further away in frequency.
}
\label{fig2}
\end{figure*}

While the two-level interaction physics is typically captured by the phenomenological system of Eq. (\ref{std_pt}), a three-level (or more) interaction can capture features inherent to an environment without any obvious analogue in the coupled oscillator system. We have identified in our simulations such a three-level interaction scenario -- see the encircled three levels at $r_{\rm hole} \approx 3.8cm$ in Fig.~\ref{fig1}(b). In this case, the gain/loss contrast couples the upper level pair having frequency separation (at $\gamma=0$) $\Delta f\approx 115Hz$ larger than $\Delta f\approx 88Hz$, associated with the lower two levels, see Fig.~\ref{fig1}(d). The three-level EP-scenario can be modeled using a $3\times 3$ CMT Hamiltonian
\begin{equation}
\Omega \psi = \omega \psi,\quad
\Omega =\omega_0
\begin{pmatrix}
1+i\gamma  & -\kappa  & \lambda_1\\
-\kappa  & 1 - i \gamma&\lambda_2\\
\lambda_1  & \lambda_2&1+\nu\\
\end{pmatrix}
\label{3by3}
\end{equation}
where $\kappa$ describes generic coupling, associated with a background sea of spectrally distant base modes between the two upper levels. The third level, having relative frequency detuning $\nu$ (relative to $\omega_0$), describes a particular environmental mode which has notably significant spatial {\it and spectral} overlap with the two torsional modes. In the most general case, each of the two torsional levels interact in a different manner with the environmental level. These interactions are described by the coupling constants $\lambda_{1,2}$ and their influence in the three-level EP formation is demonstrated in Fig.~\ref{fig1}(e). For an appropriate choice of the CMT parameters, most notably $|\lambda_2|=|\lambda_1|$, we observe an EP formation which is qualitatively the same as the one found in Fig.~\ref{fig1}(d).

We also note that either $\kappa$ or $\lambda$ alone ensure an exceptional point: we include both to accommodate a specific interaction strength for the interfering mode ($\lambda$) distinct from the environmental sea ($\kappa$) of other modes. Their values allow independent control of the relative separation of three modes away from the exceptional point. The upward skewing of the COMSOL result near the exceptional point cannot be captured by the three-level model with constant coupling terms.

\section{EXPERIMENTAL EXEPTIONAL POINT}
\label{sec:eep}

The implementation of non-Hermitian gain and loss mechanisms for the realization of EPs can be achieved via piezo-electric elements attached to each of the pillars. An alternative, experimentally simpler approach, is to introduce controllable differential loss externally applied to one of the pillars. Its downside, however, is that such differential loss configurations require a greater attention in the design of the structure. Specifically, a weak overlap of the environmental modes with the torsional modes of the pillars has to be engineered, thus enforcing the {\it weak} pillar-pillar coupling regime necessary for the realization of EP singularities. In fact, it can be shown (see the Appendix) that in this limit, the CMT model has essentially the same form as Eq.~\ref{std_pt} with the loss terms unequal and both positive:
\begin{equation}
\Omega \psi = \omega \psi,\quad
\Omega = \omega_0
\begin{pmatrix}
1 +{\kappa + i \gamma_d\over 2}  & -{\kappa \over 2} \\
-{\kappa \over 2} & 1 +{\kappa + i \Gamma \over 2}
\end{pmatrix}.
\label{difep}
\end{equation}
In the symmetric phase, the mean loss is shared equally by both modes (identical imaginary parts, rather than zero) whereas the broken phase modes essentially decay according to their individual damping, with $\Gamma$ the externally imposed damping and $\gamma_d$ the intrinsic damping.

Having designed the experimental pillar/base system (in the absence of the indent, see Fig.~\ref{fig2}(a) to assure the {\it weak coupling limit}, small piezoelectric transducers are carefully attached (see Methods) to various positions in order to study the transmission spectrum through different paths. A network analyzer excites one of the transducers and receives the signal transmitted to a second transducer, both through audio transformers to improve the transducer impedance matching to the analyzer.

Fig.~\ref{fig2}(b) illustrates the transmission spectrum through various paths. Note that the curves in the figure indicate that all modes are only weakly hybridized, in contrast to the modes in Fig.~\ref{fig1}(b) with a large cut-out. The torsion mode doublet is observed in the plate transmission, and the plate modes are observed in the pillar-pillar torsion transmission, while their oscillator strengths are appropriately emphasized. The figure encompasses the torsion doublet ($\approx16.6 kHz$) along with the closest plate modes (15.6 kHz and 17.0 kHz) on either side, accompanied by insets showing their surface displacements. These plate modes correspond to the closest modes of Fig.~\ref{fig1}(b) at $r_{hole}=0$, the lower connecting to the mode seen rising into the graph at $r_{hole} \approx 3.8$.

\begin{figure}
\includegraphics[width=8 cm]{Fig3}
\caption
{
(a) Experimental measurement of the frequency shift and linewidth resulting from varying thicknesses of putty applied uniformly over the whole top surface of a pillar. This illustrates that for thicknesses over approximately $2mm$, the the damping becomes independent of the thickness and the frequency shift due to mass loading is effectively zero due to the nature of the complete boundary layer. The solid lines are fits to the simple linear viscoelastic model shown in the inset. (b) Image of the putty application creating the incremental loss for the exceptional point data presented in Fig.~\ref{fig4}. The imposed damping was directly measured and the frequency shift was confirmed to be proportionally consistent with the results for the thick uniform layers.
}
\label{fig3}
\end{figure}

The differential loss is physically introduced by small putty balls pressed into the top of one of the pillars. At 16 kHz, materials can interact with a shearing motion of the substrate (the pillar top) through a fairly thin boundary layer that adds dissipation proportional to the area of contact without a significant mass loading. Fig.~\ref{fig3}(a) shows experimental confirmation of this phenomenon, where the single pillar resonance was measured as the complete top of one the pillars was subjected to a uniform layer of increasing thickness with the other pillar disabled. The points show the measurements at different layer thicknesses and the lines show the fit to the model shown in the inset. In blue is the frequency shift experienced by the pillar, while the red shows the dissipative linewidth in the same units. Thickness dependence occurs only when the boundary layer is not contained within the putty layer, and the observed lack of mass loading under these circumstances requires that the boundary layer also not be heavily over-damped so that out-of-phase motion in the complete boundary layer nearly cancels the mass loading. Details of the model are included in the Appendix.

With one pillar fixed at its weakly damped intrinsic aluminum loss $\gamma_d$, the additional differential loss was varied by placing small (though larger than the boundary layer) putty balls incrementally to the top of the other pillar. For each increment, the damping was first calibrated by direct measurement of the loss factor $\Gamma$ of the damped pillar with the other pillar's resonance temporarily moved out of the picture by a clamp, see Fig.~\ref{fig3}(b). With the clamp removed, the experimental transmission spectra from the weakly damped side to the damped side was measured, and fit to the function
 \begin{multline}
t(u)=t_0+ \\
\frac{\gamma_d A e^{i\phi}}{(1-u^2+\kappa+iu\gamma_d)-\kappa^2/(1-u^2+\kappa+\epsilon+iu\Gamma)}
\label{transmit}
\end{multline}
describing the expected for an ideal pair of coupled oscillators, driven on the weakly damped side and sensed on the differential loss side. The parameter $\epsilon$ was included as an additional confirmation of the small size of the mass loading. The eigenfrequencies, extracted from the equivalent un-driven modes using the same parameters are reported with blue dots in Fig.~\ref{fig4}(a). In the same figure we also compare with the numerically calculated eigenmodes of this pillar system using COMSOL simulations. In these simulations we have used as only other input the independently determined experimental loss $\Gamma$ for the loss-pillar and one overall frequency shift adjusted via the exact choice of shear modulus. Poisson's ratio was previously fixed to best match the inherently damped pillar and plate modes of Fig.~\ref{fig2}. The data after the EP is unable to significantly resolve the linewidth of the broader resonances due to their merger into the background of the other modes. A further analysis of the eigenfrequency behavior around the EP is shown in Fig.~\ref{fig4}(b) where we plot the eigenfrequency difference versus the loss parameter $\Gamma$. By considering the logarithmic behavior of this curve, we find that the frequency splitting $\Delta f$ near the exceptional point scales as $\Delta f\sim \sqrt{1-\Gamma/\Gamma_{\rm EP}}$, thus confirming the existence of an EP singularity at a critical value $\Gamma_{\rm EP}$ in the case of {\it weak coupling}.

\begin{figure}
\includegraphics[width=8 cm]{Fig4}
\caption
{(a)Numerical calculations (black dots) and experimental data (blue dots) of the real and imaginary parts of the eigenfrequencies of the two pillars oscillating in their first torsional modes as a function of the loss parameter $\Gamma$. The experimental results
are in a good agreement with the numerical simulation. In the same subfigure we report the modes of the system for two typical cases of the differential loss $\Gamma$. (b) Log-log plot of the measured frequency splitting $\Delta f$ versus $1-\Gamma/\Gamma_{\rm EP}$ near the exceptional point. The red solid line shows the square-root law $\delta f\sim \sqrt{1-\Gamma/\Gamma_{\rm EP}}$ while the black dashed line is the best linear fit $\left(1-\Gamma/\Gamma_{\rm EP}\right)^{\alpha}$ with $\alpha\approx 0.6$.
}
\label{fig4}
\end{figure}

\section{CONCLUSIONS}
\label{sec:conc}
In this work we have investigated an aluminum double-torsion-pillar system that demonstrates an EP behavior in the elastodynamic regime for a system where the inter-oscillator coupling is completely mediated by environmental modes. We show that in the case of differential losses, a necessary condition for the emergence of EPs is a weak pillar-pillar base-mediated coupling. Our work provides a new versatile platform for the study of non-Hermitian wave physics and opens up new possibilities in engineering EP singularities through judicious manipulation of background environmental modes.

\section*{ACKNOWLEDGEMENTS}
\label{sec:ackn}
We acknowledge partial support from an NSF CMMI-1925530;1925543 grants (RT, FME, TK) and from ONR N00014-16-1-2803 and AFOSR via MURI Grant No. FA9550-14-1-0037 (VDR, TK).

\section*{APPENDIX}
\label{sec:append}
{\bf Coupled mode theory of classical mechanical oscillators:} If the coupling is weak, both $\kappa$ and $\gamma$ can be taken as the same order, acknowledging that the range of interest for $\gamma$ will be limited by $\kappa$. Expressing the frequency $u$ in the matrix of Eq.~\ref{mpt_dimer} as $u=1+\Delta$, with $\Delta$ also of the same order as $\kappa$, the $1-u^2$ and $\gamma (1+\Delta)$, to first order in $\Delta$, are replaced by $-2\Delta$ and $\gamma$ respectively. The standard eigenvalue relation for $\Delta$, after dividing out $2$, is
\begin{equation}
\bm{U} \vec{x} = \Delta \vec{x};\quad{\rm where}\quad
{\bf U}=\begin{pmatrix}
{\kappa - i\gamma\over 2}& -{\kappa\over 2} \\
-{\kappa\over 2} & {\kappa + i\gamma\over 2}
\end{pmatrix}
\label{U_dimer}
\end{equation}
Eq.~\ref{std_pt} is then expressing a generic tight binding eigenvalue relation with $\bm{\Omega} = \omega_0 (\bm{U}+\bm{I})$ directly in terms of the original simple harmonic oscillator parameters.

The above expressions are for a balanced gain and loss characterized by the single gain/loss parameter $\gamma$. This is the $\cal{PT}$-symmetric case which has and exceptional point separating it's eigenmode spectrum into an exact phase, with two real frequencies having zero imaginary parts, and a broken phase having one real frequency shared by two conjugate imaginary parts \cite{GMKMRC18}. If this gain/loss balance is relaxed, the system is no longer $\cal{PT}$-symmetric. Including an overall loss term, $\gamma_0$ in Eq.(\ref{mpt_dimer}) expresses an unbalanced loss in an explicit differential form. Eq.~\ref{mpt_dimer} becomes
\begin{multline}
\hspace{-5mm}
\begin{pmatrix}
1 + \kappa + iu(\gamma_0-\gamma) - u^2 & -\kappa \\
-\kappa & 1 + \kappa + iu(\gamma_0+\gamma) - u^2
\end{pmatrix}
\begin{pmatrix}
x_1 \\
x_2
\end{pmatrix} \\
=0
\label{d_loss}
\end{multline}
where the eigenvalues $u$, found as the roots of the associated secular equation, demonstrate a strictly singular
exceptional point only when $\gamma_0 = 0$. Again, however, in the weak coupling limit, the exceptional point signature is restored. The diagonal presence of the $i \gamma_0$ term, in the lowest order matrix form without the $u$, allows a simple shift of the eigenfrequencies by $i \gamma_0 /2$. Though not $\cal{PT}$, this $2 \times 2$ coupled mode theory form does retain a strict exceptional point.

We have heuristically determined the eigenvalues of the ideal damped mass-and-spring system expressed by the matrix of Eq.~(\ref{d_loss}) in the limit of small $\kappa$ with one oscillator having a fixed small loss, $\gamma_d = \gamma_0-\gamma$, while the other oscillator's imposed loss, $\Gamma=\gamma_0+\gamma$, is increased. This is a better match to the actual experimental situation where the damping factor imposed on the loss pillar is significantly larger than that of the background aluminum, as illustrated by the small intercept for the imaginary part of the frequency shown in Fig.~\ref{fig4} of the main text. For this case, we find that the exceptional point {\it region} (vs. $\Gamma$, as in Fig.~\ref{fig4}(a) for this small-loss/high-loss mechanical system is centered at
\begin{eqnarray}
\Gamma_{\rm EP} = 2(\sqrt{1+2\kappa}-1)+\gamma_d \label{eq:epg} \\
\frac{\omega_{\rm EP}}{\omega_0} = \frac{\sqrt{1+2\kappa}(1+i)+1-{\gamma_d}^2 + i(\gamma_d-1)}{2} \label{eq:epf}
\end{eqnarray}
with the eigenmode splitting at the above position being
\begin{equation}
\frac{\Delta \omega}{\omega_0}  = \frac{2}{3}\kappa^{3/2}.
\label{eq:eps}
\end{equation}
These relationships provide guidance to how the exact simple harmonic mechanical system begins to deviate from the weak coupling limit as the coupling strength increases.

{\bf Interrogation of the Elastodynamics Modes:} In total, ten bending-mode transducers were attached to the system. Plate modes are more efficiently observed by transmission through a pair of transducers attached flush to surface of the plate, with the bending stresses coupling to all of the fundamental forms of bulk modes – one compression and two shear polarizations –  due to their hybridization into free surface modes of the plate. Alternatively, since the torsion modes were the primary focus of this study, particular care was taken to maximize the pillar-pillar torsion transmission while minimally influencing the mode symmetry. To this end, the piezoelectric elements were attached in the chirally symmetric fashion seen in Fig.~\ref{fig2} of the main text, with a small overhang at the pillar edges. This overhang translating the piezo bending motions predominantly into torques applied to the pillars, while canceling the direct pillar bending coupling created by their non-overhanging portions.

Each pillar has two pairs of transducers allowing transmission through a single pillar, or from one pillar to the other, with each opposing pair used in parallel. These transmission path options are used to confirm the equal participation of both pillars in the normal modes, as seen by the close matching of the single pillar doublet with that of the pillar-pillar doublet in Fig.~\ref{fig2}(b). The transmission path including the transducers and aluminum structure was connected to the $50 \Omega$ ports of a Keysight E8050A network analyzer with each port converted to approximately $500 \Omega$ with a United Transformer Type LX-30S audio transformer. In spite of this, even on resonance, the overall transmission is still relatively weak, $\approx -60 dB$. This is primarily due to mechanical impedance mismatch of the small, unobtrusive size chosen for the piezo transducers.

{\bf Theoretical Analysis of the Putty Frequency Shift and Dissipation:} The experimental data for the frequency shift and dissipation induced by the application of putty balls to the top of the torsion pillar showed a remarkably simple behavior: essentially no frequency shift and a dissipation proportional to the contact area. This can be understood in terms of a boundary layer interaction with the surface of the free torsion pillar in the context of a combined Maxwell (series) and Voight (parallel) viscoelastic model~\cite{basicVE} illustrated schematically in the inset of Fig.~\ref{fig3}.

The viscoelastic model is applied in three steps, acknowledging the large mechanical impedance difference that allows for a boundary-layer solution: (1) solve for the surface-stress in the linearized, pure-shear, plane-wave viscoelastic medium of arbitrary thickness; (2) express the result as an effective surface mass density and surface drag constant; and (3) solve for the pillar torsion-mode frequency bounded by the derived surface properties. The bulk wavenumber dispersion $q(\omega)$ and surface shear-stress $S_0$ from step (1), for a specified lateral surface displacement $u_0 e^{-i\omega t}$ is given by
\begin{eqnarray}
q = &q_0 \Omega_0 R, & S_0 = s_0 u_0 \frac{\Omega_0}{R} tan\left(\Omega_0 R \Delta \right), \nonumber\\
R=&\sqrt{\frac{\Omega_0+\Omega_1+i}{\Omega_0 - i \Omega_1}}&
\end{eqnarray}
with $q_0 = \eta_0 /\sqrt{\rho G}$, $s_0 = G/q_0$, $\Omega_x = \omega \eta_x / G$, and $\Delta = d \sqrt{\rho G} / \eta_0$.
The physical constants used are
\begin{center}
\begin{tabular}{l l l}
$\rho$ & density & 1600 kg/m$^3$ \\
$G$ & shear modulus & 15.8 MPa (fit) \\
$\eta_0$ & series viscosity & 72.6 MPoise (fit) \\
$\eta_1$ & parallel viscosity & 54.9 MPoise (fit) \\
$\omega$ & frequency & $2 \pi$ 16565 rad/s \\
$d$ & layer thickness & 0-10 mm \\
\end{tabular}
\end{center}
with $G$ and $\eta_x$ determined by fitting to the data, as summarized in Fig.~\ref{fig3}. The skin depth is defined as the inverse of the imaginary part of the wavenumber, $\delta = 1/Im(q)$.

The relation to equivalent surface loading can then be found by equating the surface shear stress to that is required to move an rigid surface mass density, $\sigma$, attached to the surface, along with an effective viscous surface drag coefficient, $b$ defined a $F_{drag} = -b v_u^2$ acting on the surface. In the frequency domain, $(\sigma \omega^2 + i \omega b) u_0 = S_0$ so that
\begin{equation}
\sigma = \sigma_0 \frac{1}{\Omega_0} {\cal R}e(\frac{1}{R}tan(\Omega_0 R \Delta),\,\, b = b_0 {\cal I}m(\frac{1}{R}tan(\Omega_0 R \Delta))
\label{eq:end}
\end{equation}
with $s_0 = \eta \sqrt{\rho /G}$ and $b_0 = \sqrt{\rho G}$.

Finally, the application of these relations to a free pillar resonant geometry is carried out by solving the 1D wave equation for a pillar torsion mode including the end loading forces. For now, the square pillar is analyzed assuming solid body rotation of pillar elements at each position along the torsion axis of the pillar, $\theta(z)=a e^{i q z} +b e^{-i q z}$, representing the forward and backward torsion waves in the pillar. The equations of motion are
\begin{equation}
\begin{pmatrix}
\sigma \omega^2  + i \omega b & -i q G e^{i q L} & i q G e^{-i q L} \\
-1 & e^{i q L} & e^{-i q L} \\
0  & 1 & -1 \\
\end{pmatrix}
\begin{pmatrix}
\theta_0 \\
a \\
b \\
\end{pmatrix}=0
\label{eq:pillar}
\end{equation}
where $\theta_0$ is the displacement at the end-loaded surface, $0<z<L$. Note that here, $\rho_{Al}$ and $G_{Al}$ are the respective density and shear modulus of the aluminum pillar. The first row expresses the displacement at the top ($z=L$), the second matches the end torsion strain with the surface loading quantities of Eq.~(\ref{eq:end}) at the top, and the third expresses the free end at the bottom, $z=0$.

In the weakly-loaded limit, the solutions show that the the lowest mode frequency shift, $\Delta \omega$, is related to the loading parameters by
\begin{equation}
\sigma = -\rho L \frac{Re(\Delta \omega) }{\Omega_0}, \hspace{1cm} b = -\pi \sqrt{\rho G}   \frac{Im(\Delta \omega) }{\omega_0}.
\label{eq:end_result}
\end{equation}
These are the final relations necessary to identify the putty material properties with the pillar frequency shifts shown in Fig.~\ref{fig3}.

{\bf Fitting Relations for Extraction the Eigenfrequencies:} The measurement procedure for the experimental transmission spectrum is as follows: clamp one pillar; add a putty ball to the other, measuring its transmission spectra with the drive and detector set on that pillar; remove the clamp; switch the drive set to that of the released pillar, measuring the two-pillar transmission spectrum. This process is repeated until the exceptional
point was surpassed.

To extract information of the position of both resonances as precisely as possible (the real part of the frequency) as well as their linewidths (the imaginary part of the frequency), we consider the complex frequency domain ($e^{i \omega t}$) relations for an ideal pair of coupled simple harmonic oscillators, individually damped, excited by a force acting on one oscillator, and detected on the other oscillator, defined earlier. This time, the individual drag constants in $F_{drag} = -bv$ are $b_d$, for the driven side, and $b_{\Gamma}$ for the detected side, and amplitude $F_d$ on the driven side. Experimentally, the mass under the action of the force has the inherent damping value $\gamma_d$ while the detected mass receives the incremental damping, $\Gamma$. The resulting particle displacements for this model are given in terms of scaled parameters by
\begin{equation}
\begin{pmatrix}
x_d \\
x_{\Gamma}
\end{pmatrix}
=
\begin{pmatrix}
1 + \kappa + iu\gamma_d - u^2 & -\kappa \\
-\kappa & 1 + \kappa + \epsilon + iu\Gamma - u^2
\end{pmatrix}
\begin{pmatrix}
f_d \\
0
\end{pmatrix}
\label{eq:lossloss}
\end{equation}
where $x_d$ and $x_{\Gamma}$ are the displacements of the driven and detected masses, and $u=\omega/\omega_0$, $f_d=F_d/k_0$, and $\kappa = k/k_0$. The additional parameter $\epsilon$ is a detuning parameter applied to the natural frequency of the damped (un-driven) oscillator included as a check on the mass loading associated with the experimentally applied damping mechanism. The complex transmittance expression, Eq.~\ref{transmit} used to fit the data is the displacement $x_{\Gamma}(u)$ with added factors $\gamma_d$ and $A$ in the numerator to re-define the transmission amplitude $A$ as the resonant contribution, and $t_0$ and $\phi$ accounting for an overall transmission level and phase shift introduced by the impedance transformers and transducer mechanical mismatch.