\section{\label{sec:setup}Background on Radar Systems}

A radar system uses radio waves to determine both the location of an object relative to the system and the distance between the object and the system.
It operates by transmitting electromagnetic signals toward a certain direction and monitoring the signals that are reflected back from the objects.
The reflected signals are then sent to a signal processor which analyzes the signals, aiming to extract the properties of the objects~\cite{abdallah2017study}.

\begin{figure}[h]
\centering
\includegraphics[width=0.43\textwidth]{Fig/SystemDescription.png}
\caption{An example of a typical radar system architecture.
The red circles indicate our assumed adversary locations.
Our proposed anomaly detection method is located at the link between the radar system and the consuming systems.}
\label{fig:system_description}
\end{figure}

\subsection{System Components}
A typical radar system architecture includes the following components (presented in Figure~\ref{fig:system_description}): \\
    \noindent \textbf{Antenna.} The antenna transforms the electric current into electromagnetic waves and vice versa (transmission and reception, respectively).
    Usually, one bidirectional antenna is used, however radar systems with two separate antennas (a transmitting antenna and a receiving antenna) also exist. \\
    \noindent \textbf{Radar Controller.}
    The radar controller consists of two main components:
    (1) a signal processor, which receives the signals from the antenna (usually via an optical link) and analyzes the signal using a signal processing algorithm aimed at identifying potential relevant objects, and
    (2) a tracking algorithm, which analyzes the signal processor's output, with the aim of classifying objects and tracking movement. \\
    \noindent \textbf{Control System.} The control system is responsible for analyzing radar yields and activating the different systems connected to the radar system, for example, activating a weapon system to neutralize a detected threat. \\
    \noindent \textbf{Radar System Network.} The radar system network is used as a communication channel between the radar controller and the control system.

\subsection{\label{subsec:message_description}Data Stream Description}

The radar system (radar controller) continuously sends a stream of messages to peripheral consuming systems.
The consuming systems process the data stream and make real-time, operational decisions based on the data received.
Each message contains a \textit{Plot} record. This record describes a detected object at a given time.

Since the radar system continuously scans the defined area, several $Plots$ may relate to the same object, describing its properties at different points in time.
A sequence of $Plots$ related to the same object is defined as a $Track$.
Each $Plot$ may be correlated with other $Plot$s (e.g., when they are part of an identified $Track$).
Each $Track$ is associated with an identification number (referred to as the TrackID).
Thus, the TrackID is a part of each individual $Plot$ record.
Figure~\ref{fig:plots_Tracks} illustrates a set of $Plots$ (the individual points) and a $Track$ (the sequence of connected plots).

\begin{figure}[h]
\centering
\includegraphics[width=0.34\textwidth]{Fig/PlotsTracks.png}
\caption{Objects identified at a specific timestamp are represented by points.
A sequence of points related to the same detected object are represented by a path of interconnected points (i.e., \textbf{\emph{Track}}).}
\label{fig:plots_Tracks}
\end{figure}

Each $Plot$ record contains the following attributes:
\begin{itemize}
    \item \textbf{Metadata.} Contains the source ID (unique identifier of the sending radar system), message ID, message length, etc.

    \item \textbf{Identified object's properties.} Categorical and numerical attributes that describe the identified object, such as the detected object's location and speed, object type (e.g., airplane, bird), etc.

    \item \textbf{Sequence-related properties.}
    Describe different properties of the $Plot$ record as part of a sequence of plots related to the same object.
    One property is the identification number of the $Track$ that a $Plot$ belongs to.
    Another property is the timestamp indicating when a $Plot$ has been updated on the system; this property describes the delay between the previous $Plot$ and the current $Plot$.
\end{itemize}

\section{\label{sec:dataset}Dataset}

We rely on a dataset of benign radar behavior for the development of the anomaly detection mechanism. We use benign data collected from a real radar system deployed on different vessels to identify different objects.
The data was collected from several different platforms at different times.

The data contains in total 138,639 samples from 24 different recording sessions.
The data was not uniformly distributed between the systems, and five recording sessions contains 76\% of the data (as can be seen in Figure~\ref{fig:RecordsDesc}).
Therfore, for our evaluation we used these five recording sessions: R7, R19, R20, R29, R31.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{Fig/Records Description.png}
\caption{Records Description}
\label{fig:RecordsDesc}
\end{figure}

\subsection{\label{subsec:features description}Raw Features}
The data included 50 features. Table~\ref{tab:features} presents the features.
The features can be categorized as follows:
\begin{enumerate}
\item constant (unchanged) features - eight of the features (such as srcid and msgid) were constant in the whole dataset.
The reason is that in the collected dataset only one type of messages has been recorded, and all messages were from a single source (i.e., same radar entity used for each specific recording).
The recorded message type is the one used to inform the WCS about the objects that have been identified by the radar. This message type is considered the most important message type with regards to radar operation from a systems perspective.
\item sequence-related - six features that should be evaluate while considering sequence of samples.
\item categorical - nine of the features were categorical, i.e., limited, fixed, number of possible values, with no mathematical metric.
The following are examples for those categorical features:
\begin{itemize}
    \item \textbf{trackCategory} - describes the object type, whether it airplane, cloud, etc.
    \item \textbf{hostility} - describes whether an object is an enemy or not.
\end{itemize}
\item numerical - 18 of the features were numerical. The majority of these numerical features represent the correlation between the detected object's physical properties, i.e., the components of its location and speed vectors.
\end{enumerate}

Note that the actual object location does not exist in our dataset due to information confidentiality.

\subsection{{\label{subsec:extracted}}Extracted Feature}

In addition to the raw (basic) features provided in the stream of events, we compute additional feature.
We called this features $updatingPeriod$ and it's value is the UpdateTime difference between the current $Plot$ and the previous $Plot$ of the same $Track$ (e.g., $currentUpdateTime - prevUpdateTime$).
For the first $Plot$ of the $Track$ the feature value is the same value as the second $Plot$ of the $Track$, since there is not $prevUpdateTime$.
The main use of this feature is in tracking the time difference between $Plots$ so we can detect adding/drooping events.

The feature values vary and range from 100 to more then 1,000,000 time units.
An histogram of this feature is presented in Figure~\ref{fig:updatingPeriod}.

\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{Fig/UpdatingPeriod histogram.png}
\caption{UpdatingPeriod histogram.}
\label{fig:updatingPeriod}
\end{figure}

\subsection{\label{subsec:plot tracks}Plots and Tracks}
Each sample in our dataset describes a $Plot$.
A plot is a radar record that describes a detected object at a specific time.
A plot record captures the object properties and radar-related metadata for the specific detection.
Since the radar is continuously scanning the space, several plots may relate to the same object, describing properties of the same detected object at different times.
A sequence of plots related to the same object is identified by the radar system as a $Track$.
Each track is indicated by a specific identification number (TrackID) in each plot record.
In the datasets used, plots that appear within a specific recording session and have the same TrackID correspond to the same track.
Figure~\ref{fig:plots_Tracks} present an illustration of plots (indicated by individual points) and a track (indicated by a sequence of connected points/plots).

\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{Fig/Plots and Tracks.png}
\caption{Objects identified at a specific timestamp are represented by points.
A sequence of points related to the same moving object are represented by a path of interconnected points (i.e., track).}
\label{fig:plots_Tracks}
\end{figure}

\section{\label{sec:evaluation}Evaluation}

For the evaluation we use benign data collected from four real radar systems that are deployed in different operational setups and used to identify objects.
We refer to each dataset as a \emph{recording session}.
The number of messages (i.e., radar data records) in each recording session is as follows: R1 -- 45,720 (50.7\%), R2 -- 23,017 (25.5\%), R3 -- 11,955 (13.2\%), and R4 -- 9,551 (10.6\%).
In total, there are 90,243 messages.

\begin{comment}
\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|}
\hline
\textbf{Recording session} & \textbf{Number of messages} & \textbf{Percentage} \\
\hline
R1 & 45,720   & 43\% \\ \hline
R2 & 23,017    & 21.7\% \\ \hline
R3 & 15,992    & 15.1\% \\ \hline
R4 & 11,955    & 11.2\% \\ \hline
R5 & 9,551    & 9\% \\ \hline
\end{tabular}
\caption{Dataset description.}\label{tab:dataset}
\end{table}
\end{comment}

\subsection{\label{subsec:attacks} Attack Scenarios Implemented}

Since our dataset does not include any attacks, we had to simulate the attacks and inject them into our dataset.
We simulated four attack scenarios by manipulating valuable benign data, and conducted 15 different experiments for each attack.

\textbf{Categorical feature manipulation.}
This is an integrity violation attack in which the attacker changes a categorical feature.
In the following cases, such manipulations may cause the consuming systems or radar operators to incorrectly characterize and handle the detected object.
\begin{enumerate}
    \item objectType - changing this feature may cause aerial objects to appear as ground objects and vice versa.
    This can change the way the radar operator reacts to the various objects detected; an attacker can utilize this attack in order to disguise threatening objects, leaving them untreated by radar operators.
    \item alertRaised - changing this feature may cause an alert to be issued for no reason and vice versa.
    Such an attack can create many false alarms, causing the radar operator to ignore threats.
    \item objectCategory - changing this feature may cause friendly objects to be considered enemy objects and vice versa.
    This attack can cause radar operators to fire on friendly objects or ignore enemy objects.
\end{enumerate}

Changing the features for an entire $Track$ or a $Track$ segment will be much more beneficial to the attacker than changing the features for random $Plot$s.
This is because changes in the features of random $Plot$s may be ignored by the radar operator and are not likely to influence the action taken by an operator. Therefore, we applied the above manipulations on an entire $Track$.

The following process was used to generate the test set given a benign set $B$ collected from real radar systems: (1) create $B'$ by duplicating $B$; (2) select $f$ -- the feature that should be manipulated; (3) for each $Track$ $T$ in $B'$, change the value of $f$ to a distinct random value from the set of valid values of $f$;  (4) label the manipulated $Plot$s as anomalies; and (5) combine the benign set $B$ and the manipulated set $B'$ into one test set.
This process generates a similar number of benign and malicious $Plot$s; therefore, the resulting test set is balanced.

\textbf{\emph{Plot} dropping.}
This is an availability violation attack in which the attacker drops one/several $Plot$s from a $Track$, so that the detected object will evade the consuming system for some period of time.

Dropping several consecutive $Plot$s will be much more beneficial to the attacker than dropping nonconsecutive $Plot$s or just a single $Plot$. Therefore, for a $Track$ $T$, we first select a minimum value $c$ ($c\le|T|$) of $Plot$s to be removed from $T$. We use $c=10$ since a value less than 10 does not allow an attacker to cause real harm to the data integrity of a radar system.

The following process was used to generate the test set given the benign set $B$ collected from real radar systems: (1) for each $Track$ $T$, select a random $Plot$ index $i$ and a random integer $r\in{c, |T|-1}$; (2) starting at index $i$, drop a minimum number $minimum(|T|-i, r)$ of consecutive $Plot$s; (3) label the $Plot$ that follows the dropped $Plot$s as an anomaly.
This process generates a different number of benign and malicious $Plot$s; therefore the resulting test set is imbalanced.

\subsection{\label{subsec:evalsetup}Evaluation Method}

We conducted three types of evaluations; in each case, we repeated the experiment five times (each time a different recording session was selected for testing).

A description of the types of evaluations performed is provided below, along with an example of the training/testing data in a scenario in which the R4 recording session is examined: \\
\begin{itemize}
    \item \textbf{Cross-session setup:} in this case, each time we used three recording sessions for training, and the fourth session was used to test the model.
    Example for the R4 recording session: training -  R1, R2, and R3; testing - R4.
    \item \textbf{Chronological setup:} in this case, for each recording session, we trained the model on the first 90\% of the instances (in chronological order) and tested it on the remaining 10\% of the instances.
    Example for the R4 recording session: training - 90\% of the instances of R4; testing - remaining 10\% of the instances of R4.
    \item \textbf{Transfer learning setup:} this case is a combination of the previous two setups, in which we used three recording sessions, as well as the first 10\% of the instances of the fourth session, to train the model, and the remaining 90\% of the instances of the fifth session were used to test the model.
    Example for the R4 recording session: training -  R1, R2, R3, and the first 10\% of the instances of R4; testing - the remaining 90\% of the instances of R4.
\end{itemize}
It should be noted that the cross-session evaluation setup has two significant advantages over the other two: (1) in practice, it is easier for radar engineers to deploy a model that has already been trained rather than training a model using new system data, and (2) training the model on new system data can expose the model to cyber risks (e.g., adversarial poisoning~\cite{vorobeychik2018adversarial}).

\subsection{\label{subsec:evalmetric}Evaluation Metrics}

The task of identifying anomalies in the radar data stream is a binary classification task, where benign samples are labeled as zero (negative), and malicious data is labeled as one (positive).
We used the following common metrics for the evaluation: true positive rate (TPR)/recall, false positive rate (FPR), receiver operating characteristic (ROC) curve, area under the ROC curve (AUC), precision, precision-recall curve, and average precision (AP).

\subsection{\label{subsec:results}Results}
In this section, we present the evaluation results for each attack scenario (presented in Section~\ref{subsec:attacks}) and evaluation method (presented in Section~\ref{subsec:evalsetup}).
First, we present a graph of the evaluation results as a function of different thresholds (using ROC and PRC graphs).
Then, we summarize the evaluation results for predefined anomaly thresholds.
The method for calculating the anomaly thresholds is described in Section~\ref{sec:method:details}.

\noindent \textbf{Categorical feature manipulation.}
Figures~\ref{fig:objectType} -~\ref{fig:alertRaised} present the detection results of the Field Manipulation Detection module when manipulating objectType, objectCategory, and alertRaised features respectively.
As can be seen, in all cases except for recording session R4 of the objectType feature manipulation attack, all setups had great performance in terms of the AUC and AP for all categorical feature manipulation attacks.
For recording session R4 of the objectType feature, the best performance was obtained with the chronological setup.
The best detection rates were observed for the manipulation attack on both objectCategory and alertRaised features.

\noindent\textbf{\emph{Plot} dropping.}
Table~\ref{tab:drop_packet} presents the distribution of the benign /

\noindent malicious samples in the $Plot$ dropping attack.
It is noteworthy that the data is imbalanced, since for each $Track$ we only execute the attack once.
The performance of the Timing-Based Anomaly Detection module on the generated testset is presented in Figure~\ref{fig:drop_packet}.
As can be seen, in all scenarios the model achieved good results.

\noindent\begin{minipage}{\linewidth}
\noindent\textbf{Performance for predefined anomaly thresholds.}
Table~\ref{tab:results_summary} presents the performance for each attack scenario and each setup in terms of the true positive rate (TPR) and false positive rate (FPR) for a threshold that was computed on a validation set and then applied on a test set.
As can be seen, our proposed method can detect 90\% of the $Plot$ dropping attacks, with a false positive rate of 2\%, and from 76\% to 96\% of feature manipulation attacks, with a false positive rate of 2\%.
In typical scenarios, 400 $Plot$s are generated and transferred by a radar system every minute. A false alarm rate of 2\% means that eight $Plot$s should be tracked by a system operator/automatic system every minute, which is practical.
\end{minipage}

In addition, in most cases our proposed method achieves very good results for the cross-session setup. This is an important point, since it means that a pretrained model can be migrated to new radar systems without retraining.

\begin{table}[h]
\centering
\caption{Number of benign (b) and malicious (m) samples in the \emph{Plot} dropping attack.}
\scriptsize
\begin{tabular}{|l|c|c|c|c|c|c|c|c|}
\hline
\multirow{3}{*}{} & \multicolumn{8}{c|}{Recording Session Examined} \\
\cline{2-9}  & \multicolumn{2}{c|}{R1}  & \multicolumn{2}{c|}{R2}  & \multicolumn{2}{c|}{R3}  & \multicolumn{2}{c|}{R4} \\
\cline{2-9}  & b & m & b & m & b & m & b & m \\ \hline
Cross-session  & 46,600 & 213 & 23,068 & 177 & 12,139  & 97 & 9,484 & 119 \\ \hline
Chron. (10\%) & 4,606 & 60 & 2,323 & 34  & 1,206 & 15 & 842 & 31 \\ \hline
Transfer (90\%) & 42,709 & 195 & 21,126 & 170 & 10,808 & 86 & 8,451 & 110 \\ \hline
\end{tabular}
\label{tab:drop_packet}
\end{table}

\begin{table}[h]
\caption{Evaluation results obtained for predefined anomaly thresholds.}
\centering
\scriptsize
\begin{tabular}{|l|r|r|r|r|}

\hline
\textbf{Setup} & \multicolumn{1}{l|}{\textbf{Avg AUC}} & \multicolumn{1}{l|}{\textbf{Avg PR}} & \multicolumn{1}{l|}{\textbf{Avg TPR}} & \multicolumn{1}{l|}{\textbf{Avg FPR}} \\ \hline
\multicolumn{5}{c}{Feature manipulation - objectType} \\ \hline
Cross-session    & 0.922                              & 0.930                              & 0.759                              & 0.028                             \\ \hline
Chronological (10\%) & 0.972                              & 0.712                             & 0.894                              & 0.016                             \\ \hline
Transfer (90\%)      & 0.943                              & 0.931                             & 0.710                              & 0.090                             \\ \hline
\multicolumn{5}{c}{Feature manipulation - objectCategory} \\ \hline
Cross directories    & 0.986                              & 0.973                              & 0.906                              & 0.028                             \\ \hline
Chronological (10\%) & 0.997                              & 0.994                             & 0.770                              & 0.016                             \\ \hline
Transfer (90\%)      & 0.975                              & 0.952                             & 0.769                              & 0.090                             \\ \hline
\multicolumn{5}{c}{Feature manipulation - alertRaised} \\ \hline
Cross-session    & 0.987                              & 0.975                             & 0.963                              & 0.028                             \\ \hline
Chronological (10\%) & 0.997                              & 0.988                             & 0.776                              & 0.016                             \\ \hline
Transfer (90\%)      & 0.974                              & 0.950                              & 0.750                                  & 0.090                             \\ \hline
\multicolumn{5}{c}{\emph{Plot} dropping} \\ \hline
Cross-session    & 0.975                              & 0.820                             & 0.900                              & 0.023                             \\ \hline
Chronological (10\%) & 0.987                              & 0.916                             & 0.937                              & 0.026                             \\ \hline
Transfer (90\%)      & 0.983                              & 0.830                             & 0.909                              & 0.023                             \\ \hline
\end{tabular}
\label{tab:results_summary}
\end{table}

\begin{figure*}[h]
\centering
\includegraphics[width=0.55\textwidth]{Fig/EvaluationResults_ChangeFeature_trackCategory.png}
\caption{The ROC and PRC curves for the objectType feature manipulation attack for each recording session examined.}
\label{fig:objectType}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.55\textwidth]{Fig/EvaluationResults_ChangeFeature_hostility.png}
\caption{The ROC and PRC curves for the objectCategory feature manipulation attack for each recording session examined.}
\label{fig:objectCategory}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.55\textwidth]{Fig/EvaluationResults_ChangeFeature_threatAlert.png}
\caption{The ROC and PRC curves for the alertRaised feature manipulation attack for each recording session examined.}
\label{fig:alertRaised}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=0.55\textwidth]{Fig/EvaluationResults_DropPacket.png}
\caption{The ROC and PRC curves for the \emph{Plot} dropping attack for each recording session examined.}
\label{fig:drop_packet}
\end{figure*}

\subsection{\label{subsec:preformance}Performance Analysis}

In order to understand whether the proposed method is practical and can be applied for real-time tracking of anomalies in radar system data streams, we measured the average time it takes to process a single $Plot$.
This experiment was conducted on a standard machine with an Intel core i7-7600U 2.8Ghz CPU and 32GB RAM.
The machine's operating system was Windows 10.

The results showed that our proposed method can analyze 20K $Plot$s per second.
Given that in typical scenarios a radar system generates and transfers 400 $Plot$s per minute, we conclude that our method is practical and can be used for real-time tracking of anomalies in radar system data streams.
\section{\label{sec:intro}Introduction}

Radar systems use electromagnetic radiation to detect objects within a defined scanned area~\cite{maini2018handbook}; they can also be used to classify the detected objects.
Radar systems are mainly integrated in air and terrestrial traffic control systems, autonomous vehicles, air defense systems, anti-missile systems, aircraft anti-collision systems, and ocean surveillance systems.

Typically, radar system architecture consists of the following basic components: (1) an antenna - responsible for transmitting/receiving electromagnetic waves to/from the scanned area, and (2) a radar controller - responsible for analyzing the waves received from the antenna in order to determine the properties of the object.

Usually, the radar controller is connected to a centralized switch which controls the routing of the radar system’s messages.
In many cases, the switch can be used to communicate with external entities, i.e., other systems.
Often, information regarding the objects detected by the radar system is sent to, and used by, a peripheral consuming system, such as a missile system or a graphical user interface used by an operator~\cite{abdallah2017study}.
Those systems process the data stream and make real-time, operational decisions based upon the data received.

In recent years, as technology has evolved, the use of radar systems has increased, along with reliance on their correct and reliable operation.
Radar systems often include an extended set of components, such as communication systems or SCADA systems, which are vulnerable to cyber and physical attacks.
These components can be exploited by attackers in order to compromise the radar system~\cite{cohen2019security}.
In addition, in many cases, radar systems are integrated within systems that are vulnerable to cyber attacks, such as autonomous vessels~\cite{bolbot2020novel} and smart vehicles~\cite{kumar2018brief}.
These vulnerabilities could be used by an attacker as a backdoor for an attack on the radar system.

Despite the many potential attack vectors, there has been no research performed that has suggested or evaluated a method for detecting cyber attacks on radar systems in real time. In this study, we address this gap.
We propose a deep learning-based method for detecting anomalies in critical parts of the data stream generated by the radar controller and consider attack scenarios in which the attacker manipulates the data being transferred by the radar system to peripheral consuming systems.
Such manipulation can occur in motion or in use.
The proposed method simply requires the ability to monitor the messages sent by the radar controller, and thus it can be easily and safely integrated into existing systems without the need to change the system.
Our proposed method is designed to deal with the particular data comprising the radar data stream; unlike many other domains, radar data streams are made up of sequential and heterogeneous data.

In order to evaluate the proposed method, we used data collected from four real radar systems.
We collected legitimate data streams into which we injected a variety of artificial attacks, including manipulations with high impact on the system (integrity violation and denial of service), to assess our method's performance.
The results of our evaluation show that our proposed method can detect 90\% of message dropping attacks, with a false positive rate of 2\%, and from 76\% to 96\% of feature manipulation attacks, with a false positive rate of 2\%.
These results are obtained in a cross-session experiment in which the model is trained on data collected from three radar systems and tested on data collected from another radar system, thus demonstrating the ability to migrate a pretrained model to new radar systems without retraining.

We summarize the main contributions of this study as follows:
\begin{itemize}
    \item We present a novel deep learning-based method, which consists of two modules, to detect anomalies in critical parts of the heterogeneous data stream generated by radar systems.
    \item The proposed model represents both numerical and categorical features in an architecture that can be utilized in other similar domains.
    \item The proposed method can be integrated into existing radar systems, without changing the radar system's existing components and without retraining the model.
    \item The structure of the proposed method enables it to identify the anomaly type detected, determine whether a legitimate message is missing from a sequence of messages, and understand whether a feature of an existing message has been manipulated.
    \item The effectiveness of the proposed method is demonstrated in experiments using a dataset collected from real operational radar systems and simulated attacks.
\end{itemize}

\section{\label{sec:method:details}Proposed Method}

In this section, we provide a detailed description of each component of the proposed method.

\subsection{Field Manipulation Detection}
As illustrated in Figure~\ref{fig:HLA}, this module consists of the following two computational components: (1) the $Plot$-Level Anomaly Detector, and (2) the $Track$-Level Anomaly Detector.
As mentioned earlier, the $Plot$-Level Anomaly Detector generates an anomaly score for each $Plot$ independently, while the $Track$-Level Anomaly Detector aggregates the anomaly scores of the $Plot$s and generates an alert if the value computed exceeds a certain threshold.

\textbf{\emph{Plot}-Level Anomaly Detector.}
This module focuses on detecting an anomaly within a single $Plot$. The $Plot$ features used by this module are described in~Section~\ref{features}.
In order to detect an anomaly within a given $Plot$, a special variant of an autoencoder is proposed (Figure~\ref{fig:plot_level_anom}).

The input neurons representing the categorical features are attached with an embedding layer. An embedding technique is commonly used to create a concise, numerical representation of categorical features~\cite{hancock2020survey}.
The embedding representation of the 10 categorical features is concatenated with the 17 numerical features, resulting in a vector of size 27.
This vector is then fed to the stacked autoencoder consisting of seven layers with respectively 27, 20, 15, 10, 15, 20, and 27 neurons.

Finally, the following output layer is attached:
\begin{enumerate}
    \item For the numerical features, a fully connected layer is attached, followed by the \emph{linear} activation function. This layer contains 17 neurons against 17 numerical features.
    \item For each categorical feature with $l$ possible values, $l$ output neurons are assigned, followed by the \emph{softmax} function. During inference, this function provides a probability distribution over all possible values for each categorical feature.
\end{enumerate}

During the training phase, we use data that consists solely of benign $Track$s. As described in Section~\ref{sec:setup}, each $Track$ contains several $Plot$s (training examples). We divide the $Track$s in the dataset so that 80\% is used for training \emph{tr} and 20\% is used for validation \emph{val}.
As illustrated in Figure~\ref{fig:plot_level_anom}, the label for each example is one vector of 17 entries containing the value of each numerical feature, concatenated to 10 vectors representing the one-hot encoding for each of the 10 categorical features.

The loss function used is the following combination of the \emph{mean squared error} (MSE) and the \emph{sparse categorical cross-entropy} (SCCE):

\begin{equation}
	loss = MSE(x_i: x_i \in numerical) +  \sum_{x_i \in categorical} SCCE(x_i)
\end{equation}

Given \emph{tr}, the network is trained to minimize the loss function on \emph{val} using the Adam optimizer. Training is stopped when the loss function reaches its minimum on \emph{val}.

\begin{figure}[h]
\centering
\includegraphics[width=0.64\textwidth]{Fig/model2.png}
\caption{The neural network architecture of the proposed \emph{Plot}-Level Anomaly Detector; the network is based on an autoencoder.}
\label{fig:plot_level_anom}
\end{figure}

\textbf{\emph{Track}-Level Anomaly Detector.}
This module consists of a $Track$ Score Collector and a $Track$-Level Anomaly Detector.
The scores generated by the $Plot$-Level detection module serve as the input for this model.
The model collects all of the scores for the $Plot$s that are part of the same $Track$ and outputs them for the $Track$-Level Anomaly Detector model.
All of the $Plot$s' scores for a $Track$ serve as the input to this model.

For each $Track$, we define the anomaly score as the average of all of the input scores calculated by the \emph{Plot}-Level Anomaly Detector (those are stored inside the $Track$ Score Collector).
In order to determine the anomaly threshold, we used the validation set \emph{val} that was used to train the \emph{Plot}-Level Anomaly Detector. The anomaly threshold is calculated as the maximum value over the $Track$s' average scores.

\subsection{Timing-Based Anomaly Detection}

This module focuses on detecting anomalies related to the $Plot$s' arrival times. It uses the following subset of the $Plot$ features described in Section~\ref{features}: $num1$, objectType, signalQuality, and trackType.
The module consists of the following two components: (1) the Data Preprocessing component, and (2) the Sequence-Based Anomaly Detector (see Figure~\ref{fig:lstm}), which is based on a Long Short-Term Memory (LSTM) network.

\textbf{Data Preprocessing.}
The input for this component is the current $Plot$, along with the $K$ previous $Plots$ that correspond to the same $Track$ (with a sequence length of $K+1$).
This component consists of the following three parts:
\begin{enumerate}
    \item Categorical feature one-hot encoding -
    in this part, each feature is converted to a vector that has a value of one in one entry (corresponding to the feature value) and zeros in all of the other places.
    \item UpdatingPeriod extraction - in this part, we calculate a feature called UpdatingPeriod.
    The value of this feature is the time difference between two consecutive $Plot$s in the sequence (e.g., current time - previous time).
    \item Feature scaling - in this part, we apply min-max scaling on the data for feature scaling.
\end{enumerate}

\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{Fig/lstmnew.png}
\caption{The neural network architecture of the proposed Sequence-Based Anomaly Detector.
The network is based on an LSTM network.}
\label{fig:lstm}
\end{figure}

\textbf{Sequence-Based Anomaly Detector.}
This module receives a sequence of length $K+1$ as input, and by using the first $K$ elements of the sequence, the model tries to predict the value of the $UpdatingPeriod$  (e.g., time interval) feature of the $K+1$-th element (i.e., current $Plot$).
To enable this, we designed an LSTM-based regressor.
The architecture of the regressor is presented in Figure~\ref{fig:lstm}. As can be seen, the $K$ length sequence serves as input to an LSTM network with five hidden units.
Then, the output of the LSTM component in the final step is fully connected to one neuron.
At the end, a linear activation function is applied to this neuron.

During the training phase, we use data consisting solely of benign $Track$s. For each $Track$, we generate a set of training examples such that each consists of a set of $K$ consecutive $Plot$s and is labeled by the $UpdatingPeriod$ feature of the following $K+1$-th $Plot$.
We divide the $Track$s in the dataset so that 80\% is used for training \emph{tr} and 20\% is used for validation \emph{val}. The loss function used is the MSE. Given \emph{tr}, the network is trained to minimize the loss function on \emph{val} using the Adam optimizer. Training is stopped when the loss function reaches its minimum on \emph{val}.

Once the model training is complete, the anomaly threshold $thr$ is set.
This anomaly threshold, above which an instance is considered anomalous, is calculated as the sum of the sample mean and standard deviation of MSE over \emph{val}:

\begin{equation}
	thr = mean(MSE_{val}) + std(MSE_{val})
\end{equation}

Regarding the $K$ parameter, the detection accuracy increases as long as $K$ is large. However, an anomaly cannot be detected during the first $K$ $Plot$s. Practically, for radar systems, setting $K$ at five is a good balance in this trade off.
\section{\label{sec:method:high_level}High-Level Description of the Proposed Method}

We propose an unsupervised anomaly detection method which is based on continuous monitoring of the messages transmitted from the radar to the peripheral consuming systems.
Our proposed method utilizes state-of-the-art deep learning modules to detect possible malicious data manipulation.

\subsection{Description of the Proposed Method}
The proposed anomaly detection method (see Figure~\ref{fig:HLA}) consists of two main modules: (1) a Field Manipulation Detection module, and (2) a Timing-Based Anomaly Detection module.
Each module outputs an anomaly score, and an alert is generated if the score exceeds a predefined threshold.

\textbf{Field Manipulation Detection.} For each $Track$, the Field Manipulation Detection module receives the categorical and numerical features of a $Plot$. First, it determines if the relation between the features' values is anomalous. If so, an anomaly score for the $Plot$ is generated.
An anomalous relation between the values of a $Plot$'s features indicates that a malicious change has been made, aimed at violating the radar system's integrity.
Such manipulations may cause the consuming systems or radar operators to incorrectly characterize and handle the detected objects.

Finally, to increase the sensitivity of the detection method to malicious manipulation attacks, an alert is generated if the scores' average exceeds a certain threshold.
The latter step aims to detect anomalies that are harder to detect by analyzing a single $Plot$.

\textbf{Timing-Based Anomaly Detection.} The Timing-Based Anomaly Detection module receives the last $K$ inspected $Plot$s and the current $Plot$ associated with the same $Track$, and determines if the time difference between the current $Plot$ and the previous $Plot$ is anomalous.
An anomalous time difference between $Plot$s indicates malicious $Plot$ dropping or injecting events.
Such activities may enable an object to evade detection by the detection system or cause a denial of service.

\begin{figure}[h]
\centering
\includegraphics[width=0.48\textwidth]{Fig/overall2.png}
\caption{High-level architecture of the proposed anomaly detection method.}
\label{fig:HLA}
\end{figure}

\subsection{\label{subsec:design}Design Considerations}

The following considerations were taken into account in the design of the proposed method: \\
    (1) \textbf{Analysis of heterogeneous data.} As mentioned in Section~\ref{subsec:message_description}, the $Plot$ records consist of numerical and categorical features.
    Machine learning-based models usually cannot be applied directly to heterogeneous features, so there is a need to transform those features before applying a model.

    \noindent (2) \textbf{Immediate detection.}
    Often, radar systems are integrated within real-time decision-making systems.
    This requires a detection method capable of making a quick and accurate prediction, while keeping the false alarm rate as low as possible.

    \noindent (3) \textbf{Processing sequential (stream) data.}
    As mentioned in Section~\ref{subsec:message_description}, each $Plot$ may be correlated with other $Plot$s (e.g., when they are part of an identified $Track$).
    Therefore, in addition to capturing and learning patterns among features of the same $Plot$ record, the model should be able to learn patterns within a sequence of $Plot$s.

    \noindent (4) \textbf{Explaining anomalies.}
    Understanding which part of the data is anomalous can help the system operator perform the correct action.
    If the method has the ability to identify the packets/features that contribute the most to an anomaly, it can improve the system operators' decision-making capabilities.
    The structure of the proposed method enables it to identify the anomaly type detected, determine whether a legitimate message is missing from a sequence of messages, and understand whether a feature of an existing message has been manipulated.

\subsection{Extracted Features} \label{features}
As described in Section \ref{subsec:message_description}, each $Plot$ record contains a variety of features.
In our dataset, the $Plot$ record includes 10 categorical features and 18 numerical features, all with high potential of being exploited by an attacker interested in disrupting the radar system's normal operation. \\
\noindent \textbf{Categorical features.} Five of the categorical features describe the returned signal's physical properties.
The other five are: (1) trackType - specifies the $Track$ type, (2) signalQuality - specifies the quality of the returned signal, (3) objectType - specifies the type of object detected, (4) alertRaised - specifies whether an alert has been raised on the system, and (5) objectCategory - specifies whether the object is considered hostile. \\
\noindent \textbf{Numerical features.} One numerical feature is the timestamp indicating when a $Plot$ has been updated on the system.
The other 17 numerical features ${num1, num2,...,num17}$ relate to the correlations between a detected object's locations.

Note that due to privacy concerns, a more detailed description of the features in the dataset used in this research cannot be provided.

\section{\label{sec:related}Related Work}

Most studies that used machine learning techniques to protect radar systems from attacks focused on protecting the systems from jamming attacks.
Some studies proposed jamming classification methods~\cite{hong2018classification,mendoza2017classification,soto2017optimization,wu2017jamming}, while others proposed jamming mitigation strategies~\cite{kang2018reinforcement,liu2020decentralized}.
In addition, a comprehensive survey discussing all of these methods in detail was published~\cite{lang2020comprehensive}.
In other related research, machine learning techniques were used to protect the systems generally integrated within radar systems, like the ADS-B~\cite{ying2019detecting,habler2018using} and AIS systems~\cite{obradovic2014machine}.
To the best of our knowledge, no study has proposed a method for detecting anomalies in radar system data streams.

In recent years, deep learning models have increasingly been used to analyze network traffic and detect anomalies.
Most of the methods proposed are based on autoencoders and their variants~\cite{chalapathy2019deep}.
An autoencoder (AE) is an unsupervised algorithm that compresses input into a lower dimensionality and then decompresses the
input into its original dimensionality; thus, normal instances are decompresses properly, while the abnormal instances are not.
In this way, anomalous input can be identified.
Some examples of methods based on AEs include N-BIoT~\cite{meidan2018n}, which uses an AE to detect botnet behavior in the network traffic of IoT devices, and Kitsune~\cite{mirsky2018kitsune}, which utilizes a smart feature mapper and an ensemble of AEs to detect anomalous behavior in network traffic.

LSTM-based networks are also widely used to detect anomalies in network traffic; these deep learning networks are very well suited for sequential data.
In this type of network, instead of processing the whole sequence together, the network processes each object in the sequence separately, in chronological order.
So in each step, the network's inputs are: (1) the current object in the sequence, along with (2) the output of the network from the previous step.
An example of using an LSTM-based network was presented by Bontemps et al.~\cite{bontemps2016collective} who predicted anomalies in the network data stream by using the sample, as well as the context of the sample.

Previous studies that used deep learning models to detect anomalies in network traffic relied strictly on numerical features. However, in this study, we take into consideration the fact that the data stream created by the radar system is heterogeneous, i.e., it contains both numerical and categorical features. Since manipulating both types of features may significantly violate the integrity and availability of radar systems, there is a need for a deep learning method that can detect anomalies in heterogeneous data.
Our proposed method is designed to detect attack scenarios that are valuable specially for radar systems. Thus, our proposed method focuses on real threats while reducing false alarms.

\section{\label{sec:summary}Summary \& Conclusions}

In this paper, we propose a novel deep learning-based method for detecting data manipulation attacks on radar systems.
We consider attack scenarios in which an attacker manipulates the data sent by the radar system to peripheral consuming systems.
To evaluate our method, we used data collected from four real radar systems.
This dataset consists of $Plot$ records that are sent from the radar system to a peripheral consuming system.

To evaluate our method, we generated attacks considered beneficial to the attacker, ranging from integrity violation attacks to availability violation attacks.
The results of our evaluation show that the proposed method can learn the data's normal behavior and distinguish between normal and manipulated data.

In our design process, we took into account the benefits of identifying the anomaly type detected, determining whether a legitimate $Plot$ is missing from a sequence of $Plot$s, and understanding whether
an existing $Plot$'s feature has been manipulated.
This property could help radar operators determine the correct action to take when an attack occurs.

The transferability and practicality of the proposed method is demonstrated by: (1) our method obtains very good results when training on data collected from three radar systems and testing on data collected from another radar system, and (2) a typical Intel controller can analyze $Plot$s at a frequency of 20K/sec, while radar systems generate and transfer $Plot$s at a frequency of 7/sec.

\section{\label{sec:threat_model}Assumed Threat Model}

Although radar systems are vulnerable to a variety of threats~\cite{cohen2019security}, most prior research performed on radar systems has focused on aspects of functionality~\cite{barton2004radar,barton1988modern}.
In this study, we consider attack scenarios in which an attacker manipulates the data being transferred by the radar system to peripheral consuming systems.
Such manipulation may occur in motion or in use.
Specifically, as illustrated in Figure~\ref{fig:system_description}, the adversary locations (points of manipulation) considered in this study are the radar controller and the radar system network.
Those can be compromised with physical or remote access, or through a supply chain attack (i.e., before the radar system's deployment).

Once the radar controller has been compromised by the attacker (whether remotely or through a supply chain attack), he/she can execute malicious commands that manipulate the data while it is being processed, prior to its transmission from the radar controller.
With physical access to the radar system network, the attacker can replace a networking device (e.g., switch) with a malicious networking device that manipulates the data in motion. The effects of such tampering attempts are expected to be covert and, more specifically, to be ``under the radar" of radar operators (i.e., radar operators should not be able to identify the tampering).

Based on the specified threat model, we suggest an anomaly detection method that (1) detects abnormal data behavior indicative of such malicious activity, and (2) can be implemented as a detector at the link between the radar system network and the control system, to provide anomaly-related insights to radar operators and/or systems that consume the radar data.

\subsubsection{Plots and Tracks}
The radar system continuously communicates with the WCS and sends messages to it.
These messages include among other types, $Plot$ reports.
$Plot$ describes the detected object properties at a specific time.
Hence, it's considered to be the most important with regards to the radar operation from a systems perspective.
Since the radar is continuously scanning the space, several $Plots$ may relate to the same object, describing properties of the same detected object at different times.
A sequence of $Plots$ related to the same object is identified by the radar system as a $Track$.
Each $Track$ is indicated by a specific identification number (TrackID) in each $Plot$ record.
Figure~\ref{fig:plots_Tracks} present an illustration of plots (indicated by individual points) and a track (indicated by a sequence of connected points/plots).

\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{Fig/Plots and Tracks.png}
\caption{Objects identified at a specific timestamp are represented by points.
A sequence of points related to the same moving object are represented by a path of interconnected points (i.e., track).}
\label{fig:plots_Tracks}
\end{figure}

\subsubsection{Plot message fields}
Each $Plot$ update contains fields that describe (1) the message properties, (2) the identified object properties, and (3) sequence-related properties.
The following is description of each one of them:
\begin{itemize}
    \item message properties - message metadata, such as the source ID, message ID, message length, etc.
    These fields are constant and do not change between $Plots$.
    \item identified object properties - describe the object location, type, etc.
    This properties could be divided into two group:
    (1) categorical - limited, fixed, number of possible values, with no mathematical metric, such as, the object type which describes whether the identified object is an airplane, cloud, etc.
    (2) numerical - unlimited numbers, with mathematical metric, these features mostly describe the object location and speed.
    \item sequence-related properties - have meaning only while considering a sequence of samples, such as updateTime, TrackID, etc.
\end{itemize}

A general radar system network can be connected to an organizational network, to a SCADA network that controls the utility services of the radar system (power, air condition, etc.), and in some scenarios to external entities over the Internet.
Thus, this system is exposed either to internal or external threats.

In addition, for closed radar systems which are physically isolated and thus do not communicate with external entities, we consider the following threats as well:
\begin{itemize}
    \item Possible insider threats - in many scenarios, the radar systems are integrated

    \efrat{need to complete...}
    \item Supply-chain threats - if the attacker is a supplier of one of the radar system component, or even if he's a supplier of a system that is connected to the radar system, the radar system is in particular exposed to threats.
    \item Lack of security - due to the assumption that the radar system is a standalone system which is physically isolated from any external network, such a system tend not to include critical security countermeasures~\cite{c1}.
    \item Lack of software updates - from similar reasons mentioned above, software updates that contain vulnerability fixes are not installed. Those might be exploited by attackers.
    An example of the lack of software updates at the ECDIS (Electronic Chart Display and Information System) which often deployed on vessels and used to automatically navigate the ship could be shown in~\cite{c7}
    \item System complexity - in many scenarios, the radar systems are integrated into multi-system platforms hence many components are connected to the radar network.
    Thus, an attacker may take an advantage of this complexity and exploit many possibilities to connect his own device to the network in order to perform attacks.
    \efrat{need to improve...}
\end{itemize}

In this section we provide with a brief description of the security aspects and threats as we presented in~\cite{cohen2019security}.

\textbf{Assets.} The assets of the radar systems that the attacker is interested in can be categorized as follows:
\begin{itemize}
    \item Physical assets: the physical components of the radar system (e.g. the antenna, the radar controller, etc.)
    \item Computational units: the subsystems responsible for data analyzing; For example, the signal processor and the tracking algorithm. Both are part of the radar controller. Another example is the analyzing unit which is a part of the control system.
    \item Data assets: the data present within the system (e.g. $Tracks$ or individual $Plots$).
    \item Connectivity assets: the physical components that are responsible for transferring data between physical components (e.g. the radar system network)
\end{itemize}

\textbf{Attacker Location.} The possible locations/positions of an attacker can be categorized as follows:
\begin{itemize}
    \item Physical access: the attacker has physical access to a physical asset of the radar system. In this case, the attacker can install malicious software or physically replace a legitimate device with a malicious device.
    \item Remote access: the attacker has access to the radar system via the Internet. In this case, the attacker can execute remote malicious commands on one or more of the radar system components.
    \item Close proximity: the attacker controls a device which is located near the antenna. The device has the ability to send signals at the radio domain.
    \item Supply chain: the attacker can execute a supply chain attack and compromise one or more of the components before the actual deployment.
\end{itemize}

\textbf{Threats.} The threats to radar systems are as follows:
\begin{itemize}
    \item Denial of service: harming the availability of a component of the radar system.
    \item Integrity violation: compromising the integrity of a component of the radar system and its data.
    \item Confidentiality violation: compromising the confidentiality of the data.
\end{itemize}

\section{\label{sec:security}Radar Systems Security}

Very little research has been conducted on the security of radar systems.
Although radar systems are vulnerable to a variety of threats~\cite{cohen2019security}, most of previous radar system related works focused on functionality aspects~\cite{barton2004radar,barton1988modern}.

The following are our assumption of the attacker possible locations/positions:

\begin{itemize}
    \item Radar Controller (1) - the attacker has physical access to the radar controller.
    Hence the attacker can install malicious software, replace one of the software components, or connect his/her malicious device.

    \item Radar system network (2) - the attacker has physical access to the radar system network, hence the attacker can replace the network switch, and/or connect his/her malicious device.

    \item External network (3) - the attacker has physical access to the external network of the radar.
    Hence the attacker can exploit vulnerabilities in one of the radar components in order
\end{itemize}

The effects of such tampering activity are expected to be covert; and, specifically, to be "under the radar" for the radar operators (i.e., radar operators should not be able to identify the tampering).

Based on the above, we place our anomaly detector between the radar system network to the CS.

Our proposed method consists of the following three stages:

(1) Data modeling - we utilize state of the art machine learning methods to model the data in order to be able to identify anomalies that potentially provide indications for a cyber attack.
(2) Anomalies creation - we analyze how the possible attack scenarios can be reflected in the data, and create a set of anomalous data accordingly.
(3) Model evaluation - we evaluate the model ability to distinguish between the created anomalous data and the benign data.

\subsection{{\label{subsec:extracted}}Extracted Feature}

In addition to the raw (basic) features provided in the stream of events, we compute additional feature.
We called this features $updatingPeriod$ and its value is computed using the feature $UpdateTime$.
This feature indicate of the $UpdateTime$ difference between the current $Plot$ and the previous $Plot$ of the same $Track$ (e.g., $currentUpdateTime - prevUpdateTime$).
For the first $Plot$ of the $Track$ we set the feature value to be the same value as the second $Plot$ of the $Track$, since for the first $Plot$ in the $Track$ there is no $prevUpdateTime$.
The main use of this feature is in tracking the time difference between $Plots$ so we can detect drooping events.

The feature values vary and range from 100 to more than 1,000,000 time units, this indicates that just setting a threshold on this feature could not be a good strategy for detecting a drooping events.
An histogram of this feature is presented in Figure~\ref{fig:updatingPeriod}.

\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{Fig/UpdatingPeriod histogram.png}
\caption{UpdatingPeriod histogram.}
\label{fig:updatingPeriod}
\end{figure}

\subsection{\label{subsec:model}Data Modeling}

\subsubsection{Model high level architecture}

The proposed model consist of the following two main phases, (presented in Figure~\ref{fig:HLA}): (1) Modeling of the $Plot$ structure. (2) Modeling of a sequence time intervals.
Following is a detailed description of the two phases.

\subsubsection{Modeling of the $Plot$ structure.}
The main goal of this phase is to examine whether the relation between the different features of the $Plot$ is similar to the relation that has been learned from benign $Plots$.
This phase is consist of the following three stages. (1) Categorical to numerical embedding, (2) plot-level point anomaly detector, and, (3) track-level anomaly detector.
In the next subsections we will describe each one of them.

\subsubsection{Categorical to numerical embedding}

AutoEncoder embedding representation is commonly used to create a short representation for categorical features~\cite{hancock2020survey}.
\shai{TODO: maybe need to explain about AutoEncoder}
Using this method an AutoEncoder is trained and the Encoder output is used as a representation of the input data.

One of the most famous example is Word2Vec technique which used in the natural language processing domain~\cite{c4}. (\shai{TODO: shai - maybe we need to drop this sentence.})

In our anomaly detection model, we utilize a similar method called Cat2Emb.
The categorical features in our data are multi-field features. which means that the relation between them is unknown.
Therefore, our designed Cat2Emb model was inspired by the models presented in~\cite{c5}~\cite{c6}.

The architecture of the Cat2Emb model, (presented in Figure~\ref{fig:cat2emb}), includes three parts:

\begin{enumerate}
    \item In the first part, there is a separate embedding layer for each of the categorical features.
    This embedding layer gets one integer as input and output one number.
    The number of possible values for the inputted integer is the number of possible values of the corresponding categorical feature \footnote{In some of the categorical features we shift the values before inputting them to the model, so the minimum values will be zero}.

    The embedded layer works in the following way:
    A one-hot-encoding representation/vector is created for the inputted value,
    This representation/vector is multiplied with a corresponded weights vector (W) and returned the single output value.
    \item In the second part, the nine outputs from the nine separate embedding layers are concatenated, creating a vector with a length of nine.
    \item In the third part, the concatenated vector is fully connected to nine separate vectors.
    Each of these nine vectors corresponds to a different feature.
    For this reason, the size of each of these nine vectors is equal to the number of possible values of the corresponded categorical feature.
    The activation function used in this layer is Sigmoid.
\end{enumerate}

As the input for the Cat2Emb network is a vector in a length of nine, the network goal is to reconstruct the values of the input vector such that there will be "1" in a single place in each of the 9 output vectors and "0" in all the other places.
In each output vector the "1" location is related to the input value in the correspond input.
To do so, we trained the network to minimize the SparseCategoricalCrossEntropy measure using Adam optimizer~\cite{c2}.
The global network loss is a sum of the loss of all of the nine final vectors.

In the resulting network, the concatenated vector is used as the embedding space (numerical) representation for the categorical features.
In this vector, each categorical feature is contribute to more then one element.
The intuition for this is as follows:
Each of the embedding layers represents several one-hot-encoding vectors by using only one number, then using these numbers the network tries to reproduce each of the inputs and present it using one-hot-encoding representation.
The network could not use only the one number created by a one embedding layer to reproduce its one-hot-encoding representation since the transformation from one number to several orthogonal vectors is not linear.
Therefore, to reproduce each of the one-hot-encoding vectors the network use more than one number.

\begin{figure*}[h]
\centering
\includegraphics[width=1.1\textwidth]{Fig/Embedding Representation.png}
\caption{Cat2Emb}
\label{fig:cat2emb}
\end{figure*}

\subsubsection{Plot-level anomaly detector}
By using the embedding representation for the categorical features created by the Cat2Emb model together with the numerical features, we use a stack autoencoder (AE) model for detecting anomalous plots.
The architecture of the this AE is presented in Figure~\ref{fig:fullModel}.

As can be seen, first, the embedding representation of categorical features (computed by the Cat2Emb model) is concatenated with the numerical features, resulting in a vector of size 27.
Then, a min-max scaler is applied to this vector to normalize all features.
Then, the normalized vector is fed into the stacked AE consisting of seven layers with 27, 20, 15, 10, 15, 20, 27 neurons respectively.
In all neurons the activation function used is Sigmoid.

The AE is trained using the training set to minimized  the mean square error measure (MSE) using Adam optimizer.

\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{Fig/Full Model.png}
\caption{Full Model}
\label{fig:fullModel}
\end{figure}

\subsubsection{Tracks anomaly detector.}
In most scenarios, the attacker gets a much greater benefit if he/she changes a feature of $Plots$ for the entire $Track$.
For example, an attacker who wants to change the type of the object identified by the radar (e.g., TrackCategory) so it will not be classified as an aircraft would prefer to do so for the entire $Track$.
The reason is that changing the object type for only several $Plot$ of the $Track$ has less influence in deciding which action should be taken, at the same time, the change of this feature during the $Track$ is not supposed to happen, and is very suspicious.
For this reason it is more accurate and will cause less false positives if the threshold will be decided by the mean loss of the $Track$, hence in this stage we calculate the mean loss to decide whether a feature is changed in a $Track$.
\shai{TODO: shai - maybe declare that this is another detection method.}

\subsubsection{Modeling of a sequence time intervals}
A $Plot$ that has been tested so none of its features have been changed is forward to the model second phase.
In this phase, we model the time between packets and try to understand whether the arrival time of the packet is legitimate.
This is done using the following two stages. (1) Data processing. and, (2) Sequence anomaly detector.
In the next subsections we will describe each one of them.

\subsubsection{Data processing}
At this stage, we concatenate the previous six $Plots$ of the same $Track$ as the current $Plot$ along with the current $Plot$, hence creating a sequence of a length of 7.
In this sequence we use five features, TargetRT, trackGrade, quality, trackCategory, and updateTime and process the data in the following way:
(1) Encode the categorical features (i.e., trackGrade, quality, trackCategory) using one-hot-encoding.
(2) Extract UpdatingPeriod using updateTime.
(3) Drop the first element of the sequence ending up with sequence in a length of 6.
(4) Apply min max scaler on the data for feature scaling.
(5) take the 5 first element of the sequence to be the input for the model and the value of $updating period$ feature of the sixth element to be the expected/predicted value.

\subsubsection{Time interval anomaly detector}
Long Short Term Memory (LSTM) networks are well-known deep learning networks that are very suited for sequential data.
In this type of network instead of processing the whole sequence together, the network process each object of the sequence separately in chronological order.
So in each step, the network inputs are: (1) the current object in the sequence, together with the (2) output of the network from the previous step.

By using an LSTM component we design a simple LSTM based regressor.
This LSTM regressor gets as an input the processed sequential data created in the previous stage and predict the $updating period$ (e.g., time interval) value of the next $Plot$ of the sequence.

The architecture of this regressor is presented in Figure~\ref{fig:lstm}.
As can be seen, the 5 length sequence is inputted into an LSTM network with 5 hidden units.
The output of the LSTM component at the final step is fully connected to one neuron.
At the end, a Sigmoid activation function is applied on this neuron.
The network is trained using the training set to minimized the mean square error measure (MSE) using Adam optimizer.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{Fig/LSTM network.png}
\caption{LSTM based regressor}
\label{fig:lstm}
\end{figure}

\shai{TODO: shai - add a section to explain how we give score to samples.}

We conducted the following three types of evaluations:
\begin{itemize}
    \item \textbf{Cross-session setup:} in this case, each time we used four recording sessions for training and the fifth remaining session was used for testing the model.

    \item \textbf{Chronological setup:} in this case, for each recording session we train the model on the first 90\% instances (by chronological order) and test it on the remaining 10\% instances.

    \item \textbf{Transfer learning setup:} this case is a combination of the previous two setups in which we used four recording sessions as well as the first 10\% of the instances of the fifth session for training the model, and the remaining 90\% instances of the fifth session were used for testing the model.
\end{itemize}

In all of these cases we repeated the experiments five time (each time a different recording session was selected for testing).
For example, in a scenario in which the examine session is R5, the following are the train and test data in each evaluation setup:
\begin{itemize}
    \item \textbf{Cross-session setup:} train -  R1, R2, R3 and R4 ; test - R5.
    \item \textbf{Chronological setup:} train - 90\% of first instances in R5 ; test - remaining 10\% instances of R5.
    \item \textbf{Transfer setup:} train -  R1, R2, R3, R4 and first 10\% instances of R5 ; test - remaining 90\% instances of R5.
\end{itemize}

\shai{TODO: combine the above two items}

\section{\label{sec:evaluation}Evaluation}
In order to evaluate the models, we implemented four attack scenarios and conducted 15 different experiments for each attack.

\subsection{\label{subsec:dataset}Benign Dataset for Training}

For the evaluation we use benign data collected from five real radar systems which are deployed on different operational setups and used for identifying objects.
We refer to each dataset as a \emph{recording session}.
The number of messages in each recording session is as follows: R1 -- 45,720 (43\%), R2 -- 23,017 (21.7\%), R3 -- 15,992 (15.1\%), R4 -- 11,955 (11.2\%), and R5 -- 9,551 (9\%).
In total, there are 106,235 messages.

\begin{comment}
\begin{table}[h!]
\centering
\begin{tabular}{|l|l|l|}
\hline
\textbf{Recording session} & \textbf{Number of messages} & \textbf{Percentage} \\
\hline
R1 & 45,720   & 43\% \\ \hline
R2 & 23,017    & 21.7\% \\ \hline
R3 & 15,992    & 15.1\% \\ \hline
R4 & 11,955    & 11.2\% \\ \hline
R5 & 9,551    & 9\% \\ \hline
\end{tabular}
\caption{Dataset description.}\label{tab:dataset}
\end{table}
\end{comment}

\subsection{\label{subsec:attacks} Attack Scenarios Implemented}
A labeled testset which includes both benign and malicious samples is needed to evaluate the proposed model.
Since our dataset does not include any attacks, we had to simulated the attacks and inject them into our dataset.
In the evaluation we focused on two classes of attacks that an attacker could benefit from.

\subsubsection{Categorical feature manipulation.}
This is an integrity attack in which the attacker changes one categorical feature.
Changes in the following features may cause the consuming system to characterize and handle the detected object incorrectly.
\begin{itemize}
    \item trackCategory - in this scenario, the attacker changes the category of the object, such that air objects will be seen as ground objects and vice versa.
    This may change the way the radar operator is reacting to the different objects and may help an attacker be left untreated by the radar operator.
    \item threatAlert - in this scenario, the attacker change the hostility level identified by the radar, such that friend object will be seen as foe objects, etc.
    This attack may case the radar operators to fire on friend objects or ignore enemy objects.
    \item hostility - in this scenario, the attacker change the alert that will be raised as a result of object identification, such that alert will not raised when needed and vice versa,
    This attack may create many false alarms and may also cause the radar operator to ignore foe objects.
\end{itemize}

Changing features for an entire $Track$ or a $Track$ segment will be much more beneficial to the attacker than changing features for random $Plots$.
This is because changes in the features of random $Plots$ may be ignored by the radar operator and are not likely to influence the action taken by an operator.
Therefore, in this attack scenario, we evaluate our model against changes in the entire $Track$.

The following process was used to generate the attack: (1) given a test set $D_{test}$, we first duplicate the test set; (2) in the duplicated test set, for an examined (manipulated) feature $f$ for each track $t$, we change the value of $f$ \shai{to a possible value occurred least in the $t$}, for the whole track; (3) we label the manipulated events/plots as anomalies; and (4) we combine the test set $D_{test}$ and the duplicated, manipulated data into one test set.
This process generates a similar number of benign and malicious events/plots; therefore, the resulting test set is balanced.

\subsubsection{Drop plots.} Dropping of one/several plots from a track, so that the consuming system will be less confident in the track/object that has been detected.

The process for creating this attack is as follows:
(1) given a test set $D_{test}$, we first duplicate the test set; (2) for each track $t$ in the duplicated test set, we select a random location\footnote{The random location selected must be over a given threshold (e.g., five), since previous instances are needed in order to detect the attack. In addition, this number needs to be lower than $length(t) - 2$, since we need to keep the last plot at the end of $t$ for labeling.} in $t$; (3) from this random location, we count the number of packets that there are until the last packet of the Track and denote this number as $c$; (4) we drop a number of $minimum(c,10)$ consecutive packets from the randomly selected location of $t$; (5) the event/plot that follows the dropped plots is labeled as an anomaly; and (6) we combine the test set $D_{test}$ and the duplicated, manipulated data into one test set.
From this test set, we drop duplicates.
This process generates a different number of benign and malicious events/plots; therefore the resulting test set is imbalanced.

\subsection{\label{subsec:evalsetup}Evaluation Method}

We conducted three types of evaluations; in each case, we repeated the experiments five times (each time a different recording session was selected for testing).
A description of the types of evaluations performed is provided below, along with an example of the training/testing data in a scenario in which the R5 session is examined:
\begin{itemize}
    \item \textbf{Cross-session setup:} in this case, each time we used four recording sessions for training, and the fifth session was used to test the model.
    Example for the R5 session: training -  R1, R2, R3, and R4; testing - R5.
    \item \textbf{Chronological setup:} in this case, for each recording session, we trained the model on the first 90\% of the instances (in chronological order) and tested it on the remaining 10\% of the instances.
    Example for the R5 session: training - 90\% of the instances of R5; testing - remaining 10\% of the instances of R5.
    \item \textbf{Transfer learning setup:} this case is a combination of the previous two setups, in which we used four recording sessions, as well as the first 10\% of the instances of the fifth session to train the model, and the remaining 90\% of the instances of the fifth session were used to test the model.
    Example for the R5 session: training -  R1, R2, R3, R4, and the first 10\% of the instances of R5; testing - the remaining 90\% of the instances of R5.
\end{itemize}
\shai{It should be noted that the cross-session evaluation setup has two significant advantages over the other two: (1) in practice, it is easier for radar engineers to deploy a model that has already been trained rather than training a model using the new system data, and (2) training the model on the new system data can expose the model to cyber risk (e.g., adversarial poisoning ~\cite{vorobeychik2018adversarial}).}

\subsection{\label{subsec:evalmetric}Evaluation Metrics}

The task of identifying anomalies in the radar data stream is a classification task, where benign samples are labeled as zero (negative), and malicious data is labeled as one (positive).
We used the following common metrics for the evaluation: True Positive Rate (TPR)/Recall, False Positive Rate (FPR), Receiver Operating Characteristic (ROC) Curve, Area Under the ROC Curve (AUC), Precision, Precision-Recall Curve, and average precision (AP).

\begin{comment}
\begin{itemize}
    \item \textbf{True Positive Rate (TPR)/Recall} - the ratio of the number of malicious samples identified by the model that are actually positive (TP) to the total number of malicious samples in the test data (P).
    \item \textbf{False Positive Rate (FPR)} - the ratio of the number of benign samples that were misclassified by the model (FP) as malicious to the total number of benign samples in the test data (N).
    \item \textbf{ROC (Receiver Operating Characteristic) Curve} - since the model returns the probability for a sample to be malicious, the number of benign/malicious samples predicted by the model is dependent on the probability threshold.
    When this threshold increases, the TPR increases and the FPR decreases, and vice versa.
    The ROC curve represents the relation between the TPR to the FPR for different thresholds.
    \item \textbf{AUC (Area Under Curve)} - the area under the ROC curve; a number ranging from zero to one that represents the model quality.
    \item \textbf{Precision} -  the ratio of the number of malicious samples identified by the model that are actually positive (TP) to the total number of samples that were predicted by the model to be positive (TP + FP).
    \item \textbf{PRC (Precision Recall Curve)} - represents the relation between the $Precision$ to the $Recall$ for different thresholds.
    \item \textbf{AP (Average Precision)} - weighted mean of the $Precision$ for each threshold, i.e., $\sum_n(R_n - R_{n-1})*P_n$; a number ranging from zero to one that represents the model quality.
\end{itemize}
\end{comment}

\subsection{\label{subsec:results}Results}
The following are the results for each attack scenario:

\textbf{Categorical feature manipulation attack results.}
We evaluate the purposed model ability to detect manipulation attacks on the features (1) trackCategory, (2) hostility, and (3) threatAlert,
We chose these features since manipulation of these features can help an attacker avoid detection from the radar system/system operators which is one of the most significant goals an attacker may want to achieve.
In these scenarios, the test set is balanced (i.e., contains similar number of malicious and benign samples/$Tracks$).

\textbf{Manipulating of trackCategory.}
Figure~\ref{fig:trackCategory} present the detection results of the track-level detection module on this attack.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{Fig/Evaluation Results - Change Feature - trackCategory.png}
\caption{The ROC and PRC curves for the trackCategory change feature attack for each session examined.}
\label{fig:trackCategory}
\end{figure}

\textbf{Manipulating of hostility.}
Figure~\ref{fig:hostility} present the detection results of the track-level detection module.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{Fig/Evaluation Results - Change Feature - hostility.png}
\caption{The ROC and PRC curves for the hostility change feature attack for each session examined.}
\label{fig:hostility}
\end{figure}

\textbf{Manipulating of threatAlert.}

Figure~\ref{fig:threatAlert} present the detection results of the track-level detection module.

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{Fig/Evaluation Results - Change Feature - threatAlert.png}
\caption{The ROC and PRC curves for the threatAlert change feature attack for each session examined.}
\label{fig:threatAlert}
\end{figure}

\textbf{Drop packets attack results.}
Table~\ref{tab:drop_packet} present the distribution of the benign/malicious samples in the drop packet attack.
It's could be noticed that the data is imbalanced, since for each $Track$ we execute the attack only once.
The performance of the Time interval detection module on the generated dataset is presented in Figure~\ref{fig:drop_packet}
As could be noticed, in all scenarios the model achieved significant results.

\begin{table}[h]
\centering
\small
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|}
\hline
\multirow{2}{*}{}             & \multicolumn{10}{c|}{\textbf{Session Examined}}                                                                      \\ \cline{2-11}
                              & \multicolumn{2}{c|}{\textbf{R1}}     & \multicolumn{2}{c|}{\textbf{R2}}     & \multicolumn{2}{c|}{\textbf{R3}}
                              & \multicolumn{2}{c|}{\textbf{R4}}
                              & \multicolumn{2}{c|}{\textbf{R5}}\\ \hline
                              & \textbf{b} & \textbf{m} & \textbf{b} & \textbf{m} & \textbf{b} & \textbf{m} &
                         \textbf{b} & \textbf{m} & \textbf{b} & \textbf{m}\\ \hline
Cross Directories    & 46,600          & 213                  & 23,068               & 177                  & 16,237               & 122  & 12,139 & 97 & 9,484 & 119             \\ \hline
Chronological (10\%) & 4,606               & 60                  & 2,323               & 34                  & 1,609               & 24 & 1,206 & 15 & 842 & 31               \\ \hline
Transfer (90\%)      & 42,709 & 195  & 21,126     & 170                  & 14,660               & 105 & 10,808 & 86 & 8,451 & 110                 \\ \hline

\end{tabular}
\caption{Drop packet benign (b) and malicious (m) number of samples.}
\label{tab:drop_packet}
\end{table}

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{Fig/Evaluation Results - Drop Packet.png}
\caption{The ROC and PRC curves for the drop packet attack for each session examined.}
\label{fig:drop_packet}
\end{figure}

\textbf{Setting threshold.}
\shai{We set two thresholds that achieved FPR of 0.01 and 0.05, then for each attack scenario, we averaged the TPR on the five experiments.
The results are presented in Table~\ref{tab:results_summary}.}

\shai{It's could be noticed that the model successfully classify almost all of the drop packets attack even when allowing a FPR of 0.01 and most of the change feature attacks when allowing FPR of 0.05.}

\shai{In the drop packets attack, there is no much difference between the different evaluation setups.
In the change feature attack when allowing FPR of 0.01, the models that trained in the cross directories setup, achieved competitive results compared to the two other setups.
And when allowing an FPR of 0.05, these models achieved even better results compared to the two other setups.
This may indicate that (1) the data created for the different sessions has a very similar distribution.
(2) Some of the samples in the new tested session are from a distribution that is different from the other sessions, hence they are more likely to be labeled as malicious by the model that is trained in the cross sessions setup.
In conclusion, based on our results, we suggested to train the model on the different systems and to not retrain it on the new system.}

\begin{table}[h]
\centering
\begin{tabular}{|l|l|l|l|l|l|}
\hline
\small
\textbf{Attack scenario} & \textbf{Test type}   & \textbf{\begin{tabular}[c]{@{}l@{}}Average TPR \\ (FPR=0.01)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}l@{}}Average TRP \\ (FPR=0.05)\end{tabular}} \\ \hline
Change feature - trackCategory & Cross directories    & 0.38                                                                       & 0.75                                                                        \\ \hline
Change feature - trackCategory & Chronological (10\%) & 0.25                                                                       & 0.48                                                                        \\ \hline
Change feature - trackCategory & Transfer (90\%)      & 0.25                                                                       & 0.56                                                                        \\ \hline
Change feature - hostility     & Cross directories    & 0.15                                                                       & 0.48                                                                        \\ \hline
Change feature - hostility     & Chronological (10\%) & 0.3                                                                        & 0.63                                                                        \\ \hline
Change feature - hostility     & Transfer (90\%)      & 0.22                                                                       & 0.52                                                                        \\ \hline
Change feature - threatAlert   & Cross directories    & 0.34                                                                       & 0.81                                                                        \\ \hline
Change feature - threatAlert   & Chronological (10\%) & 0.52                                                                       & 0.65                                                                        \\ \hline
Change feature - threatAlert   & Transfer (90\%)      & 0.61                                                                       & 0.61                                                                        \\ \hline
Drop packets                   & Cross directories    & 0.88                                                                       & 0.97                                                                        \\ \hline
Drop packets                   & Chronological (10\%) & 0.92                                                                       & 0.99                                                                        \\ \hline
Drop packets                   & Transfer (90\%)      & 0.89                                                                       & 0.95                                                                        \\ \hline
\end{tabular}
\caption{\shai{The average TPR for each attack scenario when setting a threshold such that it achieved FPR is 0.01 and 0.05.}}\label{tab:results_summary}
\end{table}