\section{Introduction}
The recognition of human behavior is important for many domains, like sports, the analysis of manual work processes, or as a component for situation-aware assistants. As another example, sensor-based Human Activity Recognition (HAR) can also be used for the automatic assessment of behavioral symptoms of people with dementia, like apathy or agitation. In contrast to questionnaires like the Cohen-Mansfield Agitation Inventory \citep{cmai}, HAR allows the real-time, objective assessment of symptoms, supporting caregivers in selecting appropriate interventions.
However, accurate HAR is challenging due to the high inter- and intra-subject variability of movement. Specifically, the automatic recognition of challenging behavior from wearable sensor data has been of limited success so far \citep{harchall2}.

Multiple approaches have been proposed to improve HAR accuracy, by making use of additional external knowledge \citep{leehar, xie, yordanovacooking, rueda, luedtkenn, inoue}.
In this paper, we investigate how HAR performance can be improved by making use of the fact that there is a tendency of specific subjects to show specific behaviors at certain times of the day or week. For example, sundowning is the increase in restlessness of some people with dementia in the evening \citep{sundowning}.
The knowledge of such behavioral predispositions (BPDs) could serve as a useful prior to increase HAR accuracy.

\begin{figure*}
  \centering
  \includegraphics[width=\linewidth]{figures/modelv2.pdf}
  \caption{Overview of the concept. We calculate the distribution of annotations for a given segment of the day (histograms). We then cluster these distributions. Next, we use the cluster label (BPD) to select the classifier (color indicates the corresponding classifier). The classifier predicts the behavior from the features of the motion sensor data.}
  \label{fig:overview}
\end{figure*}

Previous work relied on external knowledge to define BPDs. For example, L\"udtke et al. \citep{luedtkenn} partitioned warehouse order picking into distinct process steps, the knowledge of which can inform HAR. However, in the context of dementia care, there are no established BPDs that are externally given.
The contribution of this paper is to identify such BPDs in a data-driven way:  We calculate the distribution of annotations in time segments of a day, and then cluster these histograms. This results in clusters of time frames in which the subjects behaved similarly. Each cluster forms a BPD. Given the BPD, we then train a classifier specifically for data recorded within the BPD, i.e., there is one classifier for each cluster. This classifier is thus biased towards the more frequent classes in the BPD. An overview of this process can be found in figure \ref{fig:overview}.  In summary, this paper provides the following contributions:
\begin{itemize}
\item A clustering-based method for the data-driven construction of behavioral predispositions (BPDs)
\item An approach for utilizing these BPDs in HAR
\item An evaluation of this approach for the sensor-based recognition of behavioral symptom of persons with dementia
\end{itemize}
Empirically, our approach leads to a substantial increase of HAR performance, when assuming that the BPD for each time segment is given.

\section{Data and Observations}
This section presents the insideDem framework, as well as the data set. We make observations about the distribution of annotations given time frames, which will then lead to the construction of the BPDs and the utilization of them in the methodology section \ref{sec:methods}.

\subsection{insideDem Project}
Part of the insideDem framework was the recording of the behaviours of dementia diagnozed patients \citep{teipelinsidedem}. In two nursing homes, real-time observations and sensor data of a total of 17 residents (11 women and 6 men) with moderate to severe dementia were recorded. The behaviour of persons in one nursing home was also recorded on video. The age range was 73 to 94 years. All persons were under psycho-pharmacological treatment. Dementia medication was taken by 8 residents of the nursing homes. The behaviour of the persons was recorded in real time by an expert in the sitting room of the nursing home. This room was accessible to all residents and their visitors as well as the staff of the nursing home. The behaviour was annotated every five minutes following the annotation scheme which was developed for the insideDem project. This scheme is based on the neuropsychiatric inventory \citep{npi} and the Cohens-Mansfield agitation inventory \citep{cmai}. It consists of seven different behaviours: apathy, general restlessness, mannerisms, pacing, aggression, trying to get to a different place, normal behaviour.

During the day, between 8 a.m. and 6 p.m., the subjects wore two wristbands, which were attached to the dominant hand and the ankle. During the night, a band was worn on the ankle. The sensors recorded accelerations and rotations as well as the sound level of the environment, the light conditions and the sound pressure. In this work, we only use the acceleration sensor data.

\subsection{Observations}
Figure \ref{fig:hist} shows the distribution of annotations for each subject. The behaviour of the eight patients were observed over the course of 29 days and accumulate around 17,000 annotated five-minute intervals. For every subject the the behaviours occur in different portions. As example, the patient X111 showed 9\% apathy where as X126 showed nearly 50\% apathy during the time span of 29 days. However, for  X111, pacing was mostly annotated but X126 nearly showed no pacing.

\begin{figure}
  \centering
  \includegraphics[width=0.8\linewidth]{figures/histogram.pdf}
  \caption{Distribution of annotations per person over all days of recordings.}
  \label{fig:hist}
\end{figure}

In the literature events for triggering agitated behaviors were observed. The sunset phenomenon as described in \citep{sundowning} may be such a trigger for one BPD. At sunset people with dementia often show more agitated behavior than at other times of the day. Also, the fact of boredom can trigger agitated behavior. If there is nothing planned for an afternoon, dementia diagnosed persons might show agitated behavior because of the boredom \citep{trigger}. As a result, it is reasonable to look into noticeable time frames of the day which show one person behaving agitated more often.

The figure \ref{fig:hist_violin} (a) now shows the average occurrences of the annotations for one-hour segments of the day. The represented subject behaves in the morning more apathetic than in the later course of the day. Also at noon, the patient shows more normal behavior. Pacing is mostly happening in the afternoon. Figure \ref{fig:hist_violin} (b) shows the variances of the annotations between each day. This plot represents the distribution of the proportion each behavior has on the time segment every day. Between 10 and 11 a.m. we saw the patient behaves apathetically to 46\% in plot (a). However, in plot (b) we see that the apathy part can differ from 0\% to 100\% with equal probability from day to day.

\begin{figure}
  \centering
  \includegraphics[width=0.8\linewidth]{figures/hist_violin.pdf}
  \caption{Percentual distribution of annotations for one subject and one-hour intervals of the day. (a) shows the average of the annotation and (b) describes the distribution (variance).}
  \label{fig:hist_violin}
\end{figure}

\section{Methods}
\label{sec:methods}
Behavioural predispositions (BPD) describe the tendency to show specific behaviours. In this section, we propose a method to form such BPDs from the annotations of behaviors. In the following, we present a simple model that can use these newly found BPDs to simplify the classification.

\subsection{Extracting behavioural predispositions}
BPD describe the urge to show specific kinds of behaviors. Knowing such would intuitively mean that the prediction of the current actual behavior becomes easier since it provides an informative prior on the behavior of the subject. Unfortunately an expert model describing BPD for patients with dementia does not exist and thus the aim is to derive them from the data of the insideDem project.

\subsubsection{Time-based BPD}

Following the observations of figure \ref{fig:hist_violin} (a), one can argue to use a similar approach to \citep{inoue} or define that every BPD is one segment of the day. This means from 10 to 11 a.m. the subject always behaves similarly and by knowing the current time, we can use this information to improve the classification. However, this approach has the drawback of high variances of the annotations between each day which lead to BPDs that do not carry a lot of information about the actual behavior.

\subsubsection{Clustering-based BPD}

To overcome the problem of high variances, we propose to find similar distributions globally over all days and time segments instead of grouping them by the time of the day. Let $p(A_t)$ be the probability distribution of the annotation for the day and time segment $t$. Note that $t$ describes a specific time frame e.g. 06--15 10:00 to 06--15 11:00. This categorical distribution can be obtained by counting the relative frequencies of the annotations in the segment. This counted distribution is called a histogram. The BPD represents the tendency to behave in a certain way. Given a set of histograms, the task is to divide this set into groups. These groups should contain similar histograms. With the help of the k-means clustering algorithm, clusters of histograms can be generated which are grouped according to their spatial similarity (Euclidean distance). Each of these clusters describes a behavioral disposition. The histograms of the cluster $D=d$ therefore represent the distribution $p(A \, | \, D = d)$.

\subsection{Classification of Sensor Data}
In the scope of this work, we want to show what impact the complete knowledge of the clustered BPD can have on the classification. Therefore, we assume the BPD, i.e. the cluster, as given. Note that this would not be the case in a real world environment. Given a BPD $d$, we then have a prior knowledge of the annotation $p(A \, | \, d) $. To make use of this knowledge, we train a different classifier for each BPD. Thus, each classifier is trained with biased data and is therefore biased to more likely predict the classes that are most presently in this BPD. The classifier can be seen as expert for its BPD. The model $m$ maps sensor data $s$ and the BPD $d$ to an annotation $a$ by choosing the classifier $c_d$ that was specifically trained for $d$. The classifier then maps $s$ to $a$:
\begin{displaymath}
  m(s,d) = c_d(s) = a \, .
\end{displaymath}

\section{Experimental Evaluation}
\label{sec:eval}
The overall goal of the experiments are the evaluation of the impact of the  BPD on the HAR performance. In the following, we describe the experimental design in more detail.

\subsection{Experimental Design}
\begin{table*}
\centering
\caption{Factors and levels of the experimental design}
\begin{tabular}{p{2cm}p{5cm}p{6cm}}
\toprule
Factors & Levels & Description \\
\midrule
Clustering & K-means, time-based & Methods for generating the BPDs \\
\#{}-clusters & 1, \dots , 20 & Number of BPDs \\
Segment lengths & 0, \dots , 120 minutes (k-means)  & Length of the segment over which the histogram is calculated, only for k-means approach \\
Classifiers & Majority, naive Bayes, SVM & Classifiers to predict the annotation from sensor data \\
Subjects & X110, X111, X113, X114, X121, X122, X124, X126 & One model for each subject \\
\bottomrule
\end{tabular}
\label{tab:parameter}
\end{table*}
To evaluate the impact of the knowledge of the BPDs on HAR performance, we use a factorial design. The factors and levels are presented in table \ref{tab:parameter}. For each subject, a model with a different clustering method, number of clusters (i.e. BPDs), segment length, and classifier is trained. We randomly select 70\% of the data for training and 30\% for testing the model. Every combination of the levels is evaluated 10 times. Different clusters and segment lengths are used to test the expressiveness of the BPD. The segment length plays an important role for the k-means clustering strategy as the length is used to define the size over which the histogram is calculated and then clustered. At a segment length of 30 minutes, we obtain around 400 histograms per subject. We then cluster the histograms of a single subject, resulting in BPDs specifically for this subject. High number of clusters and small segment lengths yield finer BPDs. Note that one cluster means we have no further information about the BPD and thus gives us a baseline of performance. The time-based clustering method describes that a BPD only depends on the time. E.g., having 10 clusters and time-based clustering this method would split the day into 10 parts and for every part, one classifier is trained. We use three different methods for classifying the sensor data. The majority method functions as a baseline by always outputting the most frequent annotation of the BPD. The naive Bayes (NB) method shows the performance of a simplistic classifier. The NB uses for every BPD a different prior and observation model. Thirdly, a support vector machine (SVM) is trained. For preprocessing the accelerator data, features were calculated following the work of \citep{features}. The original data is sampled in 50 Hz. Feature were calculated over a window size of one minute and an overlap of 50\%. Statistical features, spatial relations between the three axis and frequency features were used. As measure of accuracy, we use $F_1$ score.

\subsection{Results and Discussion}

\begin{figure*}
  \centering
  \includegraphics[width=\linewidth]{figures/classifier_vs_sizes2.pdf}
  \caption{Averaged $F_1$ score for varying number of clusters and segment lengths. For time-based clustering, the segment length is implicitly given through the number of clusters. For k-means clustering, the segment lengths of the histograms are represented by the color. The segment length is in minutes. Rectangles denote similar BPD structures because of the same number of clusters and similar segment lengths.}
  \label{fig:f1s}
\end{figure*}

Figure \ref{fig:f1s} shows the average $F_1$ scores for each classifier and the strategies used for generating the BPDs. Firstly, we want to investigate only the k-means approach. The SVM shows an increase of $F_1$ score up to a number of about 10 clusters. From this point on, only smaller segment sizes benefit. For the majority classifier, the $F_1$ score no longer increases significantly above a number of 6 clusters. The naive Bayes classifier shows a nearly linear increase. Thus, for all classifiers, the $F_1$ measure increases with a larger number of clusters, as well as a decreasing segment length. The fact that the majority classifier benefits from this approach comes as no surprise. With an increasing number of clusters and finer segment lengths, the most frequent annotation occupies a higher portion of data within the cluster. Predicting only the most frequent annotation yields an $F_1$ score of around 0.1 whereas dividing the data by the BPD yields a score of around 0.5 at 20 BPDs and a segment length of 30 minutes. Note that some BPDs only describe one annotation and thus predicting can achieve 100\% accuracy by always outputting that one same class. According to this observation, the classification task should become simpler. Given such BPD with only one class, all classifiers used here can benefit and improve with knowledge of the BPD. However, the $F_1$ score of the majority classifier proposes that the knowledge of the most frequent annotation in the training data does not give us further advantages when we have more than 6 clusters. Yet the $F_1$ score of the naive Bayes classifier increases nearly linearly. Also, the SVM has a still increasing $F_1$ score at 20 BPDs, which underlines the fact that knowing the BPD simplifies the classification task by reducing the complexity of possible behaviors or even ruling them out. Note that the knowledge presented by the BPD converges with the BPD describing the exact underlying histogram, i.e. we have the same number of BPDs (clusters) as histograms.

We now investigate the time-based approach. The time-based approach shows an increase of $F_1$ score as the number of clusters rises. This can be observed for every classifier evaluated. Although the increase is not high, the knowledge of the clock time can help the classification. Thus dividing the day into segments and training a classifier for every segment each, increases the $F_1$ score. Comparing this approach with the k-means approach can be done by comparing the points where the number of BPDs and the segment length are approximately equal for both. E.g. splitting the day, where the subjects were recorded, into 20 segments yields a segment length of around 30 minutes and 20 BPDs. These points are denoted with rectangles in figure \ref{fig:f1s}. The plot yields a better performance of the k-means approach for all 4 points. The inferiority of the time-based approach can be explained through the high variances of annotations among the days (shown in figure \ref{fig:hist_violin}). The k-means approach, on the other hand, describes the histograms by their similarity and thus achieving less variance within the cluster. Having less variance means that the classifier has an easier task to solve.

\begin{figure}
  \centering
  \includegraphics[width=0.6\linewidth]{figures/confusion.pdf}
  \caption{Confusion matrix of the predicted behaviors using 20 BPDs and thus 20 SVMs. A segment length of 30 minutes were used to calculate the histrogams.}
  \label{fig:confusion}
\end{figure}

Overall, the $F_1$ score reaches a maximum of around 0.5. This score is produced by using 20 BPDs, a segment length of 30 minutes, and an SVM as a classifier. The same results can be achieved by using the majority method. The SVM outperforms the majority classifier for larger segment lengths. Figure \ref{fig:confusion} shows the confusion matrix of the SVMs for the classification with 20 BPDs and a segment length of 30 minutes.
We can see that the classes mannerisms, apathy, and normal behavior represent the majority of the annotations. Furthermore, there is great uncertainty between these classes. This can be explained by the similarity of these behaviors in this scenario. Intuitively, normal behavior is similar to apathy, as dementia patients in the nursing home live more secluded. Mannerisms are difficult to recognize when, for example, they are performed with the non-dominant hand (sensor is worn at the dominant hand). Another uncertainty is between the annotations pacing, getting to a different place, and normal behavior. This can be explained by the fact that these three annotations can be understood as the activity of walking so a misclassification can occur between these classes. The behavior is also annotated in five-minute intervals, but the classification takes place in 30-second intervals. This means that the person may have walked at one point in time, but the interval was annotated differently. Another reason is the exclusive use of the motion sensor on the wrist.

\section{Future work}
The evaluation showed that the knowledge of the BPD can yield better classification and thus further investigations are of interest. There is the question of whether finding the BPD with k-means clustering is the best choice. Since similar histograms and thus categorical probability distributions are of interest, clustering methods with entropy-based distances, such as the Jensen-Shannon divergence, are of interest. Furthermore, Latent Dirichlet Allocation (LDA) \citep{lda} can be investigated as a method for determining behavioral dispositions. The LDA is a generative probabilistic model and is mainly used in applications to assign a topic to documents based on their words. In this scenario, a segment of the day can be regarded as a document and its words are the annotations in this segment so that the resulting topics describe the BPD.

Currently, the BPD is only constructed by the annotations. A next step could be to determine the BPD based on the annotation as well as the sensor data. This would lead to BPDs that also hold information about the current motion data and its connection to the annotation. In this work, we created BPDs for a single person. As a next step, the evaluation of global BPDs across the patients is needed.

We also saw that the overall accuracy of the classification was low and thus looking into better options for classification is of interest. In this work, we only presented the approach of training a classifier for each BPD. However, this reduces the number of samples for the classifier. Another approach can be the feeding of the BPD label as input to the classifier. Also, classifiers that can make use of distributions over BPDs are of interest. Methods like mixed-effects neural networks \citep{menet} can model BPDs as another feature for activity recognition.

In reality, the BPD is not available and has to be found from sensor data. A good estimation of the BPD is therefore needed to improve the classification. With an increasing number of BPDs, the error of classification will also rise. Thus finding a good number of BPDs and a model to predict them is a crucial point for practical use. This can be done by using hierarchical models which first infer the BPD from the sensor data and then make use of the relations between behavior and BPD. Also, hidden Markov models can be of interest to exploit possible temporal properties of the BPDs.

\section{Related work}
To improve the classification of sensor data using additional information, firstly, the knowledge has to be modeled and perhaps has to be generated beforehand. Depending on the domain, the kind of knowledge can differ and thus arise different ways of incorporation into human activity recognition. Mostly the improvement of accuracy is the main purpose. However, the simplification of the recognition model can also be of interest. The authors of \citep{leehar} are aiming to create a model that still maintains the same level of accuracy under less training data than state-of-the-art methods. They use region connection calculus (RCC) \citep{rcc} to describe the spatial relations between objects and formalize the activities and their causal properties within the domain using the planning domain definition language (PDDL) \citep{pddl}. A probabilistic concurrent constraint automaton (PCCA) \citep{pcca} is generated from the PDDL model and reasons about the predicates from the RCC model. The RCC model calculates these predicates from high-level observations. The authors show that their approach can yield similar results in comparison to convolutional neural networks (CNN) but avoid the training data hungry nature of neural networks.
On the other hand, if enough training data and further information about the domain is available, the work of Xie et al. \citep{xie} aims to improve the accuracy of classification by incorporating prior knowledge in the training process of a neural network. They achieve this goal by modeling rules about the practicability of actions using linear temporal logics (LTL) \citep{ltl}. To overcome the problem of the multi-interpretable nature of LTL, the author convert them into deterministic finite automatons such that they can be embedded as hierarchical graph into the loss calculation of the neural network's training process and achieve learning consistent to the LTL rules and better accuracy at classification.
The works by Yordanaova et al. \citep{yordanovacooking} and Rueda et al. \citep{rueda} use computational causal behavioral models (CCBM) \citep{ccbm} to model domain knowledge and reason about the current state of the environment. CCBM also uses PPDL-like rules to describe the domain-specific properties of the activities. Also, the relations between user and object as well as durations of activities can be modeled in CCBM. As an observation model, a decision tree (DT) in \citep{yordanovacooking} and a convolutional neural network in \citep{rueda} are used to weight the predicted state based on the sensor data.
In contrast to these works, the prior knowledge does not need to be embedded directly in the model. L\"udtke et al. \citep{luedtkenn} demonstrate how providing external \emph{context information}  impacts classification performance. Specifically, they show that providing the current process step of structured activities can improve HAR performance, even when that information is noisy.
Even if no external information is available, the authors of \citep{inoue} made the observation that activities feature different quantities in times of a day. Thus segmenting the day and reasoning about the activities in one segment yields better accuracy than reasoning about the activities of the whole day.

\section{Conclusion}
In this paper, we proposed a data-driven method to construct behavioral predispositions and use them as informative prior to improve the performance of sensor-based human activity recognition. We presented a clustering-based approach that can determine time segments that display similar behaviors of a dementia-diagnosed person. We found such groups by first calculating the distribution of the annotations in the time segment and then clustering these distributions. The resulting clusters explain behavioral predispositions (BPDs). With complete knowledge of the current BPD, we demonstrated empirically that a substantial increase of the $F_1$ score can be achieved by using classifiers that were specifically trained for each BPD.
For practical usage, the investigation of predicting the BPDs from sensor data is of high interest, as the BPD is not given a priori and instead must be inferred.

Furthermore, the presented approach is not only applicable to predicting the behavior of persons with dementia, but also to any continuous HAR task.
Whenever the prior for occurrence of activities varies over time, the method described in this paper offers a viable option to improve HAR performance.

\bibliographystyle{unsrtnat}


\section{Introduction}
\label{Introduction}

Recent advancements in deep learning have seen artificial neural networks (ANNs) match or exceed human-level performance
in narrow tasks \cite{single_character, mastering_go, deep_blue, starcraft}.  When exposing artificial intelligence (AI)
to environments that are flexible and novel, however, narrow AI loses the ability to flexibly identify and react to
patterns.  AI remains unable to match biological intelligence in flexibly negotiating dynamically changing environments.
This is because biological learning enables the continuous acquisition of new knowledge and development of new capabilities
across multiple disparate inputs, all without the newly learned information destroying internal state representations
necessary to perform previously learned skills or utilize previously acquired knowledge.

The inability of a traditional ANN to learn multiple tasks across heterogeneous data sources without exhibiting catastrophic forgetting
is a significant obstacle on the path to generalizable AI.  Solving this challenge is a step towards enabling AI to emulate
biological continuous multidomain learning; rather than relying on separate ANNs to learn and perform multiple tasks, a single
neural network capable of learning multiple tasks can replicate the flexibility of human learning, simplifying the deployment
of the capabilities of an AI system in dynamic, chaotic environments \cite{self_driving,multitask_mixture}.

As demonstrated in this manuscript, the training procedure described herein outputs a single neural network containing multiple
subnetworks, each of which is capable of learning a single task. Practically speaking, this methodology yields a single network with the capabilities
of multiple disparate networks, while also doing so more efficiently by only utilizing a small portion of the original network's weights
for each task.

\section{Background}

In this work, we are concerned with overcoming catastrophic forgetting in the development of ANNs capable of learning multiple
tasks across disparate domains.  To accomplish this, we pose and investigate the \textit{Multiple Subnetwork Hypothesis} and
build upon it to create a multitask ANN architecture capable of learning multiple tasks across various different data types and
with different input and output shapes.

\subsection{The Multiple Subnetwork Hypothesis}

We propose the \textit{Multitple Subnetwork Hypothesis} -- that a dense, randomly-initialized feedforward ANN contains within its
architecture multiple disjoint subnetworks which can be utilized together to learn and make accurate predictions on multiple
tasks, regardless of the degree of similarity between tasks or input types.  Additionally, we have developed the network
distillation procedure \textit{Reduction in Sub-Network Neuroplasticity} (RSN2), which optimizes and sparsifies dense ANNs, to
test our hypothesis.  Our approach enables state of the art performance on multiple tasks within the same neural network without
parameter overlap, and therefore without exposing subnetworks to the conditions giving rise to interference during multitask learning.

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=\columnwidth]{MannLayersMasking-FinalForIntro.png}}
    \caption{Illustration of the Multiple Subnetwork Hypothesis. Multiple subnetworks able to represent a fully-connected neural network
    for a given task are identified in a single dense, randomly-initialized feedforward neural network.  These multiple specialist
    subnetworks can then be arrayed as a sparse tensor to isolate task representations, shielding them from catastrophic
    interference and enabling multi-domain task performance.}
    \label{fig:rsn2_introduction}
    \end{center}
    \vskip -0.2in
    \end{figure}

\subsection{Sparse Subnetwork Identification and Isolation}

It is well known and documented that ANNs are often extremely overparameterized, and this overparameterization results in redundancies and
inefficiencies which can be eliminated to conserve computational resources without negatively affecting model accuracy \cite{optimalbraindamage}.
The pruning of an ANN to reduce or eliminate extraneous parameters yields a sparse representation of the original fully-connected network, and
the challenge is therefore to identify which subnetwork of parameters within a dense ANN will best represent the entire original ANN \cite{suzuki2001simple,
han2015deep, lis2019full}.

One means of reducing parameter count and computational requirements without sacrificing the accuracy of the sparse network is through the identification of
sparse subnetworks within dense neural networks. In a randomly-initialized feed-forward network, there may be a subnetwork which can sufficiently represent
the original network, called in other works a “winning lottery ticket” \cite{lotteryticket}.  This methodology serves as the primary inspiration of our
Multiple Subnetwork Hypothesis, as we explore in this study whether the parameters excluded from a subnetwork for one task may be utilized in the formation
of subnetworks capable of learning and performing additional tasks.

\subsection{Interference-Resistant Multitask Learning}

Continuous learning in both biological and artificial neural systems is constrained by the stability-plasticity dilemma. Learning must be sufficiently
plastic to allow the acquisition of new knowledge, but also sufficiently stable to prevent the forgetting of previous knowledge \cite{mermillod2013stability}.
Interference stemming from lack of stability during sequential learning in ANNs, often referred to as Catastrophic Forgetting, is a result of the tendency of
NNs to destroy internal state representations used in learning previously acquired tasks when learning a new task \cite{cfinconnectionistnetworks,
goodfellow2013empirical, pfulb2019comprehensive}. This phenomenon is especially pronounced in continuous learning paradigms with a low degree of inter-task
relatedness \cite{aljundi2017expert,ma2018modeling,masana2021ternary}, posing a challenge to creating multitask models able to perform prediction or
classification across very different tasks, input data types, or input shapes. Effective mitigation of catastrophic forgetting is therefore a key prerequisite
to achieving the flexibility and adaptability of a multidomain multitask artificial intelligence.

Isolating parameters for a given task, thereby fixing parameter subsets for a given task, is a means of achieving maximal stability in a continuous learning
context \cite{delange2021continual}. Parameter isolation methods localize the performance of learned tasks over a physically \cite{yoon2017lifelong,
rusu2016progressive,mallya2018packnet} or logically \cite{liu2018rotate,serra2018overcoming, mallya2018piggyback} isolated region of an ANN, reducing or
eliminating the interaction of weights responsible for a given task, therefore minimizing the opportunity for the acquisition of new tasks to interfere with
the ability of an ANN to continue to perform previously learned tasks. The limitation of interactions between functional groups of neurons has its basis in
the biological necessity to not only functionally integrate information in the brain, but to isolate the processing of that information into distinct modules,
as evidenced by the importance of locally specialized regions in processing sensory information \cite{zavitz2019understanding}. While the isolation of
subnetworks is the surest way to prevent interference arising from learning new tasks, this method can scale poorly under memory and computational constraints
and therefore demands a flexible approach to limiting network capacity \cite{yoon2017lifelong} or minimizing the size of the network utilized for each task.
Our methodology aims to negate the high computational cost of parameter isolation by applying our RSN2 methodology to represent multidomain multitask models
as sparse tensors, thereby minimizing their computational footprint.

\section{Methodology}

In this section, we will discuss how we enable a single model to learn multiple tasks across multiple domains and datatypes
through both a task-specific weight representational structure and a modified training procedure which selects disjoint subsets
of network weights for each individual task.

\subsection{Multitask Representational Structure}

The weights of fully connected neural network layers traditionally consist of a kernel tensor, $k$, of shape $m \times n$, and a bias vector,
$b$, of length $n$, where $m$ denotes the number of columns in the input data and $n$ denotes the number of neurons within the layer.
To process input data $\bar{x}$, the decision function is thus $\Phi(k\bar{x} + b)$, where $\Phi$ denotes the layer's activation function.
Suppose then that there are inputs from two different distributions, $\bar{x_1}$ and $\bar{x_2}$.  Given the two-dimensional structure of the
kernel tensor in this traditional representation, there is no way to alter the decision function given the distribution currently selected.

To address this problem, we add a new dimension, $t$, to the kernel tensor and the bias vector within the layer to denote which distribution or task
the input data belongs to, leaving the kernel tensor with a shape of $t \times m \times n$ and transforming the bias vector into a bias matrix of shape
$t \times n$. The decision function for this new layer is thus altered to that in Equation \ref{eq:mt_decision_function}.

\begin{equation}
    \label{eq:mt_decision_function}
    F(\bar{x}, i) = \Phi(k_{i}\bar{x} + b_{i})
\end{equation}

We take a similar approach for convolutional layers.  For a two-dimensional convolutions with color channels, the kernel is of shape
$s_1 \times s_2 \times c \times f$, where $s_1$ and $s_2$ denote the width and height of the convolutional filters, $c$ represents the
number of channels in the input, and $f$ corresponds to the number of filters.  The bias vector in this scenario has length $f$.  In our
multitask representation of the convolutional layer, we once more add a new dimension to the front of both the kernel tensor and the bias
vector to denote the task.

\begin{algorithm}[tb]
    \caption{RSN2 Training Procedure}
    \label{alg:rsn2}
 \begin{algorithmic}
    \STATE {\bfseries Input:} Neural Network $F(x)$, Training data for multiple tasks, $X_{i}, Y_{i}, i \in \{1,2,..,N\}$
    \FOR{$i \in \{1,2,..,N\}$}
        \STATE{Choose $p_{i}, p_{i} \in (0, 100)$, ensuring $\sum_{i = 1}^{N}p_{i} <= 100$}
        \STATE{Deactivate all weights which have been used for previous tasks}
        \STATE{Activate all weights which have not been used for previous tasks}
        \STATE{Select $(x_{i}, y_{i})$ as a subset of training data and labels for the current task}
        \STATE{Calculate active weight gradients with respect to $(x_{i}, y_{i})$}
        \STATE{Set all inactive weight gradients to 0}
        \STATE{Identify the quantity $q_{i}$ such that $p\%$ of weight gradients with respect to $(x_{i}, y_{i})$ is less than $q_{i}$}
        \STATE{Deactivate all weights with gradients less than $q_{i}$ and do not allow them to train}
        \STATE{Train the network as usual, only altering identified active weights and keeping all other weights inactive}
        \STATE{Save all weight values for the specified task}
    \ENDFOR{}
    \STATE{At inference time for task $i$, activate only stored weights for task $i$ and perform inference as usual}
 \end{algorithmic}
 \end{algorithm}

\subsection{Sparsification and Subnetwork Identification}

While the multitask representational structure described above could theoretically be used to enable a network to learn multiple tasks
as-is, it alone does not help us answer our hypothesis.  Alone, this structure allows multiple networks to be combined into a single model,
as the number of parameters within each layer is increased by a factor of the number of tasks. To fully test our Multiple Subnetwork Hypothesis,
we devised the RSN2 training procedure, which ensures that only one weight along the task dimension is active for all other fixed indices
in multitask layers.

\begin{equation}
    \label{eq:indicator_function}
    I(x) =
        \left\{\begin{array}{lr}
            1, & \text{if } x \neq 0\\
            0, & \text{otherwise}
        \end{array}\right\}
\end{equation}

Mathematically, we utilize the indicator function from Equation \ref{eq:indicator_function} as an activation function to indicate whether
an individual weight is active or inactive within a layer.  Using this function, we can therefore demonstrate whether disjoint subnetworks
are active for all tasks by taking the sum of the indicator function across the task dimension for all other fixed indices.  If for all
fixed indices, $b$ and $c$, the condition present in Equation \ref{eq:multitask_indicator_condition} holds, then each weight is only active
once across all tasks, meaning the task dimension adds no additional active weights to the network layer.

\begin{equation}
    \label{eq:multitask_indicator_condition}
    \sum_{a = 1}^{t}I(k_{a,b,c}) \leq 1; \text{ } b,c \text{ fixed}
\end{equation}

We utilize binary masking in our training procedure to impose these conditions.  These masks therefore ensure only select weights are active
during training through the bitwise multiplication of the weight value and the mask value.

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=\columnwidth]{MannLayersMasking-FinalForMethodology.png}}
    \caption{Diagram of the RSN2 training procedure and multitask representational
    structure for a single fully-connected layer with two tasks.  On the left, we begin with the initialized
    layer weight matrix (center) and create two binary mask matrices (top and bottom).  These binary masks
    are then bitwise-multiplied to the weight matrix, resulting in two disjoint subsets of weights which are
    then separated across the task dimension in our representational structure.}
    \label{fig:rsn2_methods}
    \end{center}
    \vskip -0.2in
    \end{figure}

For weight selection and pruning, we utilize a gradient-based approach to identify which weights are to be selected for each task.  By
performing pruning in this manner, all pruning is done in a one-shot manner after initialization rather than iteratively as the model
trains, with the pruning rate a hyperparameter selected at training time.  The entirety of the RSN2 training procedure for a single
network layer can be found in Algorithm \ref{alg:rsn2}.  For performing inference, the masks used for training can be removed, thus
reducing the number of active weights to a maximum value of $t*p*W$, where $t$ represents the number of tasks, $p$ the maximum proportion
of weights active for any one task, and $W$ the number of weights which would be present in the network had a traditional, one-task structure
been used.  Note that to satisfy Equation \ref{eq:mt_decision_function}, $t*p \leq 1$.  As a result, the RSN2 procedure, when coupled with
the multitask representational structure defined above, identifies and separates disjoint subnetworks within a traditional network to allow
each to individually perform separate tasks. Due to the lack of interaction between these subnetworks in the resulting model, similarity
between tasks is not required, and no transfer learning will occur between tasks.  A visual representation of the RSN2 training procedure
and multitask representational structure can be found in Figure \ref{fig:rsn2_methods}.

\subsection{Experiments}

We test our methodologies across eight experiments utilizing multiple datasets to test the Multiple Subnetwork Hypothesis and RSN2 training
procedure.  In our first experiment, we test the merits of the hypothesis by identifying whether a single network can learn the same task
multiple times.  To test this, we create a single convolutional multitask network and isolate five separate subnetworks within it, each
utilizing a disjoint 20\% of the weights within the network.  Each of these subnetworks is then trained on the Fashion MNIST dataset \cite{fashion_mnist}.
We then compare the results of each of these subnetworks, as well as the results of the ensemble consisting of all five subnetworks, against the performance
of a dedicated network with identical architecture.

Our second experiment builds upon the first by identifying if a single network can learn related tasks from a single dataset.  The classes from the same
Fashion MNIST dataset are broken into two groups for this experiment; the first group consists of the five individual classes which were most easily
identified in the previous experiment.\footnote{We identified those which were learned most easily by the resulting F1 score on test data by the control model.}
The second group consists of the remaining five classes.  The results of this experiment are compared to the performance of a dedicated network with identical
architecture.

For our third experiment, we train a fully-connected network to perform both the MNIST Digit Recognition \cite{mnist} and Fashion MNIST tasks and compare the
performance of a single network to two dedicated networks.  This experiment is the first we perform to test whether a single network can truly learn different
tasks from multiple different domains and distributions, and it is the first experiment designed to show whether a fully connected network can perform multiple
tasks.

In our fourth experiment, we train a convolutional network to perform both MNIST tasks from the previous experiment.  This network is identical in architecture to
all convolutional architectures trained previously, and once again this network is compared to two dedicated networks of the same architecture.

Our fifth, sixth, and seventh experiments continue to explore our hypothesis to a greater degree, as we train individual networks to perform multiple tasks across a
wider array of data modalities. In the fifth experiment, we train a fully connected network to perform three tasks: both MNIST tasks as well as a regression task on
house prices in the Boston area \cite{boston_housing}.  Experiment 6 trains a three-task convolutional network across both MNIST tasks as well as the CIFAR-10 dataset
\cite{cifar10}.  Our seventh and final experiment tests whether a single network can learn multiple tasks across an even wider array of input data types.  For this
experiment, we train a single fully-connected network to perform four tasks: both MNIST tasks, the Boston housing regression task, and an IMDB reviews sentiment
classification task \cite{imdb}.

Our final experiment seeks to explore the feasibility of our methodologies in the context of more complicated tasks. For this experiment, we trained a single convolutional
model to perform four tasks. The first three of these are the age, gender, and race prediction tasks from the UTKFace dataset \cite{utkface}. The fourth task is the CIFAR10
task \cite{cifar10}. This experimental network is tested against four dedicated networks of identical architectures.

\section{Results}

In this section, we present the results for each experiment conducted.  Firstly, we present the results for each control model for all tasks, as well as describe the
model architectures and training schedules, as well as any data preprocessing steps which were taken.  For control results, each control model was created only once
and will therefore be presented with greater detail upon first presentation, but its results will be included in every pertinent experiment.  For more detailed indicators
of performance for all models, as well as mappings between integer labels and their respective classes, we refer the reader to the appendices.

\subsection{Experiment 1: MNIST Fashion Ensemble, Convolutional Architecture}

Our first experiment involved creating a single convolutional network with five disjoint subnetworks each trained on the MNIST Fashion dataset.  We utilized a convolutional
architecture with two convolutional blocks, with the first block containing two convolutional layers with thirty-two $3\times3$ filters and ReLU activation \cite{relu} followed
by maximum pooling over a $2\times2$ filter area.  The second convolutional block was identical to the first, except each of the convolutional layers utilized sixty-four filters.
The output to the convolutional blocks were then flattened and fed into two fully-connected layers with $256$ neurons each and ReLU activation.  A final fully-connected layer with
ten neurons and softmax activation was used to provide the final outputs.  The only preprocessing which was done on input data was a division of pixel values by $255$ to ensure all
input values were within the interval from $0$ to $1$.  The model was trained using a batch size of $512$, and early stopping was initialized after three epochs with no improvement
of greater than $0.01$ in loss on validation data.  The same architecture, preprocessing steps, and training procedure was completed in the experimental case as well, with each
subnetwork pruned so that 20\% of the network weights were allocated to each task.  A summary of results can be found in Table \ref{tab:exp1}.

\begin{table}[t]
    \caption{Experiment 1 Results.}
    \label{tab:exp1}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        Subnetwork 1 & 90\% & 80\% \\
        Subnetwork 2 & 90\% & 80\% \\
        Subnetwork 3 & 90\% & 80\% \\
        Subnetwork 4 & 90\% & 80\% \\
        Subnetwork 5 & 92\% & 80\% \\
        Ensemble & 92\% & 80\% Each \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Fashion & 92\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
    \end{table}

\subsection{Experiment 2: MNIST Fashion Two-Task Model, Convolutional Architecture}

For our second experiment, we utilized an identical architecture, data preprocessing, and training procedure to the previous experiment but trained two individual subnetworks within that architecture.  We trained the
first subnetwork on the five classes which were most easily identified by the control model, measured by F1 score, and we trained the second subnetwork on the five classes
which were the most difficult for the control model to identify, also measured by F1 score.  The results of the experiment can be found in Table \ref{tab:exp2}.

\begin{table}[t]
    \caption{Experiment 2 results.}
    \label{tab:exp2}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        Easy Task & 98\% & 80\% \\
        Hard Task & 84\% & 80\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Fashion & 92\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
    \end{table}

\subsection{Experiment 3: MNIST Digits and MNIST Fashion Model, Fully-Connected Architecture}

Our third experiment is the first experiment to truly test whether a single model can learn disparate tasks across a variety of data modalities.  To do this, we trained a single fully-connected network to
perform both the MNIST Digit Recognition and the MNIST Fashion Recognition tasks.  For this experiment, we preprocessed the data for both tasks once more by dividing each pixel value by $255$.  Each image
was then flattened into a two-dimensional vector.  The images would then be passed through six fully connected layers each containing $1000$ artificial neurons each and activated using the ReLU activation
function.  The final layer contained ten neurons which were activated with the softmax activation function.  The model was trained with the same early stopping conditions as all previous models, and a batch
size of $512$ was used once more.  The experimental model was trained using the same procedure and with the same architecture, with each task pruned to utilize 10\% of the network weights.  The results of
this experiment can be found in Table \ref{tab:exp3}.

\begin{table}[t]
    \caption{Experiment 3 results.}
    \label{tab:exp3}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        MNIST Digit & 97\% & 90\% \\
        MNIST Fashion & 86\% & 90\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Digit & 97\% & N/A \\
        MNIST Fashion & 88\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
    \end{table}

\subsection{Experiment 4: MNIST Digit and MNIST Fashion Model, Convolutional Architecture}

In experiment four, we repeated the modeling goals of the previous experiment, but this experiment utilized the convolutional architecture from the previous experiments, once again with an 80\% pruning rate
for each task.  Results for this experiment can be found in Table \ref{tab:exp4}.

\begin{table}[t]
    \caption{Experiment 4 results.}
    \label{tab:exp4}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        MNIST Digit & 99\% & 80\% \\
        MNIST Fashion & 90\% & 80\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Digit & 99\% & N/A \\
        MNIST Fashion & 92\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
    \end{table}

\subsection{Experiment 5: MNIST Digit, MNIST Fashion, and Boston Housing Model, Fully-Connected Architecture}

The fifth experiment provides the greatest test for our Multiple Subnetwork Hypothesis up to this point, as we train a model with a wider variety of datatypes and tasks, and we also train some tasks in sequence rather than all
at once.  For this experiment, we trained a single fully-connected network first on both MNIST tasks, then trained the same network on the Boston Housing dataset.  The MNIST portions of the network were identical in architecture
to the ones used in previous experiments, but due to difference in input shapes between the MNIST images and the Boston Housing dataset\footnote{The MNIST images are both 28 pixels wide and 28 pixels tall, while the Boston Housing
dataset contains only 13 input features.} an additional fully-connected layer is added after the Boston Housing input layer to create shape alignment.  This layer is activated with ReLU activation, and it is also pruned in
the experimental model.  In the test case, all subnetworks are pruned to only use 10\% of available weights.

Data preprocessing for MNIST tasks is identical to previous MNIST preprocessing for fully-connected networks.  For the Boston Housing dataset, input features were scaled such that the minimum feature value in the training data
corresponded to a value of $0$ and the maximum value of the feature in the training data corresponded to a value of $1$.  The predicted values were also scaled using the same procedures.
During training, as stated before, both MNIST tasks are learned simultaneously, utilizing the same early stopping criteria and batch sizes as the previous cases.  For the Boston Housing task, the same early stopping procedure is
used, but the batch size is decreased to $32$ in both the control and test cases due to the limited amount of data available.  The results for this experiment can be found in Table \ref{tab:exp5}.

\begin{table}[t]
    \caption{Experiment 5 results.  In the performance column, percentages correspond to accuracy percentage on test data, while decimals represent mean squared error on test data.}
    \label{tab:exp5}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Performance & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        MNIST Digit & 97\% & 90\% \\
        MNIST Fashion & 87\% & 90\% \\
        Boston Housing & 0.016 & 90\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Digit & 97\% & N/A \\
        MNIST Fashion & 88\% & N/A \\
        Boston Housing & 0.017 & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
    \end{table}

Furthermore, to identify whether any forgetting occurred after sequential training, we recorded loss values on test data for current and previously-trained tasks after each subsequent task was trained.
These results are visualized in Figure \ref{fig:exp5}, where it can be seen that no performance degradation occurred on the first two tasks after training the third task.

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=\columnwidth]{Experiment5.png}}
    \caption{Task losses obtained on test data for multitask model after training each set of tasks.}
    \label{fig:exp5}
    \end{center}
    \vskip -0.2in
\end{figure}

\subsection{Experiment 6: CIFAR10, MNIST Digit, and MNIST Fashion Model, Convolutional Architecture}

Experiment 6 was designed to expand upon the findings of previous experiments by testing whether a single convolutional network could converge across multiple classification tasks across a variety of
input images.  In this experiment, we test whether a single convolutional network can converge on first the CIFAR-10 image classification task and then train the same network on both MNIST tasks we have
utilized to this point.  To ensure that all images are of the same shape, we downsize the CIFAR-10 images so that they have the same height and width as the MNIST data.  Additionally, because of the color
channels present in the CIFAR images, a separate first convolutional layer must be created to process the CIFAR-10 inputs in the multitask model.  This dedicated layer is still pruned to the same degree as
all other processing layers within the model.  During training, the same early stopping procedures that have been used previously were employed, and batch sizes of $512$ were used for all tasks.  The results
for this experiment can be found in Table \ref{tab:exp6}, and task losses after training can be found in Figure \ref{fig:exp6}.  The test model was pruned to 90\% in this experiment.

\begin{table}[t]
    \caption{Experiment 6 results.}
    \label{tab:exp6}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        CIFAR-10 & 60\% & 80\% \\
        MNIST Digit & 98\% & 90\% \\
        MNIST Fashion & 89\% & 90\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        CIFAR-10 & 47\% & N/A \\
        MNIST Digit & 99\% & N/A \\
        MNIST Fashion & 92\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
\end{table}

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=\columnwidth]{Experiment6.png}}
    \caption{Task losses obtained on test data for multitask model after training each set of tasks.}
    \label{fig:exp6}
    \end{center}
    \vskip -0.2in
\end{figure}

\subsection{Experiment 7: MNIST Digit, MNIST Fashion, Boston Housing, and IMDB Reviews Model, Fully-Connected Architecture}

Our seventh experiment provides perhaps the most wide-ranging set of tasks for a single network, and as a result
it is perhaps the best test of our Multiple Subnetwork Hypothesis.  In this experiment, we utilized a single
fully-connected neural network and trained the network on four separate tasks; the first two of these tasks,
both the MNIST Digit and MNIST Fashion tasks, were trained simultaneously.  The third task, the Boston Housing
regression task, was then trained.  Finally, the IMDB reviews sentiment classification task was trained.
The same architecture as previously used in all fully-connected experiments was used in this experiment.  For
the MNIST and Boston Housing tasks, the same preprocessing and reshaping efforts took place as previous experiments
with fully-connected architectures.  For the IMDB task, only the most common $10000$ words were utilized and
sequences were padded (or truncated) to $128$ words, with both padding and truncating occurring at the end of the
review.  Furthermore, each of the input tokens was passed through an embedding layer, which embedded the token in a
two-dimensional vector space.  The embedded tensors were then flattened into a two-dimensional vector and passed through
the architecture used throughout this experiment\footnote{This architecture involves six fully-connected layers of $1000$
neurons and ReLU activation.}.  For the multitask model in this experiment, the first fully-connected layer which processed
inputs from the IMDB task was dedicated solely to the IMDB task due to the differences in input shapes between all tasks.
During training, the same early stopping criteria were used for all training iterations, batches sizes of $512$ were used
for the MNIST and IMDB tasks, while a batch size of $32$ was used for the Boston Housing task.  The results for this experiment
can be found in Table \ref{tab:exp7}, and task losses can be found in Figure \ref{fig:exp7}.  The model was pruned such that
each task utilizes 10\% of the available weights in fully-connected layers; the embedding layer was not pruned.

\begin{table}[t]
    \caption{Experiment 7 results. In the performance column, percentages correspond to accuracy percentage on test data, while decimals represent mean squared error on test data.}
    \label{tab:exp7}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Performance & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        MINST Digit & 97\% & 90\% \\
        MNIST Fashion & 87\% & 90\% \\
        Boston Housing & 0.011 & 90\% \\
        IMDB & 82\% & 90\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        MNIST Digit & 97\% & N/A \\
        MNIST Fashion & 88\% & N/A \\
        Boston Housing & 0.017 & N/A \\
        IMDB & 80\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
\end{table}

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
    \centerline{\includegraphics[width=\columnwidth]{Experiment7.png}}
    \caption{Task losses obtained on test data for multitask model after training each set of tasks.}
    \label{fig:exp7}
    \end{center}
    \vskip -0.2in
\end{figure}

\subsection{Experiment 8: UTKFace and CIFAR10 Four-Task Model, Convolutional Architecture}

In our final experiment, we tested the RSN2 training procedure on a larger network with more complex tasks involved. We created a single
network with a convolutional architecture on four individual tasks, three tasks from the UTKFace dataset \cite{utkface} and one multiclass
classification task from CIFAR10 \cite{cifar10}.  For the age task in the UTKFace data, we grouped values into classes by decade, excluding
a final class containing all individuals with age greater than 90.

Architecturally, the model trained in this experiment contains three convolutional blocks containing  of $16$, $32$, and $64$ $3\times3$ filters,
respectively.  Each block contains two convolutional layers with padding to preserve shape, followed by a max pooling layer.  These blocks then
feed into a fully connected architecture of three layers, each with $128$ neurons each and activated using ReLU activation \cite{relu}, followed
by an output layer of the required shape to perform the specified task.

For the multitask model, we utilized a shared convolutional embedding between all UTKFace tasks. This resulted in a two-task structure for
convolutional layers within the model, with one task channel processing the UTKFace images and the other task channel processing the CIFAR10 images.
For the fully-connected layers, each of the UTKFace tasks was isolated and a four-task architecture was used.  Along both the convolutional and
the fully-connected architectures, each task was pruned to utilize only $25\%$ of the model's available weights per task, meaning $50\%$ of the weights in the
convolutional layers were utilized\footnote{This is due to the two-task structure of the convolutional layers, as mentioned above.} and $100\%$ of the weights
across the shared fully-connected architecture was utilized.\footnote{This is due to the four-task structure of the fully-connected layers, as mentioned above.}
All four tasks were trained simultaneously, with early stopping criteria utilized with patience of five epochs. A summary of the results of this
experiment can be found in \ref{tab:exp8}. In addition to these performance results, which interestingly showed that the dedicated control
models were unable to converge but that the multitask model was able to converge, it was also found that the multitask model only utilized
$130MB$ of disk space when saved, while the combined single-task models utilized $203MB$ of disk space, over $56\%$ more than the
multitask model.

\begin{table}[t]
    \caption{Experiment 8 results.}
    \label{tab:exp8}
    \vskip 0.15in
    \begin{center}
    \begin{small}
    \begin{sc}
    \begin{tabular}{lcc}
    \toprule
    Network & Accuracy & Pruning Rate \\
    \midrule
        \multicolumn{3}{c}{\textbf{Experimental}} \\
        \hline
        UTKFace Age & 55\% & 75\% \\
        UTKFace Gender & 89\% & 75\% \\
        UTKFace Race & 77\% & 75\% \\
        CIFAR10 & 50\% & 75\% \\
        \hline
        \multicolumn{3}{c}{\textbf{Control}} \\
        \hline
        UTKFace Age & 32\% & N/A \\
        UTKFace Gender & 48\% & N/A \\
        UTKFace Race & 14\% & N/A \\
        CIFAR10 & 10\% & N/A \\
    \bottomrule
    \end{tabular}
    \end{sc}
    \end{small}
    \end{center}
    \vskip -0.1in
\end{table}

\section{Discussion}

This study was designed to provide insight into answering our Multiple Subnetwork Hypothesis, which
states that a dense, randomly-initialized feedforward neural network contains within its architecture
multiple disjoint subnetworks which can be isolated to each separately learn and make accurate predictions
on multiple tasks, regardless of the degree of similarity between tasks or input types.  To test this
hypothesis, we developed the RSN2 procedure as well as a customized neural network representational structure
designed to isolate individual subnetworks within the overall architecture of the global model on a
layer-by-layer basis.  We then utilized these techniques in a series of seven experiments across a range of
datasets utilizing both fully-connected and convolutional architectures.

Across each of these experiments, we consistently saw that a single network was able to converge on multiple
tasks by utilizing only a small fraction of network weights for each task.  Furthermore, multitask models were
able to perform each task with near identical performance to dedicated models with all weights active, even
exceeding dedicated model performance in a few cases. Additionally, we saw that there needed to be no
similarity between tasks, as in our most extreme case we were able to teach a single model to perform two
separate image classification tasks, a tabular regression task, and a sentiment classification task on natural
language.

The beneficial implications of this approach arise from the simplicity of interacting with a single model rather
than a collection of models as in the traditional machine learning deployment paradigm. Take for example the
human visual system: a unified stream of visual information is collected and passed to the optic lobe, where it
is then passed to the relevant regions of the brain for further downstream analysis. This paradigm is reflected
in the unified data pipeline of a multitask model trained using the RSN2 procedure, where invoking the model
involves invoking the relevant task or tasks automatically. Therefore, a collection of models capable of
performing multiple tasks can be deployed as a single entity within an organization's IT infrastructure,
thereby improving efficiency and lowering costs. Take, for instance, a simple example of an organization
which wants to deploy five separate models trained on five separate tasks. A typical deployment scenario would
involve this organization deploying one model each to five separate computational nodes. Using a model trained
with the RSN2 procedure to perform all five tasks, this organization could therefore reduce costs by deploying
this single multitask model to a single node, saving 80\% of the organization's original costs. Furthermore,
to improve system robustness by introducing redundancy, the organization could deploy this single model
to a set of three nodes and still reduce costs by 40\%.

Additionally, in our final experiment, even though we do not utilize sparse tensors to save the model, the
pruned multitask model utilizes less disk space when saved compared to the total amount of the identical
single-task models. This indicates that utilizing RSN2 and our multitask representational structure already
leads to more efficient representation of the models themselves, thus potentially further reducing costs by
decreasing the hardware requirements needed to save and use models for inference.

\subsection{Limitations}

There are a number of limitations to this study which limit the conclusions we can draw from the results.
Firstly, our experiments focus primarily on computer vision related tasks in this work, with all experiments
containing some image classification task.  While we do also include experiments which test our methods
on both tabular input and natural language inputs, further studies should still be conducted to more rigorously
test our hypotheses and methods on various input and data types.

Another limitation to this study is the limited size of both the datasets and models which were studied.  Many
state of the art models have been trained on millions of data samples, and the models themselves consist of
many tens of millions of parameters.  Our study, on the other hand, considered labeled data with up to tens of
thousands of samples and models which contained hundreds of thousands to millions of parameters, orders of magnitude
fewer (in both cases) than the state of the art in most cases.  It is therefore unknown whether these methods would
need to be modified\footnote{In particular, it is unknown whether our one-shot sparsification technique would
generalize to much larger models.} to accommodate larger modeling paradigms.

\subsection{Conclusions}

Despite the limitations to this study, there are still a number of conclusions we can draw from its results.
Each multitask model was clearly able to converge on all its assigned tasks, achieving performance similar to,
and in some cases exceeding, the control cases each time.  These results therefore confirm our Multiple Subnetwork
Hypothesis, showing that a single neural architecture is capable of utilizing small portions of itself to learn
individual tasks.  Additionally, we showed that our methodology for identifying and logically separating network
weights is robust to catastrophic forgetting, and we showed that we were therefore able to train multitask models
across various domains and tasks both in parallel or in sequence without any adverse effects on previously-learned
tasks.

Because the resulting model would have the ability to robustly perform multiple tasks, we can also conclude that
a single model trained with these techniques could substitute multiple models in a deployed scenario.  This
substitution would therefore reduce resource requirements for organizations deploying and utilizing models, thus
saving costs and simplifying the deployment architectures these organizations use.  These models would also be able
to acquire more downstream tasks after previously being trained on initial tasks, meaning adding new predictive
analytics would be a much simpler undertaking as well, as it would only involve updating an already-deployed model
and not require creating a completely new deployment for every new analytic.

\subsection{Future Work}

Following studies should be designed to address some of the limitations of this work as well as to
improve the methods used within.  Future studies should provide a more thorough analysis on how
these methods perform on a larger variety of data modalities, including data types, task types, data
availability, and model sizes.  Secondly, we intend to identify whether a more precise pruning method
can be applied as training occurs instead of in a one-shot manner as was done in this study.  If pruning
can be applied in a more intelligent manner, we believe the resulting model would potentially be able to
more consistently surpass unpruned model performance while also achieving greater sparsification rates than
possible with one-shot techniques.

Additionally, this study does not perform any analysis to identify the maximum amount of tasks which can be
learned by a single network.  Future work should therefore address finding the upper bounds of the number of tasks
an individual model can support.  Finally, the methods in this work do not allow for any transfer learning across
tasks; instead, it is assumed that new tasks should utilize a completely disjoint subset of weights relative to all
previously-learned tasks.  Additional studies should be conducted to identify whether transfer learning can be
leveraged during the acquisition of similar tasks to enable further reduction in the number of active weights and
the quicker acquisition of new tasks.

\newpage
\onecolumn
\bibliographystyle{unsrt}