\section{Introduction}

Machine learning-based Artificial and Robotic Systems generally follow the paradigm of training once on a large set of data after which they are deployed and rarely updated. Improving these systems as additional training data is collected or for adaptation to new tasks requires expensive, offline fine-tuning or re-training. In contrast, humans and animals continue to learn new concepts and evolve their skill sets as they act within and interact with novel environments over long lifespans. That is, biological systems demonstrate the ability to continuously acquire, fine-tune, and adequately reuse skills in novel combinations to solve novel yet structurally-related problems \cite{mendez2020lifelong}. As Artificial and Robotic Systems are increasingly relied upon for mission-critical real-world applications, it is increasingly important that they exhibit similar capabilities and are able to continually learn and adapt in dynamically-changing environments, truly becoming \emph{Lifelong Learning Machines}.

Continual learning and lifelong learning (L2) \cite{thrun1995lifelong} remain long-standing challenges for the machine learning community \cite{parisi_continual_2019}. Even for simple classification tasks, i.e., incrementally learning to classify a new category of object, leads to \textit{catastrophic forgetting} \cite{kirkpatrick_overcoming_2016}. On the other end, models may exhibit too much rigidity during learning by prioritizing preserving performance on old classes/tasks and failing to learn the new classes/tasks.
Minimizing catastrophic forgetting while maximizing the capability to learn new tasks is a key issue faced when designing lifelong learning algorithms. Balancing these two properties is known as the \textit{the Stability-Plasticity Dilemma} \cite{parisi_continual_2019}.

There has been a lot of progress on continual learning for the incremental classification task \cite{de2021continual,mai2022online}, but continual learning in a reinforcement learning setting is more challenging
\cite{khetarpal2020towards,padakandla2021survey}, and research on lifelong reinforcement learning (L2RL) is still in its infancy. Due to the interactive nature of the setting, pushing the state-of-the-art forward in L2RL and making L2RL useful for real-world applications requires going beyond researching and developing L2RL algorithms. It also requires careful investigation and research at the systems level, especially research into the appropriate methods for the non-trivial integration of multiple L2RL algorithms and components into a common framework. Hence, there is a need for a highly configurable, modular, and extendable framework targeting the L2RL domain.
In recent years, there has been a push to develop systems to facilitate research in L2RL. Frameworks developed in \cite{powers2021cora}, \cite{johnson2022l2explorer}, \cite{fendley2022continual} have looked at standardizing and benchmarking L2RL scenarios and have standardized metrics for comparing L2RL algorithm performance.
\newedit{This paper addresses the unexplored challenge of combining multiple continual learning components in a common framework. We take inspiration from Complementary Learning Systems, Parisi et al. \cite{parisi_continual_2019} and assimilate different continual learning components (each addressing different aspects of the L2RL problem).}

The socio-economic impact of such a system is enormous.
We highlight a few examples.
i) Autonomous vehicles \cite{ma2020artificial} should adapt to changing conditions (e.g., weather, lighting) and should learn from their mistakes (e.g., accidents) in order to improve in terms of safety and utility over time.
ii) caregiver/companion robots \cite{wikipedia_2022} should learn to adapt to the needs of specific human patients/partners.
iii) Systems for medical diagnosis and treatment planning need to adapt to novel conditions (e.g., new disease variants) as well as adapt to the current state of patients and their response to previous interventions.
iv) Network security systems must be able to protect against novel threats (e.g., new viruses, hacking efforts) in an expedient manner in order to minimize security breaches.
Systems exist in these application domains, but often rely on brute-force approaches (learning once from massive data) and are not truly solving the core problem. Hence, more focused efforts towards building \emph{Lifelong Learning Machines} are needed.
v) Model obsolescence \citep{sculley2015hidden, alahdab2019empirical, bogner2021characterizing} of machine learning-based software systems is a major problem facing the software industry.
The inherent plastic-yet-stable nature of lifelong learners enables these systems to be more robust to obsolescence resulting from data drift, concept drift, and task changes.
Designing Artificial and Robotic Systems with lifelong learning at their core will ultimately decrease system downtime and reduce the overhead of model re-training. The technical contributions of this paper include the following:
\begin{enumerate}
   \item We describe a \emph{Lifelong Reinforcement Learning Components Framework} \emph{(L2RLCF)} that leverages a \emph{Wake-Sleep Mechanism} \cite{raghavan2020lifelong} from Complementary Learning Systems.
   \item As an instantiation of L2RLCF, we develop a Python API allowing easy integration of novel lifelong learning components with each component encapsulated in its own module. We discuss the details of our API in Section \ref{sec:system}.
   \item  We introduce an evaluation environment to allow for the fair, comprehensive, and quantitative comparison of different combinations of components within a common evaluation environment. This environment employs different lifelong learning scenarios (sequences of tasks) consisting of \textit{Starcraft-2~} minigames, a challenging environment for L2RL (Section \ref{sec:evaluation_env}).
\item We demonstrate the utility of L2RLCF in a case study integrating a diverse set of independently-developed L2 algorithms from recent work in continual learning (Section \ref{sec:case_study}). We integrate: i) a base system for L2RL in a wake-sleep setting, ii) automatic triggering of sleep via change point detection \cite{faber2021watch}, iii) compression of experiences in the replay buffer \cite{hayes2019remind}, iv) task-specific prioritized replay, and v) representation learning via self-supervised learning.

\end{enumerate}

\section{L2RL Components Framework}
\label{sec:system}

In this section, we discuss the design of the Lifelong RL Components Framework (L2RLCF). A visual representation is seen in Figure \ref{fig:main}, and an algorithmic overview of the system is seen in Algorithm \ref{alg:wakesleep}.

\subsection{Wake-Sleep Mechanism}
Our framework employs a wake-sleep learning paradigm (fast and slow, respectively). Wake-sleep is a biologically-motivated framework that directly tackles the tradeoff in lifelong learning between \emph{plasticity}, i.e.\! learning the current task, and \emph{stability}, i.e.\! remembering past tasks. It was first introduced in \cite{raghavan2020lifelong} for class-incremental learning and extended to L2RL in \cite{daniels2022model}. It consists of two phases: i) a wake phase where standard non-lifelong learners learn aspects of the current task, e.g.\! off-the-shelf RL trained on the current task (current MDP reward and dynamics), and ii) a sleep phase where knowledge is consolidated across multiple wake periods. Additional details about our specific L2RL wake-sleep implementation can be found in Section \ref{sec:wakesleep}. Wake-sleep is well suited as the core for a unifying framework for any lifelong learning system architecture because it is easily extendable, modularized, and compatible with many existing continual learning algorithms.

\subsection{System Design}

\subsubsection{Environment}\hfill\\
In this paper, the environment can be any Partially Observable Markov Decision Process (POMDP) where an agent can perceive observations, interact with the environment, and receive a reward for the actions taken. In this paper, our environment is  the complex strategy game of \textit{Starcraft-2~} using the PySC2 \cite{vinyals2017starcraft} interface to the game engine. \textit{Starcraft-2~} mini-games require high sample complexity for single-task RL, and the issue is exacerbated when a sequence of POMDPs are presented. \textit{Starcraft-2~} requires processing percepts and requires decision-making over a large action space with multiple agents. More details about our \textit{Starcraft-2~} environment appear in Section \ref{sec:evaluation_env}.

To instantiate the sequence of tasks for the L2RL agent, we implement a ``Syllabus Runner'' that takes a configuration of task orderings with the length of each learning and evaluation period and automatically generates a sequence of simulators (parallelizing individual episodes within each learning phase).

\subsubsection{Preprocessors \& Annotators}\hfill\\
There are several base classes that continual learning algorithms can implement. In this section, we focus on two of them. The \emph{Preprocessors} class consists of a list of preprocessor objects from different components. Each preprocessor comes into play when the agent perceives an observation. Each preprocessor is passed the observation features output by the simulator and subsequently transforms them in a meaningful way (e.g., converting RGB features to features usable by later machine learning algorithms). Once the observations are preprocessed, they are added to the original observations as named tuples and later system components can utilize these preprocessed features as needed.

The \emph{Annotators} class consists of annotator objects from different components. It is triggered after an agent has stepped through an action and received a reward. The tuple of (observation at previous time step $obs_{processed}$, wake policy logits, action, reward) is passed to each annotator object, which is then queried for the annotation feature. Annotaters also have access to previous observations; this can be leveraged to add interesting functionality to the system, e.g., for creating prioritized replay buffers. In the \textit{Starcraft-2~} case study, we used an annotator for ``danger detection'', i.e., scoring the estimated level of danger of a given state, and then building a replay buffer of safe states to promote a useful bias (avoid bad terminal states) in the policy. Like the Preprocessors class, the annotator features are added as a named tuple to get $O_{ann}$, which is then passed around in the system and used as needed.

\subsubsection{Memory Model}\hfill\\
The \emph{Memory Model} class represents generic memory models. In our experiments, this entailed different styles of replay (experience replay, generative replay), but it is flexible to more complex models such as clustering-based hierarchical memory models. There are two components within the memory model: the \emph{Encoder} and the \emph{Decoder}. The Encoder is not a necessity as some memory models will not require encoding (e.g., experience replay). The API is expressive enough to define whether the policy networks are built on top of an observation space, a processed feature space from the Preprocessors, a combination of features, or even on top of a generative model's encoder.
The Decoder is equally flexible. It can be used in conjunction with genarative models to sample novel experiences (e.g.\!, we used a variational autoencoder as our generative memory model). In cases where generative memory is not necessary, it provides a mechanism to sample of old experiences stored in a buffer (for example, by returning exemplars).

\subsubsection{Wake Learner}\hfill\\
The model has a \emph{Wake Learner} that is any standalone off-the-shelf RL learner. We used Vtrace \cite{espeholt2018impala} in our experiments.

\subsubsection{Sleep Learner \& Skill Consolidation}\hfill\\
The \emph{Sleep Learner} is a different instantiation of the same RL learner as the Wake Learner and serves as a means of consolidating skills across multiple wake-sleep cycles. The sleep model is built around the idea of replaying of old experience (past samples, exemplars, generated samples, etc.) to help minimize catastrophic forgetting and learn to generalize to unseen tasks. It should be noted that in the Sleep Learner, we can model one or more sub-policies, enabling useful features such as selecting different policies based on perceptual/semantic similarity between the current and past tasks (implemented as annotator since task boundaries are unknown).

\subsubsection{Experience Buffer}\hfill\\
The \emph{Experience Buffer} can take several forms and involve several different sampling mechanisms, including:
    \begin{itemize}
        \item \textbf{Wake Buffer}: The current interactions after passing through annotators and preprocessors is saved to wake buffer. The observations are kept sequentially to create trajectories. After the buffer becomes full, it is sampled and used for training the Wake Learner.
        \item \textbf{Exemplar Sampler}: (Optional) We have observed if certain replay architectures are used (e.g., hidden replay, see Section \ref{sec:wakesleep}), there is still concept drift, which can be alleviated using exemplars. Exemplars can be selected in many ways, e.g., via random sampling of the current wake buffer, importance sampling, via clustering of data samples, or via other L2 techniques (e.g.\!, \cite{rebuffi2017icarl}).
        \item \textbf{Replay Generator}: This is used for training the memory model and skill consolidation by dynamically generating replay samples.
        \item \textbf{Batch Sampler}: The batch sampler usually acts as a random/equal weight sampler, but can have utilize other sampling mechanisms like prioritized replay, which we employ for danger detection as discussed early.
    \end{itemize}

\subsubsection{Student-Teacher Learning}\hfill\\
Various components of the model are trained following a \emph{Student-Teacher Learning} paradigm. The Wake policy model is trained via a RL algorithm of choice. The policy logits are stored in the wake trajectories along with the observations, rewards, actions taken, and other meta-data. The sleep policies are updated (i.e., skill consolidation) using a distillation loss encouraging the sleep model to imitate the (observation, action logit) pairs collected by the Wake model as well as (observation, action logit) pairs sampled from various replay buffers (varies based on system specification). In our experiments, we use experience replay, random exemplar replay, and generative replay. In the case that a generative memory model is used, it is updated via a reconstruction loss comparing raw observations to reconstructed observations or comparing some preprocessed features to reconstructed versions of those features.

Closely tied to the Student-Teacher Learning are \emph{Expert Advice} and the \emph{Skill Selector}. The Expert Advice defines a mixture probability $p_{advice}$ that tells the agent whether to use the current wake policy to process the current observation or whether to use the policy of an expert teacher (generally defined by the sleep model). For the first wake phase, no advice is taken. In subsequent wake phases, the Expert Advice module samples from the sleep model with decaying probability over time. The goal is that this will encourage the wake model to explore in a more intelligent way if there is positive forward transfer between the tasks the sleep model has seen and the current task, ultimately teaching the wake model more effective policies. The Expert Advice probability is set by the Advice Scheduler, which is highly configurable (e.g.\!, constant, linearly decaying, exponentially decaying, cyclic). The time-to-decay is also a configurable parameter. In our experiments, we set it to start with a high probability ($>0.8$) and decay to a low probability (<0.2) by the half-way point of the wake phase's learning period, after which it remains constant until the next wake phase. We have observed this type of scheduler performs well in practice.

The sleep model may have multiple sub-policies (e.g., if it consists of a mixture of experts). In this case, a Skill Selector is needed to select which policy should be employed at the start of the wake phase. In order to do this, a small buffer of observations are stored, and tested across multiple sleep policies. The policy which yields the highest reward on this test set is selected to act as the Teacher network for the wake policy. In our experiments, we have also tried a similar approach/variant wherein the weights of the best sleep policy are copied directly to the wake policy for strong initialization of the wake model for the current task, promoting strong forward transfer (as the system is using the current best known policy for the given task, similar to the jump start provided by advice). This is especially true in the case where the model learns distinct policies for individual tasks/skills such as in the case of the mixture of experts or hierarchically-clustered policies.

\begin{algorithm}[h]
   \caption{Wake-Sleep Setting}
   \label{alg:wakesleep}
\begin{algorithmic}[1]
   \STATE {\bfseries Iterates:} Generator $g_s^t$, sleep policy $\pi_{s}^t$, wake policy $\pi_{w}^t$, wake buffer $b^t_w$
	   \FOR{$t=1,2,\ldots$}
	   \FOR[Wake Phase]{$K$ times}
	   	   \STATE{Run Skill Selector to select Teacher policy}
	       \STATE{Sample observation $o$ from current task}
	       \STATE{$o_{p} \leftarrow$ Pass observation $o$ through the Preprocessors}
	       \STATE{$o_{a} \leftarrow$ Pass $o_{p}$ through the Annotators}

	       \STATE Using Expert Advice: Sample $a \sim \pi_s^t(o_{a})$ w.p. $p_{\text{advice}}$, else $a \sim \pi_w^t(o_{a})$
	       \STATE{Compute reward $r$ from taking action}
	       \STATE{Add transition $(o_{a}, r, a)$ to Experience Buffer $b_w^t$}
	       \STATE{Update wake policy $\pi^t_w$ on current task reward using $b^t_w$}
		   \STATE{Decay $p_\text{advice}$ according to Advice Scheduler}
	   \ENDFOR\\

	   \FOR[Sleep Phase: Skill Consolidation]{$N$ iterations}
		   \STATE{Sample batch $B$ from $b_w^t$ using Batch Sampler on Experience Replay Buffer.}
			 \STATE Using Memory Model: Sample batches $O_s^t \sim g_s^t$
			 \STATE Pseudo-label $A_s^t = \pi_s^t(O_s^t)$
			 \STATE (GR Buffer) $B_{S} = B \cup (O_s^t, A_s^t)$
			 \STATE Minimize distillation loss + reconstruction loss on $B_{S}$
		\ENDFOR\\

	 \ENDFOR
\end{algorithmic}
\end{algorithm}

\subsubsection{System Configuration Runner} \hfill\\
The L2RLCF \emph{System Configuration Runner (SCR)} spins off the whole system from a YAML configuration, which contains all of the information needed to instantiate and run the system.

The SCR has many similarities with hydra \cite{Yadan2019Hydra}, but is specific to our use-case. The SCR enables iteration over multiple experiment settings and quick spin-off of these systems. It has the following abilities:

\begin{itemize}

    \item Can set system parameters through a YAML configuration and also as argparse settings.
    \item Can recursively loop through the YAML configuration to load functors and instantiate class objects, removing the constraint that class arguments are restricted to \emph{Built-In} types, giving the system the ability
    to hierarchically load objects, and allowing for custom parameters and custom class objects.
    \item Can set system environment variables which might be required by environment simulators or by other components where parameter passing through object initiation is not possible.
    \item Allows for configuration templating. In L2RLCF, many parameters like learning block size or observation-actions space dimension are relevant to multiple components. The template section of the configuration allows parameters to be set (including assigning functors or instantiating \emph{shared} class objects) and have them available across multiple components.
    \item \newedit{Helps with scheduling of components to specific GPU IDs}

\end{itemize}
Unlike hydra, the SCR doesn't have hierarchical loading of hydra-configuration files, but similar feature might be helpful in further improving configuration management for L2RLCF.

\subsubsection{Containerization}\hfill\\
L2RLCF is containerized with docker with all the dependencies of the main system installed by default. New configurations are mounted to docker for running the experiments. One design decision we took was to go for a single docker (monolith) as opposed to a pod of dockers (microservices). The argument for going with a microservices architecture is that many of the components are independently developed across different institutions with their own  dependencies and  system needs. By maintaining a single docker container, we avoid fractured and inefficient structuring of the system, ensure consistent and standardized versioning of external libraries, avoid decreases in speed resulting from communication through network protocols, and enable end-to-end training across the multiple system components.

\section{Experiments}

\subsection{Integrated SC-2 Agents}

In this section, we demonstrate how the evaluation environment is useful for understanding the effects of different components. We set up an experimental environment for evaluating the components discussed in Section \ref{sec:case_study} using standardized sets of syllabi for different scenarios under identical wake/sleep conditions with the same base agent. We show results in Table \ref{tab:extensions_results} where we specifically look at the case of turning on one additional component at a time. This enables the user of the lifelong learning system to understand the advantages and disadvantages of adding one or more components. For example, we can see that compressing the replay can negatively affect the performance of the system in terms of rewards compared to the single task expert in most cases, and oppositely, self-supervised learning generally helps w.r.t. performance relative to the single task expert. Similarly, we can see adaptive sleep can often improve performance maintenance.

\subsection{Agent Run-Time}

All the experiments are run with the wake actors interacting with 8 \textit{Starcraft-2~} simulators. Each wake phase has 2 million interaction steps with 2 forced sleep phase in between. For experiments involving adaptive sleep, the number of sleep phases in a given learning block is dynamic.

Note that in this work, we don't target developing the most efficient system. Instead, our goal is to develop a framework for easily combining lifelong learning algorithms and an evaluation setting for understanding the trade-offs associated with each component/combination of components. One important trade-off is understanding how adding a component improves the performance of the system at the cost of increased run-time.

The amortized phase run-times, in seconds, are reported in Table \ref{tab:extensions_results} along with performance metrics.
As seen from the table, adding different components increases the run-time for each phase, but the learners themselves become more sample efficient, i.e., requiring about 2 million steps to learn a given task as opposed to about 10 million steps needed for single task experts.

\section{Discussion}
In this work, we introduced a common framework for integrating diverse lifelong learning components in a unified system. The system has just one assumption, i.e. only requiring the use of a wake-sleep cycle. To deploy the system in the real world, we developed a well-defined API. We also introduced a challenging evaluation environment to fairly assess the impact of integrating multiple components and assessing their effect on the overall system in a quantitative way. To demonstrate that the framework is useful in practical scenarios, we constructed a case study integrating multiple complex existing lifelong learning algorithms. The experiments showed how we could use the evaluation environment to identify some of the strengths and weaknesses of included algorithms. This process can be easily reproduced and allows for the inclusion of new algorithms, providing an effective tool for their analysis. While there is much work to be done to improve the system and ultimately promote its adaptation in academia and industry, we expect that such a system is incredible useful for translating L2RL from research to real-world applications. If adopted, our system could majorly impact some of the domains mentioned earlier: autonomous vehicles, service robots, medicine, and network security among many others, and it could be a useful tool for minimizing model obsolescence and promoting fast model adaptation in dynamically-changing environments.

\section{Evaluation Environment}
\label{sec:evaluation_env}
While our proposed framework/API is compatible with generic lifelong learning reinforcement learning settings, we introduce an evaluation environment that is sufficient complex while designed for fair, comprehensive, and quantitative comparison of different combinations of components within a common evaluation environment. Our evaluation environment employs different lifelong learning scenarios (sequences of tasks) consisting of \textit{Starcraft-2~} minigames derived from \citep{vinyals2017starcraft}. \textit{Starcraft-2~} is a real-time strategy game where a player must manage multiple units in combat, collection, and construction tasks to defeat an enemy opponent. In our evaluation environment, the RL agent has control over selecting units and directing the actions the unit should take to accomplish a given task. In the L2RL setting, the system must learn to solve one task at a time without forgetting previous tasks, and the agents performance is measured on all tasks immediately after learning a task. We selected three minigames with two variants each as our task set. Each task involves either i) combat between different unit types or ii) resource collection (\emph{CollectMineralShards}). The tasks include:
\begin{itemize}
    \item \textbf{Collect Mineral Shards – No Fog of War}: A map with 2 Marines and an endless supply of Mineral Shards. Rewards are earned by moving the Marines to collect the Mineral Shards. Whenever all 20 Mineral Shards have been collected, a new set of 20 Mineral Shards are spawned at random locations (at least 2 units away from all Marines). Fog of war is disabled.
    \item \textbf{Collect Mineral Shards – Fog of war:} A map with 2 Marines and an endless supply of Mineral Shards. Rewards are earned by moving the Marines to collect the Mineral Shards. Whenever all 20 Mineral Shards have been collected, a new set of 20 Mineral Shards are spawned at random locations (at least 2 units away from all Marines). Fog of war is enabled, meaning the agent must be able to learn without full knowledge of the current state of the environment.
    \item \textbf{DefeatZerglingsAndBanelings – One Group:} A map with 9 Marines on the opposite side from a group of 6 Zerglings and 4 Banelings. Rewards are earned by using the Marines to defeat Zerglings and Banelings. Whenever all Zerglings and Banelings have been defeated, a new group of 6 Zerglings and 4 Banelings is spawned, and the player is awarded 4 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map.
    \item \textbf{DefeatZerglingsAndBanelings – Two Groups:} A map with 9 Marines in the center with 2 groups consisting of 9 Zerglings on one side and 6 Banelings on the other side. Rewards are earned by using the Marines to defeat Zerglings and Banelings. Whenever a group has been defeated, a new group of 9 Zerglings and 6 Banelings is spawned and the player is awarded 6 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map.
    \item \textbf{DefeatRoaches – One Group:} A map with 9 Marines and a group of 4 Roaches on opposite sides. Rewards are earned by using the Marines to defeat Roaches. Whenever all 4 Roaches have been defeated, a new group of 4 Roaches is spawned and the player is awarded 5 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map.
    \item \textbf{DefeatRoaches – Two Groups:} A map with 9 Marines in the center and 2 groups consisting of 6 total Roaches on opposite sides (3 on each side). Rewards are earned by using the Marines to defeat Roaches. Whenever all 6 Roaches have been defeated, a new group of 6 Roaches is spawned and the player is awarded 7 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to starting areas of the map.
\end{itemize}

PySC2 \citep{vinyals2017starcraft} was used to interface with SC-2. For the hand-crafted observation space, We used a subset of the available observation maps: the unit type, selection status, and unit density two-dimensional observations. The action space is factored into functions and arguments, such as $\text{move}(x, y)$ or $\text{stop}()$. The agent receives positive rewards for collecting resources and defeating enemy units and negative rewards for losing friendly units. For our experiments, we consider syllabi consisting of alternating (two tasks, each seen three times) and condensed (all six tasks, each seen once) scenarios.

\subsection{Metrics for Lifelong RL}
To quantitatively evaluate the performance of a L2RL system, we consider two sets of metrics. First, we consider how the rewards achieved by an agent compare to the ``optimal'' RL agent by comparing to the terminal performance of agents trained on each task to convergence (a ``single task expert''). These ``relative reward (RR)'' to the terminal reward achieved by a single task expert metrics are introduced in \cite{daniels2022model}. Note that these metrics focus purely on understanding the dynamics of an agent at periodic evaluation blocks (EBs). Second, we compare algorithms using the lifelong learning metrics defined by New et al. in \cite{new2022lifelong}, which take into account both behavior of the agent at periodic evaluation blocks but also characteristics of the agent as it learns (during learning blocks ``LBs'') (i.e., its learning curves during wake).

We consider the following variants of the RR metric:
\textbf{Relative reward in the final EB (RR\textsubscript{$\Omega$}):} Measures how well the agent performs on all tasks after completing the syllabi.
\textbf{Relative reward on known tasks (RR\textsubscript{$\sigma$}):} Measures how well the agent performs on previously seen tasks (quantifies forgetting/ backward transfer.
\textbf{Relative reward on unknown tasks (RR\textsubscript{$\upsilon$}):} Measures how well the agent generalizes/transfers knowledge from seen to unseen tasks. Note that in all cases, more-positive values are better for all metrics.

We consider the following lifelong learning metrics defined by \citet{new2022lifelong}: \textbf{Forward Transfer Ratio (FTR):} Measures knowledge transfer to \emph{unknown} tasks.
\textbf{Backward Transfer Ratio (BTR):} Measures knowledge transfer to \emph{known} tasks. A value greater than one indicates positive transfer.
\textbf{Relative Performance (RP):} Compares the learning curves between the lifelong learner and a single task learner. A value greater than one indicates either faster learning by the lifelong learner and/or superior asymptotic performance.
\textbf{Performance Maintenance (PM):} Measures catastrophic forgetting over the entire syllabus. A value less than $0$ indicates forgetting.

\section{Case Study: Integration of Multiple Lifelong Learning Algorithms in a Unified System}
\label{sec:case_study}
In this section, we discuss the integration of multiple lifelong learning algorithms using our L2RLCF framework and API in a fully-realized real-world system. In this case study, we integrate the following algorithmic components:
\begin{itemize}
    \item Base system for L2RL based on generative replay in a wake-sleep setting

    \item Automatic triggering of sleep via changepoint detection
    \item Compression of experiences in the replay buffer
    \item Task-specific prioritized replay mechanism for dangerous state detection
    \item Representation learning from RGB-observations via self-supervised learning
\end{itemize}
\textbf{Note that our system is not limited to these components. It is flexible to be used with any lifelong learning algorithm that can be integrated within a wake-sleep mechanism.} To provide context for the complexity of the integration effort and flexibility of L2RLCF, the details of each component used in this case study is described in the following sections.

\subsection{Standalone Optimal policy Trajectory Dataset}
\newedit{We learn single task experts (STEs) by running the RL policy in just the wake phase for a given task. Since deep RL in \textit{Starcraft-2~} is computationally demanding, we release a dataset of trajectories of trained single-task expert policies for future research \footnote{\url{https://github.com/sri-l2m/l2m\_data}}. We curate these trajectories to create the Optimal Policy Trajectory dataset. This dataset is used for the standalone pretraining of some of the components that are discussed in this section.}

\subsection{Wake-Sleep Generative Replay Algorithm}
\label{sec:wakesleep}

As previously mentioned, the fundamental tradeoff in lifelong learning is between \emph{plasticity}, i.e.\! learning the current task, and \emph{stability}, i.e.\! remembering past tasks. To address this tradeoff in the context of lifelong RL, we extend the wake-sleep mechanism first introduced by Raghavan et al.\! in \cite{raghavan2020lifelong}.
This approach utilizes two phases:
\begin{itemize}
    \item \textbf{Wake Phase:} A \emph{plastic} wake policy $\pi_w$ is optimized for the current task by interacting with the environment and using an off-the-shelf RL method. Transition tuples are collected during training and stored in a buffer; each tuple contains $(o, r, a)$, the observation $o$, reward for the previous action $r$, and the policy output $a$ (e.g., the policy logits or one-hot encoded action). The sleep policy $\pi_s$ provides ``advice'' (with importance decaying over time) in order to encourage the wake RL agent to begin exploring the current task using the consolidated policy learned from all previous tasks to encourage faster adaptation to the new task. In our experiments, we use an off-policy RL algorithm such as VTrace \cite{espeholt2018impala} to accommodate this off-policy action selection in the optimization of $\pi_w$.
    \item \textbf{Sleep Phase:} In the sleep phase, the \emph{stable} sleep policy $\pi_s$ is optimized to maximize the incorporation of new knowledge (the action selection in the wake buffer) while minimally forgetting current knowledge. While not a general requirement of the wake-sleep mechanism, in the particular implementation of wake-sleep used in this study, we also employ an additional replay type akin to generative replay in supervised learning. The augmented dataset(s) are created by combining wake transitions with tuples from a generative model $g_s$, which generates observations that are subsequently pseudo-labelled by the previous sleep policy. The sleep policy and the generative model are jointly trained.
\end{itemize}
Our base model has a unique architecture. In contrast to most generative replay-based models which reconstruct the observations in observation space, the architecture we use for testing purposes consists of a model that learns separate branches for i) reconstructing/generating intermediate hidden states from a feature extractor and ii) a policy network for predicting which action to take given an observation (while sharing a common feature extractor). We call this the ``Hidden Replay Architecture''. More details of this base model can be found in \cite{daniels2022model}. It should be noted that our system can work with any architecture/replay mechanism as long as it is implemented in a wake-sleep setting, and in fact, we have validated our integrated system with other, more traditional architectures, but do not report results here.

\subsection{Self-Monitoring for Sleep via WATCH}
\label{sec:self-monitoring}
In the base model, sleep is triggered after a fixed number of interactions with the environment. This can be inefficient due to the overhead of sleep, and lead to reduced performance due to over emphasizing memory consolidation versus learning. We hypothesize that triggering sleep adaptively at opportune times will lead to better performance with less overhead. We apply an unsupervised change-point detection method \cite{faber2021watch, faber2022lifewatch} to the features extracted from SC-2 observations using a pre-trained VGG-16 model \cite{simonyan2014very}
because (a) our setting does not assume knowledge of task change points, and (b) sleep can be beneficial even when the task has not changed, i.e. a significant change in the policy can cause it to visit novel states and observations. In principle, change point detection can be applied to episodic reward as well.

Preliminary experiments showed that standard methods for change point detection like CUSUM appear unreliable in the presence of  the high-dimensional features, whereas the recently proposed (LIFE)WATCH \cite{faber2021watch,faber2022lifewatch} method performs better. It has a few crucial benefits exploited in the system. First, it compares two sets of points instead of assuming any specific distribution, providing a more flexible approach. Second, leveraging Wasserstein distance allows for more accurate and robust detection in dynamic high-dimensional data. Moreover, instead of applying the absolute constants threshold, WATCH adapts to the discovered distribution learning it over time.
\newedit{This component is added as preprocessor block into the framework. The changepoint detector checks for changes in the observation space and signals the system to go to sleep if needed.}

\subsection{Experience Compression via REMIND}
Clever strategies for efficient management of replay buffers has been shown to improve supervised continual learning \cite{riemer2019scalable, wang2021acae} and lifelong RL \cite{caccia2020online,zhang2017deeper,schaul2015prioritized}.
In contrast to prior work that rejects most transitions from being stored in the buffer, we explore an approach that stores all the transitions but in a highly compressed form

Compression allows a buffer of a given size to contain a larger number of transitions, increasing the chance of retaining diverse examples from different tasks. We use the REMIND \cite{hayes2019remind} method, based on Product Quantization (PQ) \cite{jegou2010product}, to compress the agent's observations before storing them in the replay buffer, allowing the system to store more samples. The agent observations are image-like \texttt{float32} tensors with $3$ channels, and thus each pixel occupies $96$ bits. In the compressed observation, each pixel is quantized to an 8-bit integer, a 24x reduction in size. The PQ model is pre-trained on observations collected while following a random policy on a subset of tasks that is disjoint from the set of evaluation tasks.
\newedit{This component is also integrated as a preprocessor. It quantizes the current observation and makes it available for the downstream task.}

\subsection{Danger Detection for Prioritized Replay}
\label{sec:danger}
In addition to compression of experiences, we examine a novel form of prioritized experience replay \cite{schaul2015prioritized} based on detecting dead-end states (expressed as dangerous states in our \textit{Starcraft-2~} evaluation environment). We hypothesize that increasing the lifetime of an agent by avoiding dead end states is a useful bias. The Danger Detector outputs a ``danger score'' of how likely the agent is to lose the battle from a given state. This score is used as a replay priority. The policy's actions in the battle change over time, making the danger detection task a continual learning problem. We used Deep Streaming Linear Discriminant Analysis (DeepSLDA) \cite{hayes2019lifelong} for our danger detector. At the end of each episode, we obtain ground truth of the result of the battle and continually update the danger detector.
Deep SLDA works on top of a fixed feature extractor; we pre-trained a feature extractor based on the FullyConv architecture of \citet{vinyals2017starcraft} using data generated from single task experts (agents trained to convergence using a standard RL algorithm for a single task).
\newedit{This component is integrated as an annotator block. The danger detector annotates the observations on the likelihood of if the state is dangerous; by following safe policies during wake and biasing the data collection process, this amounts to a form of prioritized replay used during the sleep phase's memory consolidation.}

\subsection{Self-Supervised Representation Learning for Generalization}

The \textit{Starcraft-2~} evaluation framework can represent observations either as hand-crafted representations provided by the PySC2 simulator or as RGB depictions of the screen. Our proposed system is capable of operating over both types of representation. By default, the system operates over the first type of representation. In practice, agents must learn a  representation from RGB percepts as tasks change over time. In this section, we consider self-supervised continual representation learning.

Motivated by recent work that representation learned using self-supervision auxiliary losses can boost lifelong learning performance~\cite{gallardo2021self}. The intuition is that features learned from an auxiliary losses generalize more than features learned from task-specific losses, and therefore may be less vulnerable to forgetting. The representation is generated from RGB input from two SC-2 tasks, processed by a ResNet18~\cite{he2016deep}, then refined using Barlow Twins~\cite{zbontar2021barlow} self-supervised learning. We chose this approach by validating on object detection performance using Average Precision 50(AP50) as a metric within the SC-2 framework, finding that 1) the ImageNet1k pre-training was beneficial and 2) Barlow Twins outperformed other SSL methods such as MoCo-V2~\cite{chen2020improved} and SimCLR~\cite{chen2020simple}.
\newedit{This component is also integrated as a preprocessor where the self-supervised features of the observations are extracted and made available for the downstream tasks.}

\section{Appendix}

\subsection{Starcraft 2 Task Variants}
\label{sec:sc2_task_descriptions}
The task variants we consider are as follows:

\begin{itemize}
    \item \textbf{Collect Mineral Shards – No Fog of War}: A map with 2 Marines and an endless supply of Mineral Shards. Rewards are earned by moving the Marines to collect the Mineral Shards, with optimal collection requiring both Marine units to be split up and moved independently. Whenever all 20 Mineral Shards have been collected, a new set of 20 Mineral Shards are spawned at random locations (at least 2 units away from all Marines). Fog of war is disabled.
     \item \textbf{Collect Mineral Shards – Fog of war:} A map with 2 Marines and an endless supply of Mineral Shards. Rewards are earned by moving the Marines to collect the Mineral Shards, with optimal collection requiring both Marine units to be split up and moved independently. Whenever all 20 Mineral Shards have been collected, a new set of 20 Mineral Shards are spawned at random locations (at least 2 units away from all Marines). Fog of war is enabled, meaning the agent must be able to learn without full knowledge of the current state of the environment.
     \item \textbf{DefeatZerglingsAndBanelings – One Group:} A map with 9 Marines on the opposite side from a group of 6 Zerglings and 4 Banelings. Rewards are earned by using the Marines to defeat Zerglings and Banelings. Whenever all Zerglings and Banelings have been defeated, a new group of 6 Zerglings and 4 Banelings is spawned and the player is awarded 4 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map
     \item \textbf{DefeatZerglingsAndBanelings – Two Groups:} A map with 9 Marines in the center with two groups consisting of 9 Zerglings one one side and 6 Banelings on the other side. Rewards are earned by using the Marines to defeat Zerglings and Banelings. Whenever a group has been defeated, a new group of 9 Zerglings and 6 Banelings is spawned and the player is awarded 6 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map.
     \item \textbf{DefeatRoaches – One Group:} A map with 9 Marines and a group of 4 Roaches on opposite sides. Rewards are earned by using the Marines to defeat Roaches, with optimal combat strategy requiring the Marines to perform focus fire on the Roaches. Whenever all 4 Roaches have been defeated, a new group of 4 Roaches is spawned and the player is awarded 5 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to opposite sides of the map.
     \item \textbf{DefeatRoaches – Two Groups:} A map with 9 Marines in the center and two groups consisting of 6 total Roaches on opposite sides (3 on each side). Rewards are earned by using the Marines to defeat Roaches, with optimal combat strategy requiring the Marines to perform focus fire on the Roaches. Whenever all 6 Roaches have been defeated, a new group of 6 Roaches is spawned and the player is awarded 7 additional Marines at full health, with all other surviving Marines retaining their existing health (no restore). Whenever new units are spawned, all unit positions are reset to starting areas of the map.
\end{itemize}

\subsection{Implementation Details}
\label{exp:archs}

\subsubsection{Starcraft 2}
\label{sec:imp_sc2}

In this section we describe the implementation details for our \textit{Starcraft-2~}{} agent and experiments. The wake agent is a VTrace agent \citep{espeholt2018impala}. The feature extractor component of the wake agent consists of an architecture that is a slight modification of the FullyConv model of \citet{vinyals2017starcraft}. We further split this into a feature extractor component and a policy network component to facilitate easy weight copying between the wake and sleep agents. For the hidden replay model, we add a LayerNormalization layer to the output of the feature extractor to constrain the space of the features, which helps to improve feature reconstructions and limit feature drift. The architecture produces features of dimension 8192.

The sleep agent feature extractor and policy network architectures are identical to the wake model. We use a five-layer convolutional neural network architecture with for the encoder and a five-layer deconvolutional architecture for the decoder for the VAEs for the sequential and two-headed models. In both cases, we use ReLU activations between the layers and add batch normalization before the ReLUs. We use two-layered multi-layered perceptrons with ReLU activation functions for the encoder and decoder for the VAE for the hidden replay model. To improve numerical stability, the logvar predicted by the encoder is bounded to take values in $[-5, 5]$ using a scaled-hyperbolic tangent layer.

Three styles of replay are used during sleep for the hidden replay architecture: experience replay on a buffer of 10,000 observation-action pairs from a FIFO queue collected during the wake phase, generative replay on the feature vectors, and random replay. Only experience replay and generative replay is used to train the sleep agent for the sequential and two-headed architectures.

For the sequential architecture: the weight of the imitation loss (cross-entropy on action logits) is 50.0, the weight of the reconstruction loss is 1.0, and the weight of the KL loss of the VAE is 3.0. For the two-headed architecture: the weight of the imitation loss (cross-entropy on action logits) is 30.0, the weight of the reconstruction loss is 1.0, and the weight of the KL loss of the VAE is 2.0. For the hidden replay architecture: the weight of the imitation loss (cross-entropy on action logits) is 50.0, the weight of the reconstruction loss is 200.0, and the weight of the KL loss of the VAE is 5.0. These were determined via hand-tuning.

The agent enters sleep three times per task at even intervals for the sequential and two-headed architectures. The agent enters sleep twice per task at even intervals for the hidden replay architecture. Each training iteration of the wake model is trained on 32 trajectories. Each sleep consists of 4,500 iterations of training using batch sizes of 64 (64 wake samples, 64 generated samples, and 64 random observations from the random replay buffer). Generative replay is not used until the second sleep. 96 observations are selected randomly from the wake buffer every sleep to be added to the random replay buffer for the hidden replay architecture. When the agent wakes up, it resets the PPO model and offers advice with a 80\% probability linearly decaying to 0\% after for one-half the the duration between sleeps. No weight copying is used for the sequential variant, but weight copying (running a mini-evaluation block after sleep and copying the best Eigentask's parameters into the wake agent) and advice are both used for the two-headed and hidden replay variants.

The Adam optimizer is used to train both the wake and sleep agents with a learning rate of 1.0e-3.

\textbf{Evaluation:} We selected 3 minigames involving battles  (\emph{DefeatRoaches} and \emph{DefeatZerglingsAndBanelings} tasks) or resource collection (\emph{CollectMineralShards}). We created two variants of each task, differentiated by the starting locations of units, the presence of ``fog-of-war'', and/or the number of friendly and enemy units. Task variants are described in Section~\ref{sec:sc2_task_descriptions}. We used PySC2 \citep{vinyals2017starcraft} to interface with SC-2. We used a subset of the available observation maps namely the unit type, selection status, and unit density two-dimensional observations. The action space is factored into functions and arguments, such as $\text{move}(x, y)$ or $\text{stop}()$. Following \citet{vinyals2017starcraft}, our policy networks output probabilities for the different action factors independently.
The agent receives positive rewards for collecting resources and defeating enemy units, and negative rewards for losing friendly units. Single task experts (STEs) were trained to convergence ($\sim$ 30 million environment steps using the VTrace algorithm \citep{espeholt2018impala}). We consider the pairwise, alternating, and condensed scenarios where each LB consists of 2 million environment steps (about 6\% of the STE samples), and each EB consists of 30 episodes per task.

Each lifetime of a condensed scenario takes approximately 3-4 days running on systems with RTX2080 GPUs.

\subsection{Example Learning Curves for Lifelong Curricula on \textit{Starcraft-2~}{}}

\begin{figure}[H]
\centering
\includegraphics[width=0.95\textwidth]{figs/sc2_learning_curve_cropped.png}
\caption{Learning curve for our lifelong learning agent applied to a six minigame scenario in the \textit{Starcraft-2~}{} domain. Performance comparable to the converged single-task expert is \textasciitilde 100.}
\label{fig:sc2_learning_curves}
\end{figure}

\subsection{Understanding the Interplay Between Different Replay Mechanisms}
\label{sec:justification_of_exemplars}

We've observed that generative replay alone is not sufficient for the hidden replay architecture. This is because there is no constraint limiting the drift in the feature space between sleeps, so the feature extractor can completely change between two time steps if the tasks are perceptually dissimilar or dissimilar in policy. To help to overcome this issue, we employ ``random replay'' where we save a small batch of random observation-action pairs in the original observation space after every sleep, and randomly replay them during every sleep.  In Fig.\ \ref{fig:exemplar_replay_1}, we show the necessity of combining hidden replay with random replay. On the left, we see that using hidden replay without random replay results in the feature space for task 0 dramatically changing after training on task 1, and PM is low. On the right, we see that the distribution of task 0 before and after training on task 1 is significantly better aligned when hidden replay is used in conjunction with random replay, and PM significantly improves. Additional visualizations for pairs of dissimilar tasks are seen in Fig.~\ref{fig:exemplar_replay_2}. In Table \ref{tab:exemplars_vs_no_exemplars}, we see using random replay generally improves PM and FTR, which agrees with our hypothesis that it helps to maintain the feature space, leading to less forgetting and potentially features that generalize better to other tasks.

\begin{table}[H]
    \centering
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & No Random Replay & $-6.28$ ($ \pm 12.82$) & $1.35$ ($ \pm 0.24$) & $0.80$ ($ \pm 0.37$) & $1.08$ ($ \pm 0.19$) \\
         & Random Replay & $-3.03$ ($ \pm 4.92$) & $1.65$ ($ \pm 0.28$) & $0.92$ ($ \pm 0.13$) & $1.04$ ($ \pm 0.17$) \\
         \hline
        Condensed & No Random Replay & $-9.35$ ($ \pm 3.69$) & $1.37$ ($ \pm 0.15$) & $1.20$ ($ \pm 0.24$) & $1.16$ ($ \pm 0.16$) \\
         & Random Replay & $-3.05$ ($ \pm 1.76$) & $1.42$ ($ \pm 0.11$) & $1.00$ ($ \pm 0.03$) & $1.17$ ($ \pm 0.11$) \\
         \hline
    \end{tabular}
    \caption{Comparison of the hidden replay model with and without random replay. Results are averaged over multiple lifetimes, and we also record the standard deviation of the metrics.}
    \label{tab:exemplars_vs_no_exemplars}
\end{table}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figs/m18_exemplars.png}
\caption{We show the necessity of combining hidden replay with random replay. Left: Using hidden replay without random replay results in the feature space for task 0 dramatically changing after training on task 1, and the feature space for both classes converge to a point. PM is -10.68. Right: The distribution of task 0 before and after training on task 1 are significantly better aligned when hidden replay is used in conjunction with random replay. PM is -1.93.}
\label{fig:exemplar_replay_1}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.95\textwidth]{figs/m18_exemplars_2.png}
\caption{We show two examples where hidden + random replay maintains the feature space of the previous task after learning a new dissimilar SC-2 task (``Collect'' $\rightarrow$ ``Defeat'' and vice versa).}
\label{fig:exemplar_replay_2}
\end{figure}

\subsection{Applying STAM to SC-2}
\label{sec:stam_app}

In this section, we describe the details of how STAM can be applied for use in the SC-2 domain.

\paragraph{Background Removal (Preprocessing)}
As a preprocessing step, STAM includes a simple background removal method for SC-2 RGB observations based on erosion and dilation operations on images that exploits the simple structure of the background. The background largely stays the same under erosion and dilation operations, while objects become darker under erosion and brighter under dilation. So, we can extract objects and ignore patches with background by comparing eroded and dilated images and thresholding the background. The threshold is given by a hyperparameter "eps" which determines the threshold to eliminate noise (we suggest eps = 2 for SC-2).

\paragraph{Feature Extraction}
With a given input (RGB Frames in this case) STAM begins by decomposing the image into a potential set of features in the form of overlapping patches. Instead of neurons or hidden layer units, STAM consists of STAM units – in its simplest form, a STAM unit functions as an online clustering module that operates directly on pixel space. Each STAM unit processes one patch, and the patches are overlapping, with a small stride to accomplish translation invariance (similar to CNNs). Patch size is based on the receptive field size, which is a hyperparameter. We used a single receptive field of $16\times16$ based on common object sizes that appear in tasks for SC-2. Therefore, we will assume a single receptive field size (STAM layer) for the rest of the report.

\paragraph{Centroid Learning (Learning)}
Every patch is clustered, in an online manner, to a set of centroids. Given the $m^{th}$ non-background input patch $x_m$ and set of centroids C, the nearest centroid $c_j$ is selected for patch $x_m$  as:
$$c_j=\text{argmin}_{(c \in C)} \; d(x_m, c)$$
Where $d(x_m,c)$ is the Euclidean distance between the patch $x_m$ and centroid c. The selected centroid is updated based on a learning rate parameter $\alpha$ as follows:
$$c_j = \alpha x_m + (1-\alpha) c_j, \;0<\alpha<1$$

\paragraph{Novelty Detection (Learning)}
When an input patch $x_m$ is significantly different than all centroids (i.e., its distance to the nearest centroid is a statistical outlier), a new centroid is created in C based on $x_m$. This function is necessary so that the architecture can learn novel features when the data distribution changes. To do so, we estimate in an online manner the distance distribution between input patches and their nearest centroid. The novelty detection threshold is defined as the 95th percentile ($\beta = 0.95$) of this distance distribution.

\paragraph{Dual-memory Organization (Learning)}
New centroids are stored temporarily in a Short-Term Memory (STM) of limited capacity $\Delta$. If an STM centroid $c_j$ is selected more than $\theta$ times, it is copied to the Long-Term Memory (LTM) for that layer. In the original STAM model, the LTM has (practically) unlimited capacity and a learning rate of zero. However, because of the much larger scale of the SC-2 data and tasks, LTM has been modified to have a fixed capacity. This memory organization is inspired by the Complementary Learning Systems framework where the STM role is played by the hippocampus and the LTM role by the cortex.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\textwidth]{figs/embedding_rep.png}
    \caption{Summary of STAM output embedding creation}
    \label{fig:stam_embed}
\end{figure}

\paragraph{Embedding Creation (Post-processing)}
The centroids learned in the LTM often include shifted versions of the same qualitatively identifiable object due to the nature of extracting overlapping patches with a small stride Figure \ref{fig:stam_embed}-A. This means that each unique object has multiple centroids (with different ID’s) that are associated with it. In order to deal with this, we first pick the most central patch for each object to have an accurate localization Figure \ref{fig:stam_embed}-B. Second, we utilize centroid indices to assign object type identifiers for the central patches Figure \ref{fig:stam_embed}-C. We developed a voting-based method. Each central patch votes for k centroids that it would pick based on distance if its assigned centroid did not exist. Then using accumulated votes, STAM decides on object type identifiers for the centroids. This step is necessary because STAM can learn hundreds of centroids for a single type of object, and we need to map all of these centroids to a unique identifier for consistent representation. Once we assign object type identifiers, we have an output matrix-like Figure \ref{fig:stam_embed}-D.

\subsection{Validating Barlow Twins for SSRL using Object Detection}
\label{sec:barlow_app}
In this section, we provide evidence that Barlow Twins are well-suited as a self-supervised learning (SSL) algorithm for representation learning in the SC-2 domain. Table~\ref{tab:ssrl_od} shows the evaluation of the SSL trained backbones used in the Object Detection pipeline. To ensure that other components of the pipeline(RPN, ROI pooling) do not mask the contribution of the backbone, we ran the experiments on three pipelines (C4, FPN and DC5) offered by Detectron2\citep{wu2019detectron2}.
\begin{table}[H]
    \centering
    \begin{tabular}{|c|c|c|c|}
    \hline Backbone/Pipeline & C4 & FPN & DC5 \\
    \hline
         Barlow Twins & 12.943 & 42.641 & 6.083 \\
         MoCo & 6.083 & 43.110 & 6.675 \\
         Barlow Twins, initialized with ImageNet & 9.382 & 44.325 & 7.841 \\
         MoCo , initialized with ImageNet & 9.693 & 42.802 & 6.395 \\MoCo , initialized with ImageNet & 9.693 & 42.802 & 6.395 \\
         ImageNet & 14.903 & 42.107 & 6.885 \\
  \hline
    \end{tabular}
    \caption{Evaluation of SSL trained backbones using Object Detection as downstream task}
    \label{tab:ssrl_od}
\end{table}

\subsection{Comparisons to Closest Model for Component Extensions}
\label{sec:closest_baseline}
In this section, we compare each component for extending the SC-2 system with its closest comparable model (i.e., a control model) to see how lifelong metrics are affected by adding the component. In all cases (excluding the condensed scenario for danger detection), we try to compare using the same set of syllabi between the baseline agent and extended agent with similar pre-training assumptions. Note that the Two-Headed Model is trained via wake-sleep phases whereas models based on ``Wake'' do not undergo sleep.

\subsubsection{Self-Monitoring Sleep w/ WATCH}

\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Two-Headed Model & $-5.4$ ($ \pm 4.9$) & $1.13$ ($ \pm 0.25$) & $0.84$ ($ \pm 0.12$) & $1.06$ ($ \pm 0.1$) \\
         & Two-Headed Model + Adaptive Sleep & $-1.93$ ($ \pm 5.46$) & $1.46$ ($ \pm 0.54$) & $1.08$ ($ \pm 0.28$) & $1.05$ ($ \pm 0.19$)  \\
         \hline
        Condensed & Two-Headed Mode & $-5.68$ ($ \pm 2.13$) & $1.42$ ($ \pm 0.1$) & $1.14$ ($ \pm 0.12$) & $1.18$ ($ \pm 0.08$) \\
         & Two-Headed Model + Adaptive Sleep & $-0.53$ ($ \pm 4.49$) & $1.43$ ($ \pm 0.66$) & $1.02$ ($ \pm 0.33$) & $0.87$ ($ \pm 0.17$) \\
         \hline
    \end{tabular}
    }
    \caption{Comparison of the two-headed model with and without self-triggering sleep. Results are averaged over multiple lifetimes and we also record the 95\% confidence interval according to the t-distribution. Each sleep agent used two Eigentasks.}
    \label{tab:self_triggering_sleep_comparison}
\end{table}

\subsubsection{Experience Compression: REMIND}
\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Two-Headed Model & $-3.35$ ($ \pm 4.3$) & $1.45$ ($ \pm 0.4$) & $0.92$ ($ \pm 0.15$) & $1.12$ ($ \pm 0.21$) \\
         & Two-Headed Model + REMIND & $-2.23$ ($ \pm 2.33$) & $1.58$ ($ \pm 0.22$) & $0.93$ ($ \pm 0.22$) & $0.97$ ($ \pm 0.13$) \\
         \hline
          Alternating & Two-Headed Model & $-8.44$ ($ \pm 5.35$) & $1.04$ ($ \pm 0.31$) & $0.81$ ($ \pm 0.13$) & $0.89$ ($ \pm 0.07$) \\
         & Two-Headed Model + REMIND & $-6.51$ ($ \pm 4.58$) & $1.29$ ($ \pm 0.35$) & $0.78$ ($ \pm 0.2$) & $0.92$ ($ \pm 0.04$) \\
         \hline
        Condensed & Two-Headed Model & $-4.08$ ($ \pm 4.17$) & $1.52$ ($ \pm 0.25$) & $1.2$ ($ \pm 0.47$) & $1.1$ ($ \pm 0.2$) \\
         & Two-Headed Model + REMIND & $-3.67$ ($ \pm 3.92$) & $1.5$ ($ \pm 0.18$) & $1.13$ ($ \pm 0.42$) & $1.11$ ($ \pm 0.18$) \\
         \hline
    \end{tabular}
    }
    \caption{Comparison of the two-headed model with and without a compressed wake experience buffer (REMIND). Results are averaged over 12 lifetimes and we also record the 95\% confidence interval according to the t-distribution. Both agents use pre-training on two tasks not in the test syllabus. Each sleep agent used two Eigentasks.}
    \label{tab:remind_comparison}
\end{table}

\subsubsection{Prioritized Replay for Danger Detection}
\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Wake & $-0.37$ ($ \pm 2.58$) & $1.62$ ($ \pm 0.38$) & $0.99$ ($ \pm 0.12$) & $1.05$ ($ \pm 0.05$) \\
         & Wake + Danger Detection & $0.39$ ($ \pm 2.18$) & $1.73$ ($ \pm 0.43$) & $1.04$ ($ \pm 0.21$) & $0.98$ ($ \pm 0.05$) \\
         \hline
          Alternating & Wake & $-1.32$ ($ \pm 1.35$) & $1.62$ ($ \pm 0.38$) & $0.96$ ($ \pm 0.07$) & $1.02$ ($ \pm 0.02$)\\
         & Wake + Danger Detection & $1.79$ ($ \pm 2.77$) & $1.73$ ($ \pm 0.43$) & $1.16$ ($ \pm 0.26$) & $0.95$ ($ \pm 0.02$) \\
         \hline
        Condensed & Wake* & $-3.72$ ($ \pm 3.95$) & $1.23$ ($ \pm 0.24$) & $1.0$ ($ \pm 0.07$) & $1.17$ ($ \pm 0.12$) \\
         & Wake + Danger Detection & $0.01$ ($ \pm 2.05$) & $1.55$ ($ \pm 0.34$) & $1.29$ ($ \pm 0.4$) & $1.04$ ($ \pm 0.04$)\\
         \hline
    \end{tabular}
    }
    \caption{Comparison of the wake agent with and without danger detection. Results are averaged over 8 lifetimes and we also record the 95\% confidence interval according to the t-distribution. Each syllabus consists of only ``defeat'' tasks from the SC-2 minigames, with the exclusion of the condensed scenario for the wake-only agent, which is run on tasks starting with two defeat tasks, but include collect tasks within the syllabi.}
    \label{tab:danger_detection_comparison}
\end{table}

\subsubsection{Representation Learning: STAM}
\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Wake &$-6.82$ ($ \pm 10.43$) & $1.15$ ($ \pm 0.37$) & $0.95$ ($ \pm 0.37$) & $1.01$ ($ \pm 0.09$) \\
         & Wake + STAM & $-4.53$ ($ \pm 6.65$) & $1.48$ ($ \pm 0.33$) & $0.87$ ($ \pm 0.31$) & $0.96$ ($ \pm 0.07$) \\
         \hline
          Alternating & Wake & $-8.34$ ($ \pm 5.37$) & $1.15$ ($ \pm 0.37$) & $0.86$ ($ \pm 0.22$) & $0.91$ ($ \pm 0.09$) \\
         & Wake + STAM & $-1.28$ ($ \pm 2.16$) & $1.48$ ($ \pm 0.33$) & $0.84$ ($ \pm 0.2$) & $0.92$ ($ \pm 0.04$) \\
         \hline
        Condensed & Wake & $-0.09$ ($ \pm 0.33$) & $1.02$ ($ \pm 0.02$) & $1.0$ ($ \pm 0.02$) & $1.07$ ($ \pm 0.07$) \\
         & Wake + STAM & $1.12$ ($ \pm 1.89$) & $1.18$ ($ \pm 0.05$) & $1.1$ ($ \pm 0.09$) & $0.91$ ($ \pm 0.09$) \\
         \hline
    \end{tabular}
    }
    \caption{Comparison of the wake agent with and without STAM. Results are averaged over 12 lifetimes and we also record the 95\% confidence interval according to the t-distribution.}
    \label{tab:stam_comparison}
\end{table}

\subsubsection{Representation Learning: Self-Supervised Learning}

\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Wake & $-7.47$ ($ \pm 11.45$) & $1.16$ ($ \pm 0.42$) & $0.94$ ($ \pm 0.42$) & $1.01$ ($ \pm 0.11$) \\
         & Wake + SSRL & $-12.75$ ($ \pm 12.46$) & $1.5$ ($ \pm 0.22$) & $0.67$ ($ \pm 0.27$) & $1.09$ ($ \pm 0.11$) \\
         \hline
        Condensed & Wake & $-0.13$ ($ \pm 0.25$) & $1.01$ ($ \pm 0.02$) & $1.0$ ($ \pm 0.02$) & $1.13$ ($ \pm 0.09$) \\
         & Wake + SSRL & $-7.46$ ($ \pm 4.19$) & $1.52$ ($ \pm 0.18$) & $1.37$ ($ \pm 0.42$) & $1.14$ ($ \pm 0.11$) \\
         \hline
         \end{tabular}
    }
    \caption{Comparing the wake agent with and without SSRL. Results are averaged over 12 lifetimes and we also record the 95\% confidence interval according to the t-distribution.}
    \label{tab:ssrl_comparison}
\end{table}

\begin{table}[H]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline
         Scenario & Agent & PM & FT & BT & RP \\
         \hline
          Pairwise & Hidden Replay Model + Pretraining & $-4.63$ ($ \pm 3.76$) & $1.0$ ($ \pm 0.05$) & $0.84$ ($ \pm 0.12$) & $1.07$ ($ \pm 0.12$) \\
         & Wake + SSRL & $-6.52$ ($ \pm 2.67$) & $1.2$ ($ \pm 0.5$) & $0.6$ ($ \pm 0.17$) & $0.91$ ($ \pm 0.07$) \\
         \hline
         \end{tabular}
    }
    \caption{Understanding the performance of SSRL on a small set of pairwise syllabi. Results are averaged over multiple lifetimes and we also record the 95\% confidence interval according to the t-distribution. The comparable baseline model is a hidden replay model with pretraining on the four tasks that do not appear in the pairwise syllabus for each run.}
    \label{tab:ssrl_comparison_2}
\end{table}