text:
\section{Conclusion}
In this paper, we propose a new method, SS-IL, that addresses the score bias and the bias preserving property of GKD. Based on the systematic analysis on gradients of ordinary cross entropy, we find the root cause of the score bias is in softmax. By using separated softmax (SS) and ratio preserving (RP) mini-batch, we solve this issue without any score post-processing. Furthermore, we also find that such bias is rather preserved by GKD and TKD will not get affected by the task-wise bias. The experiment results show that our SS and RP mini-batch with TKD achieve the highest performance compared to the other state-of-the-art methods and balanced prediction among tasks. In analyses on KD, we observe the existence of bias on GKD and we show TKD always outperforms GKD, which means TKD is a better choice for preserving the knowledge.

\section{Experiments}
In this section, we compare our SS-IL with other state-of-the-art CIL methods with various experimental scenarios. For evaluation, we use two large scale datasets: ILSVRC 2012 (ImageNet) \cite{(Imagenet)deng2009} and Google Landmark Dataset v2 (Landmark-v2) \cite{(Landmark-v2)}. In addition, extensive analyses are carried out to show the effectiveness of SS-IL, and the importance of each component consisting the proposed method is analyzed through ablation study. Also, in detailed analyses about the distillation methods, we show the excellence of $\mathcal{L}_{\text{TKD}}$ by comparing $\mathcal{L}_{\text{TKD}}$ and $\mathcal{L}_{\text{GKD}}$.
\subsection{Datasets and evaluation protocol}
For ImageNet and Landmark-v2 datasets, we use all classes in ImageNet dataset, and choose 1,000 and 10,000 classes in Landmark-v2 dataset to make two variations. The detailed explanation on each dataset is as follows:

\noindent\textbf{ImageNet and Landmark-v2}:\ \
ILSVRC 2012 dataset consists of 1,000 classes, which has nearly 1,300 images per class. Google Landmark Dataset v2 consists of 203,094 classes, and each class has $1\sim10,247$ images. We construct Landmark-v2-1K and Landmark-v2-10K which are composed of 1,000 and 10,000 classes respectively.

\noindent\textbf{Evaluation protocol}:\ \
By following the benchmark protocol in \cite{(icarl)rebuffi17}, we arrange the classes of each dataset in a fixed random order. To construct various training scenarios, we vary the total number of incremental tasks as $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ in 1K datasets and $m=\{2000,1000,500\}$ in 10K dataset, respectively. For the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$ for 1K datasets and $|\mathcal{M}|=\{20k,40k,60k\}$ for 10K dataset, respectively. We use the Ringbuffer approach used in \cite{(ER)Chaudhry2019} for constructing the exemplars. For the evaluation of CIL models, we use ILSVRC 2012 validation set for ImageNet-1K, and we randomly selected test samples that are not in the training set.

The details on constructing the datasets and the evaluation protocol are explained in Supplementary Materials.

\subsection{Implementation detail}
The Resnet-18 \cite{(Resnet)HEZhang15} architecture is used in all experiments, and all the implementations are done with the Pytorch framework\cite{(PyTorch)Paszke2017}. For training the neural network, we always use the stochastic gradient descent (SGD) with learning rate 0.1, weight decay 0.0001, and Nesterov momentum 0.9. The batch size used for $\mathcal{D}_t$, $N_{\mathcal{D}_t}$, is 128, and we use different replay batch size, $N_{\mathcal{M}}$, depending on the number of different incremental tasks; \ie,  $N_{\mathcal{M}}=16 / 32 / 64$ for $T=20 / 10 / 5$, respectively. Thus, the ratio of $N_{\mathcal{D}_t}$ over $N_{\mathcal{M}}$ is $8 /4/2$, respectively. The number of epochs for training incremental task is 100, and the learning rate is divided by 10 at epochs 40 and 80.

We compare our SS-IL with iCaRL\cite{(icarl)rebuffi17}, vanilla Fine-Tuning (FT) proposed in \cite{(IL2M)Belouadah}, IL2M\cite{(IL2M)Belouadah}, EEIL\cite{(End2EndIL)Castro2018}, and BiC\cite{(LargeScaleIL)Wu2019}. For iCaRL, as proposed in \cite{(Revisiting)Javed2018}, instead of using binary cross entropy loss for each class output, we use multi-class cross entropy loss for both classification loss and KD loss, which achieves much higher accuracy than the original paper. In Supplementary Materials, all the training details for baselines are explained. For data pre-processing, the random re-sized cropping and horizontal flipping is adopted to all datasets as data augmentation, and normalization with mean and standard deviation is performed only for the ImageNet dataset.

\subsection{Results}\label{subsec:results}
Table \ref{table:Overall_results} shows the results on Average Top-1 and Top-5 accuracy. The left half of the table reports the results of fixed $T=10$ with varying exemplar-memory size $|\mathcal{M}|$, and the right half shows the results of fixed $|\mathcal{M}|$ with varying $T$.
\begin{figure*}[ht]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/ablation.pdf}

    \vspace{-.07in}
    \caption{(a) Ablation study on SS-IL. (b)$\sim$(e) Confusion matrix of four different variations: (b) SS-IL, (c) $\mathcal{L}_{\text{TKD}}$ w/SS, (d) $\mathcal{L}_{\text{TKD}}$ w/RP, (e) $\mathcal{L}_{\text{TKD}}$}\label{fig:Ablation}
    \vspace{-.15in}

\end{figure*}

From the table, we can make the following observations. Firstly, among the baselines, there is no clear winner; EEIL tends to excel for small $|\mathcal{M}|$, while BiC achieves much higher accuracy than other baselines  for large $|\mathcal{M}|$. Secondly, SS-IL consistently dominates other baselines for all the large-scale datasets throughout every possible scenarios. In particular, SS-IL outperforms other state-of-the-art baselines with significant margin in ImageNet-1K and Landmark-v2-10K. Lastly, when smaller $|\mathcal{M}|$ is used, the accuracy drop is marginal in all datasets, which indicates SS-IL is somewhat robust to $|\mathcal{M}|$.

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/Top-5.pdf}
    \vspace{-.15in}
    \caption{Incremental learning results on ImageNet-1K, Landmark-1K, and Landmark-10K datasets for $T=10$. The exemplar size is $|\mathcal{M}|=20k$ in ImageNet-1K and Landmark-1K datasets, and $|\mathcal{M}|=60k$ in Landmark-v2-10K dataset.}\label{fig:Top_5}
    \vspace{-.13in}
\end{figure}

Figure \ref{fig:Top_5} shows the overall result on each dataset with respect to the incremental task, when $|\mathcal{M}|=20k$ and $T=10$, and the tasks are denoted as classes. In this figure, we denote jointly trained approach as the Upper-bound. Note that SS-IL again mostly dominates the baselines, and the performance gap over the baselines widens as the incremental task increases. Especially, in ImageNet-1K, compared with other baselines which have more performance degradation from the Upper-bound, our SS-IL is less affected by catastrophic forgetting. Furthermore, we observe that iCaRL and EEIL achieves lower accuracy in the first incremental task. Weak Nearest Exemplar Mean (NEM) classifier in iCaRL and inefficient training schedule in EEIL are the main reasons of low accuracy.

\begin{comment}
Figure \ref{fig:Analysis}(a) shows the misclassified error ratios $e_t(p,n)$ and $e_t(n,p)$ more in details for ImageNet-1K dataset with $T=10$ and $|\mathcal{M}|=10k$. From the figure, which essentially adds the results of iCaRL and SS-IL to Figure \ref{fig:Bias_motivation}, we first observe that $e_t(p,n)$ of FT and IL2M drastically increases as the incremental task increases, while $e_t(n,p)$ is significantly small due to the classification score bias. Namely, the most of errors they make are simply due to predicting most of the old classes as the new classes. We note IL2M does improve upon FT after the post-processing, but it still suffers from the heavy bias.
On the other hand, for iCaRL, we observe that $e_t(p,n)$ is significantly lower than those of FT and IL2M, while $e_t(n,p)$ is now much higher than those of FT and IL2M. This suggests that the knowledge distillation of iCaRL does induce to make more predictions toward the old classes, but the low accuracy shows that it hinders the accurate predictions within the old classes.
Furthermore, we clearly observe that our SS-IL shows the most balanced error ratios among the methods. That is, we note $e_t(p,n)$ for SS-IL is significantly lower than those of FT and IL2M (similar to that of iCaRL) and $e_t(n,p)$ also stays more or less similar to $e_t(p,n)$ throughout the incremental tasks. This confirms that our SS layer learned with RP mini-batch makes much more balanced classification decisions, resulting in much fewer confusions between the old and new classes, hence, higher prediction accuracy as shown in Figure \ref{fig:Top_5}.
\end{comment}

\subsection{Ablation study}\label{subsec:ablation}
In this section, we perform various detailed analyses to show the effectiveness of replay batch sizes while varying the total number of incremental tasks. Also, we do ablation study on the components of our SS-IL method and demonstrate their impact.

\noindent\textbf{Ablation study on SS and RP}
In this section, we validate our approach by ablating each component of SS-IL. Figure \ref{fig:Ablation} shows the ablation study results for ImageNet-1K with $|\mathcal{M}|=10k$, $T=10$. In this figure, ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' stands for the model that selects mini-batches for SGD as in our SS-IL, but does not have the separated softmax layer, ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' stands for the model that has the separated softmax layer as our SS-IL but randomly selects mini-batches from $\mathcal{D}_t\cup\mathcal{M}$, and ``$\mathcal{L}_{\text{TKD}}$'' stands for the model that only uses $\mathcal{L}_{\text{TKD}}$ without SS and RP. In Figure \ref{fig:Ablation} (a), thanks to the effectiveness of SS, ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' achieves higher accuracy than ``$\mathcal{L}_{\text{TKD}}$'', and by comparing Figure \ref{fig:Ablation} (c) and (e), using SS makes much more balanced decisions. Furthermore, we observe that ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' achieves almost same accuracy as SS-IL. One may think that using RP has no effect on CIL problem. However, a comparison of Figure (b) and (c) shows that SS-IL makes more balanced decisions than ``$\mathcal{L}_{\text{TKD}}$ w/ SS'', which means using SS and RP together achieves the highest accuracy and makes more balanced decisions. Interestingly, ``$\mathcal{L}_{\text{TKD}}$'' largely outperforms ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' which uses RP only, and the decision of ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' is highly biased toward new classes.

\begin{table}[h]
\small
\centering
\vspace{-.0in}
\caption{Results on ImageNet-1K with varying $N_\mathcal{M}$ and $T$.
}\label{table:batch_state_results}
\vspace{.0in}
\resizebox{1.0\columnwidth}{!}{\begin{tabular}{cc|c}
\hline
\multicolumn{1}{c|}{$T$/$N_\mathcal{M}$} & 16 / 32 / 64        & 16 / 32 / 64        \\ \hline\hline
                             & Average Top-1 accuracy    & Average Top-5 accuracy   \\ \hline\hline
\multicolumn{1}{c|}{20}      & 58.8 / 59.0 / 58.9 & 82.9 / 82.6 / 82.4 \\
\multicolumn{1}{c|}{10}      & 64.3 / 64.5 / 64.1 & 86.6 / 86.4 / 86.0 \\
\multicolumn{1}{c|}{5}       & 68.4 / 68.4 / 68.2 & 88.8 / 88.6 / 88.4 \\ \hline
\end{tabular}}
\vspace{-.05in}
\end{table}

\noindent\textbf{Analysis on $N_{\mathcal{M}}$}\ \ Table \ref{table:batch_state_results} shows the results on Average Top-1 and Top-5 accuracy with respect to varying replay batch size, $N_\mathcal{M}$, and the total number of incremental tasks, $T$, for ImageNet-1K on $|\mathcal{M}|=10k$. From Table \ref{table:batch_state_results}, we observe that no matter what $N_{\mathcal{M}}$ is being used, the accuracy difference is negligible. This indicates that, using RP is still effective
regardless of the ratio between old and new class samples in the mini-batch,
if the old class examples are guaranteed to some extent.

\subsection{Analyses on KD}\label{subsec:kd}

In this section, given different bias correction schemes, we carry out several experiments to compare TKD with GKD and FT. We use three different bias correction schemes: balanced fine-tuning (BFT) \cite{(End2EndIL)Castro2018}, score correction \cite{(LargeScaleIL)Wu2019}, and SS+RP (Ours). For a fair comparison, same training settings are used for TKD and GKD. Note that we use same FT as we described in section \ref{subsec:results}. All the training details are explained in Supplementary Materials.

\noindent\textbf{Comparison of $\mathcal{L_{\text{TKD}}}$ and  $\mathcal{L_{\text{GKD}}}$} Table \ref{table:local_global_acc} shows the  Average Top-1 and Top-5 accuracy with respect to varying KD loss and bias correction method.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\columnwidth]{figures/global_to_local.pdf}
    \vspace{-.1in}
    \caption{Top-1 accuracy for $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$ on two different bias correction schemes at $t=3,6,9$}\label{fig:global_to_local}
    \vspace{-.12in}
\end{figure}

\begin{table}[h]
\small
\centering
\vspace{.1in}
\caption{Results on ImageNet-1K with varying the KD losses and bias correction schemes
}\label{table:local_global_acc}
\vspace{.05in}
\resizebox{1.0\columnwidth}{!}{\begin{tabular}{ccc}
\hline
\multicolumn{1}{c|}{Schemes / KD losses} & \multicolumn{1}{c|}{FT / GKD / TKD}      & FT / GKD / TKD       \\ \hline\hline
                                       & Average Top-1 accuracy                     & Average Top-5 accuracy \\ \hline\hline
\multicolumn{1}{c|}{No bias correction}          & \multicolumn{1}{c|}{53.6 / 54.7 / \textbf{56.8}} & 78.8 / 77.3 / \textbf{81.8}  \\ \hline
\multicolumn{1}{c|}{Balanced Fine-Tuning}    & \multicolumn{1}{c|}{61.3 / 63.6 / \textbf{64.8}} & 83.6 / 85.3 / \textbf{86.2}  \\ \hline
\multicolumn{1}{c|}{Score Correction }            & \multicolumn{1}{c|}{53.6 / 63.3 / \textbf{63.8}} & 78.8 / 84.3 / \textbf{85.3}  \\ \hline
\multicolumn{1}{c|}{SS + RP}              & \multicolumn{1}{c|}{59.3 / 64.5 / \textbf{65.3}} & 82.9 / 86.5 / \textbf{86.7}  \\ \hline
\end{tabular}}
\vspace{-.1in}
\end{table}

Models are trained with GKD, TKD, and without KD (\ie FT) while applying different bias correction schemes for ImageNet-1K on $|\mathcal{M}|=20k$, $T=10$. As shown in Table \ref{table:local_global_acc}, compared to FT, TKD achieves the highest accuracy in every case while GKD is not always effective. According to \cite{(IL2M)Belouadah,(scail)belouadah2020}, using distillation in CIL which inherently includes class imbalance causes a drop of performance. However, we observe better performance of TKD over FT even in the absence of bias correction method, which implies that using TKD can be effective. Similarly, \cite{(UnlabeledIL)Lee2019} mention that using TKD misses the knowledge about discrimination between old tasks. Meanwhile, our result implies that TKD may be a better way of using KD in CIL. Overall, TKD is an effective way to prevent catastrophic forgetting in exemplar-memory based CIL.

\noindent\textbf{Existence of bias on $\mathcal{L}_{\text{GKD}}$} In this section, for further analysis on $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$, we carry out another experiment with newly designed training scenario. The scenario is as follows:
\begin{enumerate}
    \item Train a model using $\mathcal{L}_{\text{GKD}}$ until incremental task $t-1$, then we obtain $\bm \theta_{t-1}$.
    \vspace{-.07in}
    \item At incremental task $t$, train $\bm \theta_{t-1}$ using two different KD losses, $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$. As a result, we obtain two different models, $\bm \theta_{\text{GKD},t}$ and $\bm \theta_{\text{TKD},t}$.
\end{enumerate}

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/prediction_map.pdf}
    \vspace{-.15in}
    \caption{Task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$}\label{fig:prediction_map}
    \vspace{-.15in}
\end{figure}

We apply above scenario to the models that use BFT and SS+RP. As a result, two different KD models are obtained per each bias correction method, and we evaluate each model on same test data. Note that since identical $\bm \theta_{t-1}$ is used for KD, we can directly compare $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$.

Figure \ref{fig:global_to_local} shows the Top-1 accuracy of $\bm \theta_{\text{GKD},t}$ and $\bm \theta_{\text{TKD},t}$ on BFT and SS+RP at $t=3,6,9$, and Figure \ref{fig:prediction_map} shows the task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$.

In Figure \ref{fig:global_to_local}, for the same $\bm \theta_{t-1}$, the accuracy of $\bm \theta_{\text{TKD},t}$ is higher than that of $\bm \theta_{\text{GKD},t}$ at task 3, 6, 9. We assume that the main reason for the accuracy difference is due to the bias preservation caused by the score bias of $\bm \theta_{t-1}$ on $\bm x \in \mathcal{D}_t$. As shown in Figure \ref{fig:prediction_map}, the ratio of the prediction on the latest old task is higher than the ratio of any other tasks. For example, at incremental task 9, the predictions for $\bm x \in \mathcal{D}_9$ are predominantly made toward task 8. Therefore, since $\bm p^{\tau}_{1:t-1}(\bm x, \bm \theta_{t-1})$ used in  (\ref{eq:GKD_loss}) can be heavily biased due to the biased score, $\mathcal{L}_{\text{GKD}}$ rather induces catastrophic forgetting. On the other hand, similar to the intuition of SS, TKD performs distillation using task specific separated softmax which is not affected by score bias between tasks. As a result, using TKD in CIL can be a better choice compared to using GKD.

\section{Introduction}

Incremental or continual learning, in which the agent continues to learn with incremental arrival of new training data, is one of the grand challenges in artificial intelligence and machine learning.
Such setting, which does not assume the full availability of old training data, is recently gaining more attention particularly from the real-world application perspective. The reason is because storing all the training data, which can easily become large-scale, in one batch often becomes unrealistic for memory- and computation-constrained applications, such as mobile phones or robots, hence the continuous yet effective update of the learning agent without accessing the full data received so far is indispensable.

A viable candidate for such agent is the end-to-end learning based deep neural network (DNN) models. Following the recent success of DNN in many different applications \cite{HinLecBen15,(AutonomousDriving)Al-Qizwini,(Medical)Caruana}, the DNN-based incremental learning methods have been also actively pursued in recent years. Although they achieved some promising results, they also possess a critical limitation: the \emph{catastrophic forgetting}, which refers to the problem that the generalization performance on the old data severely degrades after a naive fine-tuning of the model with the new data.

In this paper, we focus on the DNN-based \emph{class} incremental learning (CIL) problem, which we refer to learning a classifier to classify \emph{new} object classes from every incremental training data and testing the classifier on all the classes learned so far.
Among several different proposed approaches, the exemplar-memory based approaches \cite{(icarl)rebuffi17,(End2EndIL)Castro2018,(LargeScaleIL)Wu2019,(WA)Zhao2020,(IL2M)Belouadah,(scail)belouadah2020}, which allows to store small amount of training data from old classes in a separate memory,
has attained promising results. It has been shown that using the small exemplar memory plays an important role in mitigating the catastrophic forgetting, and allowing such small size of memory while learning is also tolerable in practical scenarios as well.

The main challenge of using the exemplar-memory is to resolve the severe data imbalance issue between the training data points for the new classes and those for the old classes in the exemplar-memory. That is, the naive fine-tuning with such imbalanced data may still heavily skew the predictions toward the newly learned classes, hence, the accuracy for the old classes would dramatically drop,
again resulting in significant forgetting.
Recent state-of-the-arts \cite{(End2EndIL)Castro2018,(LargeScaleIL)Wu2019,(WA)Zhao2020,(IL2M)Belouadah,(scail)belouadah2020} found that significantly higher classification scores for the newly learned classes is the main reason for such prediction bias, thus, they proposed to correct the score bias by some additional post-processing, \textit{e.g.}, score re-scaling or balanced fine-tuning, after learning the model.

While above mentioned methods were effective to some extent in terms of improving the accuracy, we argue that they lack systematic analysis on the root cause of such bias and some component of their schemes, \textit{e.g.}, knowledge distillation (KD) \cite{(Distill)HintonOriolDean15} , was naively used without proper justification \cite{(LargeScaleIL)Wu2019,(UnlabeledIL)Lee2019,(WA)Zhao2020,(ConfCalIncre)Kang2020}. To that regard, in this paper, we first analyze the root cause of such prediction score bias, then propose a method that mitigates the cause in a sensible way. Namely, we argue that the bias is injected by the fact that the softmax probability used in the usual cross-entropy loss is always computed by combining the output scores of \textit{all} classes, which forces the heavy penalization of the output probabilities for the old classes due to data imbalance. Furthermore, we show that a naive use of the General KD (GKD) method, which also combines the output scores of \textit{all} old classes to compute the soft target, may preserve the bias and even hurt the accuracy, if the prediction bias is already present in the model.

To resolve above issues, we propose Separated Softmax for Incremental Learning (SS-IL), which consists of three components. Firstly, we devise
\textit{separated softmax (SS)} output layer that mutually blocks the flow of the score gradients between the old and new classes, hence, mitigates the imbalanced penalization of the output probabilities for the old classes. Secondly, for the mini-batch stochastic gradient descent (SGD) steps, we utilize \textit{ratio-preserving (RP)} mini-batches that guarantee the minimum ratio of the samples from old classes steps such that a balance between learning the new classes and preserving the knowledge for old classes can be realized. Thirdly, we show the \textit{Task-wise KD (TKD)}, which also computes the soft-target for distillation in a task-separated manner, is particularly well-suited for our SS layer, since it attempts to preserve the task-wise knowledge without preserving the prediction bias that may present among tasks. In our extensive experimental results, we show that our SS-IL achieves very strong state-of-the-art accuracy on various different large-scale CIL benchmark datasets. We also stress that our method significantly mitigates the prediction score bias without any additional post-processing steps, unlike other recent baselines. We also present systematic ablation study on our method, justifying each of the three components in SS-IL.

\section{Related Work}
In this section, we summarize algorithms related to continual learning and class incremental learning (CIL). Here, the algorithms denoted by ``continual learning'' make an assumption that task information is available at the test time, and the algorithms denoted by ``class incremental learning'', the focus of this paper, does not make such assumption. Hence, the CIL deals with a more challenging setting than the continual learning algorithms mentioned below.

\noindent\textbf{Memory based continual learning}\ \ By utilizing the information in exemplar memory, \cite{(GEM)LopezRanzato17} solves the constrained optimization problem by using gradients for each task using exemplars. However, due to hard constraint on the gradient of each task, new tasks become harder to learn. In \cite{(A-GEM)Chaudhry2019}, they propose the relaxed version of constraint which tries to compute the average of the gradient, and solves a much simpler constrained optimization problem. In \cite{(ER)Chaudhry2019}, instead of computing gradients, they concatenate two batches which are drawn from exemplar and current task dataset.

\noindent\textbf{Generative replay based CIL}\ \
By generating the auxiliary data on previous tasks, \cite{(DGR)ShinLeeKimKim17} generate the previous task data using Generative Adversarial Network(GAN)\cite{(GAN)goodfellow2014} and consider the ``continual learning'' scenario.
So far, we introduce the ``continual learning'' algorithms. Form now on, we introduce ``class incremental learning'' algorithms.
In \cite{(Fearnet)kemker18}, they generate the features of old classes from a pre-trained model using stored class statistics, and apply it to the ``class incremental learning'' scenario. In different way, \cite{(CGANIL)Xiang2019} tries to generate the intermediate feature of pre-trained network using Conditional GAN\cite{(ConditionalGAN)Mirza2014}.

\noindent\textbf{KD based CIL and bias removal methods}\ \ The earliest of KD based methods, \cite{(LwF)LiHoiem16}, uses KD to keep the scores of previous tasks when learning new tasks. The developed version of \cite{(LwF)LiHoiem16}, iCaRL \cite{(icarl)rebuffi17}, which uses memory exemplar, preserves the feature using KD and then classifies the classes using the Nearest Mean of Exemplars (NME) classification. However, the methods using exemplar, such as iCaRL, have been shown experimentally that there is a bias in the final FC layer.

To tackle this problem, bias removal techniques \cite{(End2EndIL)Castro2018,(UnlabeledIL)Lee2019,(LargeScaleIL)Wu2019, (WA)Zhao2020, (ConfCalIncre)Kang2020, (IL2M)Belouadah, (scail)belouadah2020} are proposed. In \cite{(End2EndIL)Castro2018}, to remove the prediction bias, they proposed a balanced fine-tuning method, which is fine-tuning the network using a balanced dataset. Another balanced fine-tuning approach \cite{(UnlabeledIL)Lee2019} proposed a gradient scaling to remove the prediction bias. In \cite{(LargeScaleIL)Wu2019, (WA)Zhao2020,(ConfCalIncre)Kang2020}, after the training process, \cite{(LargeScaleIL)Wu2019} corrects the bias of scores using bias correction layer with validation set, \cite{(WA)Zhao2020} corrects the biased weights in the FC layer, and \cite{(ConfCalIncre)Kang2020} scales the scores of new classes in training time. The common property of these approaches is that all of them use KD to preserve the knowledge of the past tasks.
Unlike the above methods, \cite{(IL2M)Belouadah, (scail)belouadah2020} correct the score bias without any distillation methods. Using the statistics on each task, \cite{(IL2M)Belouadah} rectifies the output softmax probability and \cite{(scail)belouadah2020} scales the classifier weights.

\section{Preliminaries}\label{sec:notation}
\subsection{Notations and problem setting}

In CIL, we assume every incrementally-arrived training data, which is often called as the incremental \textit{task}, consists of data for \textit{new} $m$ classes that have not been learned before.
More formally, the training data for the incremental task $t$ is denoted by $\mathcal{D}_t=\{(\bm x_{t}^{(i)},  y_{t}^{(i)})\}_{i=1}^{n_{t}}$, in which $\bm x_{t}^{(i)}$,
 $y_{t}^{(i)}$, and $n_{t}$ denote input data for task $t$, the corresponding (integer-valued) target label, and the number of training samples for the corresponding task, respectively. The total number of classes up to task $t$ is denoted by $C_t=mt$, which leads to the labeling $y_{t}^{(i)}\in\{C_{t-1}+1,\ldots,C_{t}\}\triangleq \mathcal{C}_t$.
During learning each incremental task, we assume a separate exemplar-memory $\mathcal{M}$  is allocated to store exemplar data for old classes. Namely, when learning the incremental task $t$, we store $\lfloor\frac{|\mathcal{M}|}{C_{t-1}}\rfloor$ data points from each class that are learnt until the incremental task $t-1$. Thus, as the incremental task grows, the number of exemplar data points stored for each class decreases linearly with $t$ and we assume $|\mathcal{M}|\ll n_{t}$. The total number of incremental tasks is denoted by $T$.

Our classification model consists of a feature extractor, which has the deep convolutional neural network (CNN) architecture, and the classification layer, which is the final fully-connected (FC) layer with softmax output. We denote $\bm \theta$ as the parameters for our classification model. At incremental task $t$, the parameters of the model, $\bm \theta_t$, are learned using data points in $\mathcal{D}_t\cup \mathcal{M}$. After learning, the class prediction for a given sample $\bm x_{\text{test}}$ is obtained by
\begin{eqnarray}

 \hat{y}_{\text{test}}=\arg\max_{y\in\mathcal{C}_{1:t}}z_{ty}(\bm x_{\text{test}}, \bm \theta_t),\label{eq:test}
\end{eqnarray}
in which $z_{ty}(\bm x_{\text{test}},\bm\theta_t)$ is the output score (before softmax) of the model $\bm \theta_t$ for class $y\in \mathcal{C}_{1:t}\triangleq\{1,\ldots,C_t\}$. Then, we denote the task which $\hat{y}_{\text{test}}$ belongs as $\hat{t}_{\text{test}}=i \ \ \text{for} \ \ \hat{y}_{\text{test}}\in\mathcal{C}_i$.
Namely, at test time, the final FC layers are consolidated and the prediction among all classes in $\mathcal{C}_{1:t}$ is made as if by an ordinary multi-class classifier.

\subsection{Knowledge distillation}

In several recent CIL methods, knowledge distillation (KD) is often used to maintain the knowledge learned from the past tasks. There are two variations of KD, General KD (GKD) and Task-wise KD (TKD), and the loss function defined for each method for learning task $t$ is as follows: for an input data $\bm x\in\mathcal{D}_t\cup\mathcal{M}$,
\begin{align}
    &\mathcal{L}_{\text{GKD}, t}(\bm x, \bm \theta) \triangleq \mathcal{D}_{KL}(\bm p^\tau_{1:t-1}(\bm x, \bm \theta_{t-1}) \|\bm p^\tau_{1:t-1}(\bm x, \bm \theta)) \label{eq:GKD_loss} \\
    &\mathcal{L}_{\text{TKD}, t}(\bm x, \bm \theta) \triangleq \sum_{s=1}^{t-1}\mathcal{D}_{KL}(\bm p_s^\tau(\bm x, \bm \theta_{t-1}) \|\bm p_s^\tau(\bm x, \bm \theta)) \label{eq:TKD_loss},
    \vspace{-.2in}
\end{align}
in which $\mathcal{D}_{KL}(\cdot\|\cdot)$ is the Kullback-Leibler divergence, $\tau$ is a temperature scaling parameter, $\bm\theta$ are the model parameters that are being learned for task $t$, and $\bm\theta_{t-1}$ are the model parameters \textit{learned} up to task $t-1$. Furthermore, in (\ref{eq:GKD_loss}) and (\ref{eq:TKD_loss}), we define the $c$-th component of the probability vectors $\bm p_{s}^\tau(\bm x, \bm\theta)\in\Delta^m$ and $\bm p_{1:s}^\tau(\bm x,  \bm\theta)\in\Delta^{C_s}$ as
\begin{align}
    p_{s,c}^{\tau}(\bm x, \bm \theta )=&\frac{e^{z_{sc}(\bm x,\bm \theta)/\tau}}{\sum_{k\in\mathcal{C}_s}e^{z_{sk}(\bm x,\bm \theta)/\tau}}\ \ \text{and}\nonumber\\
    p_{1:s,c}^{\tau}(\bm x, \bm \theta )=&\frac{e^{z_{sc}(\bm x,\bm \theta)/\tau}}{\sum_{k\in\mathcal{C}_{1:s}}e^{z_{sk}(\bm x,\bm \theta)/\tau}}, \nonumber
    \vspace{-.2in}
\end{align}
respectively. Namely, in words, $\bm p_{s}^\tau(\bm x, \bm\theta)$ is the probability vector obtained by \textit{only} using the output scores for task $s$ when computing the softmax probability, and $\bm p_{1:s}^\tau(\bm x, \bm\theta)$ is the probability vector obtained by using all the output scores for tasks $1\sim s$ when computing the softmax probability. Thus, minimizing (\ref{eq:GKD_loss}) or (\ref{eq:TKD_loss}) will both resulting in regularizing with the past model $\bm\theta_{t-1}$, but (\ref{eq:GKD_loss}) uses the global softmax probability across all past tasks, $\bm p^\tau_{1:t-1}(\bm x, \bm \theta_{t-1})$, while (\ref{eq:TKD_loss}) uses the task-wise softmax probabilities, $\{\bm p_s^\tau(\bm x, \bm \theta))\}_{s=1}^{t-1}$, obtained separately for each task.
In recent CIL baselines, (\ref{eq:GKD_loss}) is used in \cite{(LargeScaleIL)Wu2019,(UnlabeledIL)Lee2019,(WA)Zhao2020}, and (\ref{eq:TKD_loss}) is used in \cite{(LwF)LiHoiem16,(End2EndIL)Castro2018}. The difference between (\ref{eq:GKD_loss}) and (\ref{eq:TKD_loss}) is illustrated in Figure \ref{fig:TKD_GKD_illustration}.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=1.0\columnwidth]{figures/tkdgkd9.pdf}
    \vspace{-.2in}
    \caption{Illustration of $\mathcal{{L}_{\text{GKD, t}}}(\bm x, \bm \theta)$ (left) and $\mathcal{{L}_{\text{TKD, t}}}(\bm x, \bm \theta)$ (right)}
    \label{fig:TKD_GKD_illustration}
    \vspace{-.2in}
\end{figure}

\section{Motivation}\label{sec:motivation}

As mentioned in the Introduction, several previous work  \cite{(End2EndIL)Castro2018,(UnlabeledIL)Lee2019,(UnifiedIL)Hou2019,(LargeScaleIL)Wu2019,(IL2M)Belouadah, (WA)Zhao2020, (scail)belouadah2020} identified that the major challenge of the exemplar-memory based CIL is to resolve the classification score bias that the model suffers from. Here, we give a more detailed argument and convincing example on the root cause of such score bias and motivate our SS-IL.

\subsection{Bias caused by ordinary cross-entropy}

The ordinary cross-entropy loss for learning task $t$ used by the typical CIL methods can be expressed as
\begin{eqnarray}
    \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta) = \mathcal{D}_{KL}(\bm y_{1:t} \| \bm p_{1:t}(\bm x, \bm \theta)),

    \label{eq:ce_loss}
    \vspace{-.2in}
\end{eqnarray}
in which $\bm y_{1:t}$ is a one-hot vector in $\mathbb{R}^{C_t}$ that has value one at the $y$-th coordinate, and  $\bm p_{1:t}(\bm x, \bm \theta)$ is $\bm p^{\tau}_{1:t}(\bm x, \bm \theta)$ with $\tau = 1$.

Now, in order to systematically analyze the root cause of the prediction bias commonly present in typical CIL methods, we carried out an experiment with a simple CIL method that uses the following loss
\begin{eqnarray}
    \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta)+\mathcal{L}_{\text{GKD}, t}(\bm x, \bm \theta)\label{eq:simple_gkd}
\end{eqnarray}
with $(\bm{x},y)\in\mathcal{D}_t\cup \mathcal{M}$ for learning task $t$. Namely, it learns the task $t$ with the cross-entropy loss while trying to preserve past knowledge by $\mathcal{L}_{\text{GKD}}$. As shown in Figure \ref{fig:Bias_motivation}, we experimented with the ImageNet dataset with $m=100$ and $|\mathcal{M}|=10k$, hence with total 10 tasks.

The left plot in Figure \ref{fig:Bias_motivation} shows the confusion matrix at the task level after learning Task 10. It clearly shows the common prediction bias; namely, most of the prediction for past tasks are overly biased toward the most recent task. We argue that the root cause of this bias is in the well-known gradient for the softmax classifier:
\begin{eqnarray}
    \frac{\partial \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta)}{\partial z_{tc}} = p_{1:t,c}(\bm x,\bm\theta) - \mathds{1}_{\{c=y\}},\label{eq:ce_grad}
        \vspace{-.2in}
\end{eqnarray}
in which $\mathds{1}_{\{c=y\}}$ is the indicator for $c=y$. Note that since (\ref{eq:ce_grad}) is always positive for $c\neq y$, we can easily observe that when the model is being updated with data in $\mathcal{D}_t\cup \mathcal{M}$,
the classification scores for the old classes will continue to decrease during the gradient descent steps done for the abundant samples for the new classes in $\mathcal{D}_t$. Thus,
we believe that this imbalanced gradient descent steps for the classification scores of the old classes makes the significant score bias toward the new classes due to above point. The toy illustration of gradient descent steps is illustrated in Figure \ref{fig:SS_motivation}.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=1.0\columnwidth]{figures/Confusion_matrix.pdf
    \vspace{-.0in}
    \caption{Left: the confusion matrix based on the predictions of CIL model for test data. Right: the task ratio of Top-1 predictions made by $\bm\theta_{t-1}$ on $\mathcal{D}_t$. Note that the dashed region in the right plot indicates the task ratio of latest old classes, and it represents the bias on logits used for $\mathcal{L}_{\text{GKD}}$. All the results are on ILSVRC dataset with $m=100$ and $|\mathcal{M}|=10k$}\label{fig:Bias_motivation}
    \vspace{-.15in}
\end{figure}

\subsection{Bias preserved by GKD}\label{subsec:gkd}

Now, as mentioned above, several previous work use GKD for the purpose of preserving the knowledge learned from past tasks. However, when the gradient from the cross-entropy loss causes a significant bias as mentioned in the previous section, we argue that using GKD would preserve such bias in the older model and even could hurt the performance. That is,
in $\mathcal{L}_{\text{GKD}}$ defined in (\ref{eq:GKD_loss}), $\bm p_{1:t-1}^\tau(\bm x,\bm \theta_{t-1})$ is the \textit{soft} target computed from the \textit{old} model $\bm\theta_{t-1}$, that is used for knowledge distillation. Now, Figure \ref{fig:Bias_motivation} (right) suggests that this soft target can be heavily skewed due to the bias caused by the cross-entropy learning. Namely, the figure shows the ratio of the \textit{tasks} among $\{1,\ldots,t-1\}$, predicted by the \textit{old} model $\bm\theta_{t-1}$ when the \textit{new} task data points $\bm x$'s from $\mathcal{D}_{t}$ was given as input, for each new task $t$ (horizontal axis). We can observe that the predictions are overwhelmingly biased toward the most recent old task (\textit{i.e.}, task $t-1$), which is due to the bias generated during learning task $t-1$ with the cross-entropy loss. This suggests that the soft target $\bm p_{1:t-1}^\tau(\bm x,\bm \theta_{t-1})$ also would be heavily skewed toward the most recent old task (task $t-1$), hence, when it is used in GKD loss as (\ref{eq:GKD_loss}), it will preserve such bias and could highly penalize the output probabilities for the \textit{older} tasks. Hence, it could make the bias, or the \textit{forgetting} of older tasks, more severe. In Section \ref{subsec:kd}, we indeed show that when GKD is naively used, it can even hurt the performance of simple fine-tuning that only uses $\mathcal{L}_{\text{CE},t}$.

Above two observations suggest that main reason for the prediction bias could be due to computing the softmax probability by combining the old and new tasks \textit{altogether}. Motivated by this, we proposed Separated Softmax for Incremental Learning (SS-IL) in the next section.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=0.8\columnwidth]{figures/motivation_ss.pdf}
    \vspace{-.0in}
    \caption{The toy illustration of gradient descent steps for $m=2$ and $T=2$ on imbalanced $\mathcal{D}_2\cup\mathcal{M}$. As seen above, scores for class $c\in\mathcal{C}_{1}$ continue to decrease due to the imbalanced gradient descent steps.}
    \label{fig:SS_motivation}
    \vspace{-.15in}
\end{figure}

\section{Main Method}\label{sec:method}

Our SS-IL consists of three components, all motivated from the intuition built from the previous section: (1) separated softmax (SS) output layer, (2) ratio-preserving (RP) mini-batch selection, and (3) the task-wise KD (TKD).
Before concretely presenting them, we introduce some additional notations. For the incremental task $t$, we denote the classes of the old tasks by $\mathcal{P}_t=\mathcal{C}_{1:t-1}$ and the classes of the new task $t$ by $\mathcal{N}_t=\mathcal{C}_t$.

\vspace{.03in}

\noindent{\textit{(1) Separated softmax (SS) layer:}  } For $(\bm x, \bm y)\in\mathcal{D}_t\cup \mathcal{M}$, we define a separate softmax output layer by modifying the cross-entropy loss function as
\begin{eqnarray}
    \mathcal{L}_{\text{CE-SS},t}((\bm x, y),\bm\theta) =
    &\mathcal{L}_{\text{CE},t-1}((\bm x,y),\bm\theta)\cdot \mathds{1}\{y\in \mathcal{P}_t\} + \nonumber\\
    &\mathcal{D}_{KL}(\bm y_{t} \| \bm p_{t}(\bm x, \bm \theta))\cdot \mathds{1}\{y\in \mathcal{N}_t\}, \label{eq:ss}

\end{eqnarray}
in which $\bm y_{t}$ stands for the one-hot vector in $\mathbb{R}^{\mathcal{N}_t}$ and $\bm p_{t}(\bm x, \bm \theta))$ is $\bm p_{t}^\tau(\bm x,  \bm\theta)$ with $\tau=1$. Namely, in words,
depending on whether $(\bm x, y)\in\mathcal{M}$  or $(\bm x, y)\in\mathcal{D}_t$, the softmax probability is computed separately by only using the output scores for $\mathcal{P}_t$ or $\mathcal{N}_t$, respectively, and the cross-entropy loss is computed separately as well.
While (\ref{eq:ss}) is a simple modification of the ordinary cross-entropy (\ref{eq:ce_loss}), we can now observe that
$\frac{\partial \mathcal{L}_{\text{CE-SS}}}{\partial z_{tc}}=0$ for $c\in\mathcal{P}_t$ when $(\bm x, y)\in\mathcal{D}_t$. Therefore, the gradient from the new class samples in $\mathcal{N}_t$ will not have overly penalizing effect in the classification scores for the old classes in $\mathcal{P}_t$.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/SSIL.pdf}

    \caption{Illustration of SS-IL}\label{fig:SSIL}
    \vspace{-.1in}
\end{figure}

\vspace{.03in}
\noindent \textit{(2) Ratio-preserving (RP) mini-batch: } Another subtle change we implemented is the ratio-preserving (RP) mini-batches for the SGD updates of the model. Note that when random mini-batches are sampled from $\mathcal{D}_t\cup\mathcal{M}$ for SGD, the severe imbalance between new classes and old classes carries over to the mini-batches as well. Such imbalance in mini-batches would significantly downplay the updates of the model for the old classes in our SS layer, since the gradient from the first part of (\ref{eq:ss}) will be generated scarcely. From this observation and to assure the main role of exemplars in $\mathcal{M}$, \ie, to fine-tune the representations and decision boundaries of old classes in response to learning the new classes in $\mathcal{D}_t$, we always generated the mini-batches such that the minimum ratio of the samples from $\mathcal{M}$ can be guaranteed.
Motivated by Experience Replay \cite{(ER)Chaudhry2019} method, this can be simply implemented by concatenating the fixed-sized random samples from $\mathcal{M}$, denoted by \textit{replay batch} ($B_{\mathcal{M}}$) in the later sections, with the random samples from $\mathcal{D}_t$ ($B_{\mathcal{D}_t}$). In our experiments, we set the ratio of new class samples over the old class samples to $2\sim8$ to set the balance between learning new classes and preserving the knowledge learned for old classes.

\vspace{.03in}

\noindent{\textit{(3) Task-wise KD:}  }  With the SS layer, we can easily see that it is natural to use TKD (\ref{eq:TKD_loss}), which also uses the separated softmax for each task, for the knowledge distillation. That is, in TKD, since the soft targets, $\{\bm p_s^\tau(\bm x, \bm \theta))\}_{s=1}^{t-1}$, are computed only within each task, TKD will not get affected by the task-wise bias that may present in the old model $\bm\theta_{t-1}$, as opposed to the GKD shown in Section \ref{subsec:gkd}. Hence, we can expect that TKD is particularly well-suited for the SS layer, which will be shown in our experimental results.

\vspace{.03in}
\noindent \textbf{Final loss function for SS-IL: }
By combining $\mathcal{L}_{\text{CE-SS},t}$ in (\ref{eq:ss}) and $\mathcal{L}_{\text{TKD},t}$ in (\ref{eq:TKD_loss}), the overall loss for SS-IL becomes:
\begin{align}
    \mathcal{L}_{\text{SS-IL},t}((\bm x, y),\bm\theta) =  \mathcal{L}_{\text{CE-SS},t}((\bm x,  y),\bm\theta) + \mathcal{L}_{\text{TKD},t}(\bm x,\bm\theta),\nonumber \label{eq:SSIL}
    \vspace{-0.3in}
\end{align}
and the mini-batch SGD to minimize the loss is done with RP mini-batches.
Figure \ref{fig:SSIL} and Algorithm \ref{alg:Algorithm} illustrates and summarizes our method, respectively. We show in our experimental results that our SS-IL can significantly correct the prediction bias, without \textit{any} score post-processing as is done in many other baselines, as achieve the state-of-the-art accuracy for various CIL benchmark datasets.

\begin{algorithm}[t]
\caption{Separated Softmax for Incremental Learning (SS-IL)}
\begin{algorithmic}
\REQUIRE $\{\mathcal{D}_t\}_{t=1}^T$: Training dataset
\REQUIRE $\mathcal{M}\leftarrow \{\}$: Memory buffer
\REQUIRE $E$: The number of epochs per task.
\REQUIRE $N_{\mathcal{D}_t}, N_{\mathcal{M}}$: Training \& replay batch sizes
\REQUIRE $\alpha$: Learning rate
\REQUIRE $\bm\theta$ : Network parameters
\STATE \textcolor{blue}{\# Start class incremental learning}
\STATE Randomly initialize $\bm\theta$
\FOR{$t=1,...,T$}
\FOR{$e=1,...,E$}
\STATE \textcolor{blue}{\# Sample a mini-batch of size $N_{\mathcal{D}_t}$}
\FOR{$B_{\mathcal{D}_t} \sim \mathcal{D}_t$}
\STATE \textcolor{blue}{\# Sample a mini-batch of size $N_{\mathcal{M}}$}
\STATE $B_{\mathcal{M}} \sim \mathcal{M}$
\STATE $\mathcal{L}_t(\bm \theta) = \sum_{(\bm x, \bm y)\in B_{\mathcal{D}_t} \cup B_{\mathcal{M}}} \mathcal{L}_{\text{SS-IL},t}((\bm x, \bm y), \bm \theta)$
\STATE $\bm\theta \leftarrow \bm\theta - \frac{\alpha}{N_{\mathcal{D}_t}+N_{\mathcal{M}}} \cdot \nabla_{\bm\theta}\mathcal{L}_t(\bm \theta)$
\ENDFOR
\ENDFOR
\STATE $\mathcal{M}\leftarrow$ UpdateMemory($\mathcal{D}_t$, $\mathcal{M}$)
\ENDFOR
\end{algorithmic}
\label{alg:Algorithm}
\end{algorithm}

\section{Introduction}

Incremental or continual learning, in which the agent continues to learn with incremental arrival of new training data, is one of the grand challenges in artificial intelligence and machine learning.
Such setting, which does not assume the full availability of old training data, is recently gaining more attention particularly from the real-world application perspective. The reason is because storing all the training data, which can easily become large-scale, in one batch often becomes unrealistic for memory- and computation-constrained applications, such as mobile phones or robots, hence the continuous yet effective update of the learning agent without accessing the full data received so far is indispensable.

A viable candidate for such agent is the end-to-end learning based deep neural network (DNN) models. Following the recent success of DNN in many different applications \cite{HinLecBen15,(AutonomousDriving)Al-Qizwini,(Medical)Caruana}, the DNN-based incremental learning methods have been also actively pursued in recent years. Although they achieved some promising results, they also possess a critical limitation: the \emph{catastrophic forgetting}, which refers to the problem that the generalization performance on the old data severely degrades after a naive fine-tuning of the model with the new data.

In this paper, we focus on the DNN-based \emph{class} incremental learning (CIL) problem, which we refer to learning a classifier to classify \emph{new} object classes from every incremental training data and testing the classifier on all the classes learned so far.
Among several different proposed approaches, the exemplar-memory based approaches \cite{(icarl)rebuffi17,(End2EndIL)Castro2018,(LargeScaleIL)Wu2019,(WA)Zhao2020,(IL2M)Belouadah,(scail)belouadah2020}, which allows to store small amount of training data from old classes in a separate memory,
has attained promising results. It has been shown that using the small exemplar memory plays an important role in mitigating the catastrophic forgetting, and allowing such small size of memory while learning is also tolerable in practical scenarios as well.

The main challenge of using the exemplar-memory is to resolve the severe data imbalance issue between the training data points for the new classes and those for the old classes in the exemplar-memory. That is, the naive fine-tuning with such imbalanced data may still heavily skew the predictions toward the newly learned classes, hence, the accuracy for the old classes would dramatically drop,
again resulting in significant forgetting.
Recent state-of-the-arts \cite{(End2EndIL)Castro2018,(LargeScaleIL)Wu2019,(WA)Zhao2020,(IL2M)Belouadah,(scail)belouadah2020} found that significantly higher classification scores for the newly learned classes is the main reason for such prediction bias, thus, they proposed to correct the score bias by some additional post-processing, \textit{e.g.}, score re-scaling or balanced fine-tuning, after learning the model.

While above mentioned methods were effective to some extent in terms of improving the accuracy, we argue that they lack systematic analysis on the root cause of such bias and some component of their schemes, \textit{e.g.}, knowledge distillation (KD) \cite{(Distill)HintonOriolDean15} , was naively used without proper justification \cite{(LargeScaleIL)Wu2019,(UnlabeledIL)Lee2019,(WA)Zhao2020,(ConfCalIncre)Kang2020}. To that regard, in this paper, we first analyze the root cause of such prediction score bias, then propose a method that mitigates the cause in a sensible way. Namely, we argue that the bias is injected by the fact that the softmax probability used in the usual cross-entropy loss is always computed by combining the output scores of \textit{all} classes, which forces the heavy penalization of the output probabilities for the old classes due to data imbalance. Furthermore, we show that a naive use of the General KD (GKD) method, which also combines the output scores of \textit{all} old classes to compute the soft target, may preserve the bias and even hurt the accuracy, if the prediction bias is already present in the model.

To resolve above issues, we propose Separated Softmax for Incremental Learning (SS-IL), which consists of three components. Firstly, we devise
\textit{separated softmax (SS)} output layer that mutually blocks the flow of the score gradients between the old and new classes, hence, mitigates the imbalanced penalization of the output probabilities for the old classes. Secondly, for the mini-batch stochastic gradient descent (SGD) steps, we utilize \textit{ratio-preserving (RP)} mini-batches that guarantee the minimum ratio of the samples from old classes steps such that a balance between learning the new classes and preserving the knowledge for old classes can be realized. Thirdly, we show the \textit{Task-wise KD (TKD)}, which also computes the soft-target for distillation in a task-separated manner, is particularly well-suited for our SS layer, since it attempts to preserve the task-wise knowledge without preserving the prediction bias that may present among tasks. In our extensive experimental results, we show that our SS-IL achieves very strong state-of-the-art accuracy on various different large-scale CIL benchmark datasets. We also stress that our method significantly mitigates the prediction score bias without any additional post-processing steps, unlike other recent baselines. We also present systematic ablation study on our method, justifying each of the three components in SS-IL.

\section{Related Work}
In this section, we summarize algorithms related to continual learning and class incremental learning (CIL). Here, the algorithms denoted by ``continual learning'' make an assumption that task information is available at the test time, and the algorithms denoted by ``class incremental learning'', the focus of this paper, does not make such assumption. Hence, the CIL deals with a more challenging setting than the continual learning algorithms mentioned below.

\noindent\textbf{Memory based continual learning}\ \ By utilizing the information in exemplar memory, \cite{(GEM)LopezRanzato17} solves the constrained optimization problem by using gradients for each task using exemplars. However, due to hard constraint on the gradient of each task, new tasks become harder to learn. In \cite{(A-GEM)Chaudhry2019}, they propose the relaxed version of constraint which tries to compute the average of the gradient, and solves a much simpler constrained optimization problem. In \cite{(ER)Chaudhry2019}, instead of computing gradients, they concatenate two batches which are drawn from exemplar and current task dataset.

\noindent\textbf{Generative replay based CIL}\ \
By generating the auxiliary data on previous tasks, \cite{(DGR)ShinLeeKimKim17} generate the previous task data using Generative Adversarial Network(GAN)\cite{(GAN)goodfellow2014} and consider the ``continual learning'' scenario.
So far, we introduce the ``continual learning'' algorithms. Form now on, we introduce ``class incremental learning'' algorithms.
In \cite{(Fearnet)kemker18}, they generate the features of old classes from a pre-trained model using stored class statistics, and apply it to the ``class incremental learning'' scenario. In different way, \cite{(CGANIL)Xiang2019} tries to generate the intermediate feature of pre-trained network using Conditional GAN\cite{(ConditionalGAN)Mirza2014}.

\noindent\textbf{KD based CIL and bias removal methods}\ \ The earliest of KD based methods, \cite{(LwF)LiHoiem16}, uses KD to keep the scores of previous tasks when learning new tasks. The developed version of \cite{(LwF)LiHoiem16}, iCaRL \cite{(icarl)rebuffi17}, which uses memory exemplar, preserves the feature using KD and then classifies the classes using the Nearest Mean of Exemplars (NME) classification. However, the methods using exemplar, such as iCaRL, have been shown experimentally that there is a bias in the final FC layer.

To tackle this problem, bias removal techniques \cite{(End2EndIL)Castro2018,(UnlabeledIL)Lee2019,(LargeScaleIL)Wu2019, (WA)Zhao2020, (ConfCalIncre)Kang2020, (IL2M)Belouadah, (scail)belouadah2020} are proposed. In \cite{(End2EndIL)Castro2018}, to remove the prediction bias, they proposed a balanced fine-tuning method, which is fine-tuning the network using a balanced dataset. Another balanced fine-tuning approach \cite{(UnlabeledIL)Lee2019} proposed a gradient scaling to remove the prediction bias. In \cite{(LargeScaleIL)Wu2019, (WA)Zhao2020,(ConfCalIncre)Kang2020}, after the training process, \cite{(LargeScaleIL)Wu2019} corrects the bias of scores using bias correction layer with validation set, \cite{(WA)Zhao2020} corrects the biased weights in the FC layer, and \cite{(ConfCalIncre)Kang2020} scales the scores of new classes in training time. The common property of these approaches is that all of them use KD to preserve the knowledge of the past tasks.
Unlike the above methods, \cite{(IL2M)Belouadah, (scail)belouadah2020} correct the score bias without any distillation methods. Using the statistics on each task, \cite{(IL2M)Belouadah} rectifies the output softmax probability and \cite{(scail)belouadah2020} scales the classifier weights.

\section{Datasets and evaluation protocol}

\noindent\textbf{ImageNet}:\ \
ILSVRC 2012 dataset consists of 1,000 classes, which has nearly 1,300 images per class. By following the benchmark protocol in \cite{(icarl)rebuffi17}, we arrange the classes of each dataset in a fixed random order. We experiment with varied total number of incremental tasks, $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ per task, and for the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$. When constructing exemplar-memory, we use Ringbuffer approach proposed in \cite{(ER)Chaudhry2019}, which simply samples random data from old classes.
We always maintain balanced number of exemplars across all the old classes. Thus, as the incremental task increases, we delete equal number of exemplars from the old classes and add exemplars for the newly learned classes. For the evaluation of CIL models, we use ILSVRC 2012 validation set for testing.

\noindent\textbf{Landmark-v2}:\ \
Google Landmark Dataset v2 consists of 203,094 classes, and each class has $1\sim10,247$ images. Since the dataset is highly imbalanced, we sample 1,000 and 10,000 classes in the order of largest number of samples per class. We denote Landmark-v2 dataset with 1,000 and 10,000 classes as Landmark-v2-1K and Landmark-v2-10K, respectively. After sampling the classes, we arrange the classes in a fixed random order. Similarly as in ImageNet, we vary the total number of incremental tasks as $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ in Landmark-v2-1K and $m=\{2000,1000,500\}$ in Landmark-v2-10K, respectively. For the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$ for Landmark-v2-1K and $|\mathcal{M}|=\{20k,40k,60k\}$ for Landmark-v2-10K, respectively. Same as in ImageNet, we use the Ringbuffer approach for constructing the exemplars.
For evaluation, we randomly select 50 and 10 images per each class in Landmark-v2-1K and Landmark-v2-10K that are not in the training set for testing.

\section{Implementation details}\label{sec:Implementation details}
All the baselines use the Resnet-18\cite{(Resnet)HEZhang15} architecture and are implemented using Pytorch framework\cite{(PyTorch)Paszke2017}. We always use the stochastic gradient descent (SGD) with weight decay 0.0001 and momentum 0.9. We planned to use WA\cite{(WA)Zhao2020} as one of our baselines for comparison. However we could not compare our method with it since it did not publish its official code and we failed to reproduce it. Including SS-IL and all the other baselines, the code implementations will be publicly available.

\noindent\textbf{iCaRL}:\ \
iCaRL is implemented based on \cite{(Revisiting)Javed2018}. We use multi-class cross entropy loss for both classification loss and KD loss instead of binary cross entropy. The number of epochs for training incremental task is 60. The learning rate starts at 0.1 and is divided by 5 at 20, 30, 40 and 50 epochs. The size of mini-batches is 128.

\noindent\textbf{FT and IL2M}:\ \
FT and IL2M are implemented based on \cite{(IL2M)Belouadah}. The first incremental task consists of 100 epochs with initial learning rate 0.1 and the learning rate is divided by 10 at 40 and 80 epochs. The rest incremental tasks consists of 25 epochs with initial learning rate lr=$\frac{0.1}{\text{t}}$, where t is the incremental task. The learning rate is divided by 10 at 10 and 20 epochs. The size of mini-batches is 128.

\noindent\textbf{EEIL}:\ \
EEIL is implemented based on \cite{(End2EndIL)Castro2018}. Each incremental step consists of 40 epochs for training and an additional 30 epochs for balanced fine-tuning. For the first 40 epochs, the learning rate starts at 0.1 and is divided by 10 at 10, 20 and 30 epochs. For balanced fine tuning, the learning rate starts at 0.01 and is divided by 10 at 10 and 20 epochs. The size of mini-batches is 128.

\noindent\textbf{BiC}:\ \
BiC is implemented based on \cite{(LargeScaleIL)Wu2019}. The number of epochs for training incremental task is 100. The learning rate starts at 0.1 and is divided by 10 at 30, 60 and 80 epochs. In addition, the number of epochs for training bias correction layer at incremental task is 200. The learning rate for bias correction layer starts at 0.001 and is divided by 10 at 60, 120, 180 epochs. The size of mini-batches is 256.

\section{Additional results}
Figure \ref{fig:results_top_1} and \ref{fig:results_top_5} show the detailed results used to generate (Table 1, manuscript). Overall, SS-IL achieves much higher accuracy than other baselines for most of scenarios. Especially, for the smallest $|\mathcal{M}|$, SS-IL significantly outperforms other baselines, and the accuracy is not much smaller than for the largest $|\mathcal{M}|$, which means it is sufficient for SS-IL to use less memory in CIL.

\begin{figure}[ht]
    \centering
    \subfigure[$T=10$ and $|\mathcal{M}|=5k$(1K), $20k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_20000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=20k$(1K), $60k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_60000_1000.pdf}}
    \subfigure[$T=20$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_500.pdf}}
    \subfigure[$T=5$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_2000.pdf}}
    \caption{Top-1 accuracy on ImageNet-1K, Landmark-1K and, Landmark-10K.}\label{fig:results_top_1}
\end{figure}

\begin{figure}[ht]
    \centering
    \subfigure[$T=10$ and $|\mathcal{M}|=5k$(1K), $20k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_20000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=20k$(1K), $60k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_60000_1000.pdf}}
    \subfigure[$T=20$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_500.pdf}}
    \subfigure[$T=5$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_2000.pdf}}
    \caption{Top-5 accuracy on ImageNet-1K, Landmark-1K and, Landmark-10K.}\label{fig:results_top_5}
\end{figure}

\section{Analyses on KD}

\subsection{Training Details}\label{subsec:training_details}

We use the Resnet-18\cite{(Resnet)HEZhang15} architecture, implemented on Pytorch\cite{(PyTorch)Paszke2017} framework. For training the neural network with TKD and GKD, we use the stochastic gradient descent (SGD) with learning rate 0.1, weight decay 0.0001, and momentum 0.9. The batch size used for $\mathcal{D}_t$ is 128. The number of epochs for training incremental task is 100, and the learning rate is divided by 10 at epochs 30, 60, 90. For temperature scaling parameter $\tau$ in (Eq.(2) and Eq.(3), manuscript), we set $\tau=2$. The training schedule for FT is same as mentioned in \ref{sec:Implementation details}. After learning the model, we correct the bias by some additional post processing: Balanced Fine-Tuning (BFT) and Score correction, respectively. In case of SS+RP, we carry out the bias correction step on the model without any additional post-processing steps.

\noindent\textbf{Balanced Fine-Tuning}:\ \
BFT is implemented based on \cite{(End2EndIL)Castro2018,(IL2M)Belouadah}. After training incremental task $t$, we perform additional 30 epochs for BFT. Learning rate is set to $\frac{0.001}{t}$ which is proposed in \cite{(IL2M)Belouadah} when applying to FT, GKD, and TKD.

\noindent\textbf{Score correction}:\ \
Score correction is implemented based on \cite{(LargeScaleIL)Wu2019}. The number of epochs for training bias correction layer at incremental task $t$ is 200 for TKD, GKD and 50 for FT. The learning rate starts at 0.001 and is divided by 10 at 60, 120, 180 epochs for TKD, GKD and at 20, 40 epochs for FT. The size of mini-batches is 128.

\noindent\textbf{SS+RP}:\ \
SS+RP is implemented based on (Eq.(7), manuscript). The size of RP mini-batches is 32, and all the other training schemes are same as specified in manuscript.

\subsection{Existence of bias on $\mathcal{L}_{\text{GKD}}$}\label{subsec:bias_existence}

\begin{figure}[h]
    \centering
    \includegraphics[width=1\columnwidth]{figures/global_to_local_supplementary.pdf}
    \vspace{-.1in}
    \caption{Top-1 accuracy for $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$ on two different bias correction schemes at each task $t$ ($t = 3 \sim 10$)}
    \label{fig:global_to_local_supplementary}
    \vspace{-.12in}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/prediction_map_supplementary.pdf}
    \vspace{-.15in}
    \caption{Task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$ at each task $t$ ($t = 3 \sim 10$)}\label{fig:prediction_map_supplementary}
    \vspace{-.15in}
\end{figure}

Figure \ref{fig:global_to_local_supplementary} and Figure \ref{fig:prediction_map_supplementary} show the detailed versions of (Figure 7 and Figure 8, manuscript), respectively. For both BFT and SS+RP, the results are presented at task $t = 3 \sim 10$.
Observation starts from $t = 3$ since training at least 2 tasks is needed for the bias on $\mathcal{L}_{\text{GKD}}$ to occur.

In Figure \ref{fig:global_to_local_supplementary}, for the same $\bm \theta_{t-1}$, the Top-1 accuracy of $\bm \theta_{\text{TKD},t}$ is higher than that of $\bm \theta_{\text{GKD},t}$ at every task $t$ in the range of $t = 3 \sim 10$. We assume that the main reason for the accuracy difference is due to the bias preservation caused by the score bias of $\bm \theta_{t-1}$ on $\bm x \in \mathcal{D}_t$. As shown in Figure \ref{fig:prediction_map_supplementary}, the ratio of the prediction on the latest old task is higher than the ratio of any other tasks.
Especially, for the case of BFT, this significantly increases as the incremental task $t$ grows.
It leads to the result that the accuracy difference between $\bm \theta_{\text{TKD},t}$ and $\bm \theta_{\text{GKD},t}$ for BFT is large in comparison with that for SS+RP.

\clearpage

{\small
\bibliographystyle{ieee_fullname}

\section{Conclusion}
In this paper, we propose a new method, SS-IL, that addresses the score bias and the bias preserving property of GKD. Based on the systematic analysis on gradients of ordinary cross entropy, we find the root cause of the score bias is in softmax. By using separated softmax (SS) and ratio preserving (RP) mini-batch, we solve this issue without any score post-processing. Furthermore, we also find that such bias is rather preserved by GKD and TKD will not get affected by the task-wise bias. The experiment results show that our SS and RP mini-batch with TKD achieve the highest performance compared to the other state-of-the-art methods and balanced prediction among tasks. In analyses on KD, we observe the existence of bias on GKD and we show TKD always outperforms GKD, which means TKD is a better choice for preserving the knowledge.

\section{Preliminaries}\label{sec:notation}
\subsection{Notations and problem setting}

In CIL, we assume every incrementally-arrived training data, which is often called as the incremental \textit{task}, consists of data for \textit{new} $m$ classes that have not been learned before.
More formally, the training data for the incremental task $t$ is denoted by $\mathcal{D}_t=\{(\bm x_{t}^{(i)},  y_{t}^{(i)})\}_{i=1}^{n_{t}}$, in which $\bm x_{t}^{(i)}$,
 $y_{t}^{(i)}$, and $n_{t}$ denote input data for task $t$, the corresponding (integer-valued) target label, and the number of training samples for the corresponding task, respectively. The total number of classes up to task $t$ is denoted by $C_t=mt$, which leads to the labeling $y_{t}^{(i)}\in\{C_{t-1}+1,\ldots,C_{t}\}\triangleq \mathcal{C}_t$.
During learning each incremental task, we assume a separate exemplar-memory $\mathcal{M}$  is allocated to store exemplar data for old classes. Namely, when learning the incremental task $t$, we store $\lfloor\frac{|\mathcal{M}|}{C_{t-1}}\rfloor$ data points from each class that are learnt until the incremental task $t-1$. Thus, as the incremental task grows, the number of exemplar data points stored for each class decreases linearly with $t$ and we assume $|\mathcal{M}|\ll n_{t}$. The total number of incremental tasks is denoted by $T$.

Our classification model consists of a feature extractor, which has the deep convolutional neural network (CNN) architecture, and the classification layer, which is the final fully-connected (FC) layer with softmax output. We denote $\bm \theta$ as the parameters for our classification model. At incremental task $t$, the parameters of the model, $\bm \theta_t$, are learned using data points in $\mathcal{D}_t\cup \mathcal{M}$. After learning, the class prediction for a given sample $\bm x_{\text{test}}$ is obtained by
\begin{eqnarray}

 \hat{y}_{\text{test}}=\arg\max_{y\in\mathcal{C}_{1:t}}z_{ty}(\bm x_{\text{test}}, \bm \theta_t),\label{eq:test}
\end{eqnarray}
in which $z_{ty}(\bm x_{\text{test}},\bm\theta_t)$ is the output score (before softmax) of the model $\bm \theta_t$ for class $y\in \mathcal{C}_{1:t}\triangleq\{1,\ldots,C_t\}$. Then, we denote the task which $\hat{y}_{\text{test}}$ belongs as $\hat{t}_{\text{test}}=i \ \ \text{for} \ \ \hat{y}_{\text{test}}\in\mathcal{C}_i$.
Namely, at test time, the final FC layers are consolidated and the prediction among all classes in $\mathcal{C}_{1:t}$ is made as if by an ordinary multi-class classifier.

\subsection{Knowledge distillation}

In several recent CIL methods, knowledge distillation (KD) is often used to maintain the knowledge learned from the past tasks. There are two variations of KD, General KD (GKD) and Task-wise KD (TKD), and the loss function defined for each method for learning task $t$ is as follows: for an input data $\bm x\in\mathcal{D}_t\cup\mathcal{M}$,
\begin{align}
    &\mathcal{L}_{\text{GKD}, t}(\bm x, \bm \theta) \triangleq \mathcal{D}_{KL}(\bm p^\tau_{1:t-1}(\bm x, \bm \theta_{t-1}) \|\bm p^\tau_{1:t-1}(\bm x, \bm \theta)) \label{eq:GKD_loss} \\
    &\mathcal{L}_{\text{TKD}, t}(\bm x, \bm \theta) \triangleq \sum_{s=1}^{t-1}\mathcal{D}_{KL}(\bm p_s^\tau(\bm x, \bm \theta_{t-1}) \|\bm p_s^\tau(\bm x, \bm \theta)) \label{eq:TKD_loss},
    \vspace{-.2in}
\end{align}
in which $\mathcal{D}_{KL}(\cdot\|\cdot)$ is the Kullback-Leibler divergence, $\tau$ is a temperature scaling parameter, $\bm\theta$ are the model parameters that are being learned for task $t$, and $\bm\theta_{t-1}$ are the model parameters \textit{learned} up to task $t-1$. Furthermore, in (\ref{eq:GKD_loss}) and (\ref{eq:TKD_loss}), we define the $c$-th component of the probability vectors $\bm p_{s}^\tau(\bm x, \bm\theta)\in\Delta^m$ and $\bm p_{1:s}^\tau(\bm x,  \bm\theta)\in\Delta^{C_s}$ as
\begin{align}
    p_{s,c}^{\tau}(\bm x, \bm \theta )=&\frac{e^{z_{sc}(\bm x,\bm \theta)/\tau}}{\sum_{k\in\mathcal{C}_s}e^{z_{sk}(\bm x,\bm \theta)/\tau}}\ \ \text{and}\nonumber\\
    p_{1:s,c}^{\tau}(\bm x, \bm \theta )=&\frac{e^{z_{sc}(\bm x,\bm \theta)/\tau}}{\sum_{k\in\mathcal{C}_{1:s}}e^{z_{sk}(\bm x,\bm \theta)/\tau}}, \nonumber
    \vspace{-.2in}
\end{align}
respectively. Namely, in words, $\bm p_{s}^\tau(\bm x, \bm\theta)$ is the probability vector obtained by \textit{only} using the output scores for task $s$ when computing the softmax probability, and $\bm p_{1:s}^\tau(\bm x, \bm\theta)$ is the probability vector obtained by using all the output scores for tasks $1\sim s$ when computing the softmax probability. Thus, minimizing (\ref{eq:GKD_loss}) or (\ref{eq:TKD_loss}) will both resulting in regularizing with the past model $\bm\theta_{t-1}$, but (\ref{eq:GKD_loss}) uses the global softmax probability across all past tasks, $\bm p^\tau_{1:t-1}(\bm x, \bm \theta_{t-1})$, while (\ref{eq:TKD_loss}) uses the task-wise softmax probabilities, $\{\bm p_s^\tau(\bm x, \bm \theta))\}_{s=1}^{t-1}$, obtained separately for each task.
In recent CIL baselines, (\ref{eq:GKD_loss}) is used in \cite{(LargeScaleIL)Wu2019,(UnlabeledIL)Lee2019,(WA)Zhao2020}, and (\ref{eq:TKD_loss}) is used in \cite{(LwF)LiHoiem16,(End2EndIL)Castro2018}. The difference between (\ref{eq:GKD_loss}) and (\ref{eq:TKD_loss}) is illustrated in Figure \ref{fig:TKD_GKD_illustration}.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=1.0\columnwidth]{figures/tkdgkd9.pdf}
    \vspace{-.2in}
    \caption{Illustration of $\mathcal{{L}_{\text{GKD, t}}}(\bm x, \bm \theta)$ (left) and $\mathcal{{L}_{\text{TKD, t}}}(\bm x, \bm \theta)$ (right)}
    \label{fig:TKD_GKD_illustration}
    \vspace{-.2in}
\end{figure}

\section{Motivation}\label{sec:motivation}

As mentioned in the Introduction, several previous work  \cite{(End2EndIL)Castro2018,(UnlabeledIL)Lee2019,(UnifiedIL)Hou2019,(LargeScaleIL)Wu2019,(IL2M)Belouadah, (WA)Zhao2020, (scail)belouadah2020} identified that the major challenge of the exemplar-memory based CIL is to resolve the classification score bias that the model suffers from. Here, we give a more detailed argument and convincing example on the root cause of such score bias and motivate our SS-IL.

\subsection{Bias caused by ordinary cross-entropy}

The ordinary cross-entropy loss for learning task $t$ used by the typical CIL methods can be expressed as
\begin{eqnarray}
    \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta) = \mathcal{D}_{KL}(\bm y_{1:t} \| \bm p_{1:t}(\bm x, \bm \theta)),

    \label{eq:ce_loss}
    \vspace{-.2in}
\end{eqnarray}
in which $\bm y_{1:t}$ is a one-hot vector in $\mathbb{R}^{C_t}$ that has value one at the $y$-th coordinate, and  $\bm p_{1:t}(\bm x, \bm \theta)$ is $\bm p^{\tau}_{1:t}(\bm x, \bm \theta)$ with $\tau = 1$.

Now, in order to systematically analyze the root cause of the prediction bias commonly present in typical CIL methods, we carried out an experiment with a simple CIL method that uses the following loss
\begin{eqnarray}
    \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta)+\mathcal{L}_{\text{GKD}, t}(\bm x, \bm \theta)\label{eq:simple_gkd}
\end{eqnarray}
with $(\bm{x},y)\in\mathcal{D}_t\cup \mathcal{M}$ for learning task $t$. Namely, it learns the task $t$ with the cross-entropy loss while trying to preserve past knowledge by $\mathcal{L}_{\text{GKD}}$. As shown in Figure \ref{fig:Bias_motivation}, we experimented with the ImageNet dataset with $m=100$ and $|\mathcal{M}|=10k$, hence with total 10 tasks.

The left plot in Figure \ref{fig:Bias_motivation} shows the confusion matrix at the task level after learning Task 10. It clearly shows the common prediction bias; namely, most of the prediction for past tasks are overly biased toward the most recent task. We argue that the root cause of this bias is in the well-known gradient for the softmax classifier:
\begin{eqnarray}
    \frac{\partial \mathcal{L}_{\text{CE},t}((\bm x,y), \bm \theta)}{\partial z_{tc}} = p_{1:t,c}(\bm x,\bm\theta) - \mathds{1}_{\{c=y\}},\label{eq:ce_grad}
        \vspace{-.2in}
\end{eqnarray}
in which $\mathds{1}_{\{c=y\}}$ is the indicator for $c=y$. Note that since (\ref{eq:ce_grad}) is always positive for $c\neq y$, we can easily observe that when the model is being updated with data in $\mathcal{D}_t\cup \mathcal{M}$,
the classification scores for the old classes will continue to decrease during the gradient descent steps done for the abundant samples for the new classes in $\mathcal{D}_t$. Thus,
we believe that this imbalanced gradient descent steps for the classification scores of the old classes makes the significant score bias toward the new classes due to above point. The toy illustration of gradient descent steps is illustrated in Figure \ref{fig:SS_motivation}.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=1.0\columnwidth]{figures/Confusion_matrix.pdf
    \vspace{-.0in}
    \caption{Left: the confusion matrix based on the predictions of CIL model for test data. Right: the task ratio of Top-1 predictions made by $\bm\theta_{t-1}$ on $\mathcal{D}_t$. Note that the dashed region in the right plot indicates the task ratio of latest old classes, and it represents the bias on logits used for $\mathcal{L}_{\text{GKD}}$. All the results are on ILSVRC dataset with $m=100$ and $|\mathcal{M}|=10k$}\label{fig:Bias_motivation}
    \vspace{-.15in}
\end{figure}

\subsection{Bias preserved by GKD}\label{subsec:gkd}

Now, as mentioned above, several previous work use GKD for the purpose of preserving the knowledge learned from past tasks. However, when the gradient from the cross-entropy loss causes a significant bias as mentioned in the previous section, we argue that using GKD would preserve such bias in the older model and even could hurt the performance. That is,
in $\mathcal{L}_{\text{GKD}}$ defined in (\ref{eq:GKD_loss}), $\bm p_{1:t-1}^\tau(\bm x,\bm \theta_{t-1})$ is the \textit{soft} target computed from the \textit{old} model $\bm\theta_{t-1}$, that is used for knowledge distillation. Now, Figure \ref{fig:Bias_motivation} (right) suggests that this soft target can be heavily skewed due to the bias caused by the cross-entropy learning. Namely, the figure shows the ratio of the \textit{tasks} among $\{1,\ldots,t-1\}$, predicted by the \textit{old} model $\bm\theta_{t-1}$ when the \textit{new} task data points $\bm x$'s from $\mathcal{D}_{t}$ was given as input, for each new task $t$ (horizontal axis). We can observe that the predictions are overwhelmingly biased toward the most recent old task (\textit{i.e.}, task $t-1$), which is due to the bias generated during learning task $t-1$ with the cross-entropy loss. This suggests that the soft target $\bm p_{1:t-1}^\tau(\bm x,\bm \theta_{t-1})$ also would be heavily skewed toward the most recent old task (task $t-1$), hence, when it is used in GKD loss as (\ref{eq:GKD_loss}), it will preserve such bias and could highly penalize the output probabilities for the \textit{older} tasks. Hence, it could make the bias, or the \textit{forgetting} of older tasks, more severe. In Section \ref{subsec:kd}, we indeed show that when GKD is naively used, it can even hurt the performance of simple fine-tuning that only uses $\mathcal{L}_{\text{CE},t}$.

Above two observations suggest that main reason for the prediction bias could be due to computing the softmax probability by combining the old and new tasks \textit{altogether}. Motivated by this, we proposed Separated Softmax for Incremental Learning (SS-IL) in the next section.

\begin{figure}[t]
    \centering
    \vspace{-.0in}
    \includegraphics[width=0.8\columnwidth]{figures/motivation_ss.pdf}
    \vspace{-.0in}
    \caption{The toy illustration of gradient descent steps for $m=2$ and $T=2$ on imbalanced $\mathcal{D}_2\cup\mathcal{M}$. As seen above, scores for class $c\in\mathcal{C}_{1}$ continue to decrease due to the imbalanced gradient descent steps.}
    \label{fig:SS_motivation}
    \vspace{-.15in}
\end{figure}

\section{Main Method}\label{sec:method}

Our SS-IL consists of three components, all motivated from the intuition built from the previous section: (1) separated softmax (SS) output layer, (2) ratio-preserving (RP) mini-batch selection, and (3) the task-wise KD (TKD).
Before concretely presenting them, we introduce some additional notations. For the incremental task $t$, we denote the classes of the old tasks by $\mathcal{P}_t=\mathcal{C}_{1:t-1}$ and the classes of the new task $t$ by $\mathcal{N}_t=\mathcal{C}_t$.

\vspace{.03in}

\noindent{\textit{(1) Separated softmax (SS) layer:}  } For $(\bm x, \bm y)\in\mathcal{D}_t\cup \mathcal{M}$, we define a separate softmax output layer by modifying the cross-entropy loss function as
\begin{eqnarray}
    \mathcal{L}_{\text{CE-SS},t}((\bm x, y),\bm\theta) =
    &\mathcal{L}_{\text{CE},t-1}((\bm x,y),\bm\theta)\cdot \mathds{1}\{y\in \mathcal{P}_t\} + \nonumber\\
    &\mathcal{D}_{KL}(\bm y_{t} \| \bm p_{t}(\bm x, \bm \theta))\cdot \mathds{1}\{y\in \mathcal{N}_t\}, \label{eq:ss}

\end{eqnarray}
in which $\bm y_{t}$ stands for the one-hot vector in $\mathbb{R}^{\mathcal{N}_t}$ and $\bm p_{t}(\bm x, \bm \theta))$ is $\bm p_{t}^\tau(\bm x,  \bm\theta)$ with $\tau=1$. Namely, in words,
depending on whether $(\bm x, y)\in\mathcal{M}$  or $(\bm x, y)\in\mathcal{D}_t$, the softmax probability is computed separately by only using the output scores for $\mathcal{P}_t$ or $\mathcal{N}_t$, respectively, and the cross-entropy loss is computed separately as well.
While (\ref{eq:ss}) is a simple modification of the ordinary cross-entropy (\ref{eq:ce_loss}), we can now observe that
$\frac{\partial \mathcal{L}_{\text{CE-SS}}}{\partial z_{tc}}=0$ for $c\in\mathcal{P}_t$ when $(\bm x, y)\in\mathcal{D}_t$. Therefore, the gradient from the new class samples in $\mathcal{N}_t$ will not have overly penalizing effect in the classification scores for the old classes in $\mathcal{P}_t$.

\begin{figure}[t]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/SSIL.pdf}

    \caption{Illustration of SS-IL}\label{fig:SSIL}
    \vspace{-.1in}
\end{figure}

\vspace{.03in}
\noindent \textit{(2) Ratio-preserving (RP) mini-batch: } Another subtle change we implemented is the ratio-preserving (RP) mini-batches for the SGD updates of the model. Note that when random mini-batches are sampled from $\mathcal{D}_t\cup\mathcal{M}$ for SGD, the severe imbalance between new classes and old classes carries over to the mini-batches as well. Such imbalance in mini-batches would significantly downplay the updates of the model for the old classes in our SS layer, since the gradient from the first part of (\ref{eq:ss}) will be generated scarcely. From this observation and to assure the main role of exemplars in $\mathcal{M}$, \ie, to fine-tune the representations and decision boundaries of old classes in response to learning the new classes in $\mathcal{D}_t$, we always generated the mini-batches such that the minimum ratio of the samples from $\mathcal{M}$ can be guaranteed.
Motivated by Experience Replay \cite{(ER)Chaudhry2019} method, this can be simply implemented by concatenating the fixed-sized random samples from $\mathcal{M}$, denoted by \textit{replay batch} ($B_{\mathcal{M}}$) in the later sections, with the random samples from $\mathcal{D}_t$ ($B_{\mathcal{D}_t}$). In our experiments, we set the ratio of new class samples over the old class samples to $2\sim8$ to set the balance between learning new classes and preserving the knowledge learned for old classes.

\vspace{.03in}

\noindent{\textit{(3) Task-wise KD:}  }  With the SS layer, we can easily see that it is natural to use TKD (\ref{eq:TKD_loss}), which also uses the separated softmax for each task, for the knowledge distillation. That is, in TKD, since the soft targets, $\{\bm p_s^\tau(\bm x, \bm \theta))\}_{s=1}^{t-1}$, are computed only within each task, TKD will not get affected by the task-wise bias that may present in the old model $\bm\theta_{t-1}$, as opposed to the GKD shown in Section \ref{subsec:gkd}. Hence, we can expect that TKD is particularly well-suited for the SS layer, which will be shown in our experimental results.

\vspace{.03in}
\noindent \textbf{Final loss function for SS-IL: }
By combining $\mathcal{L}_{\text{CE-SS},t}$ in (\ref{eq:ss}) and $\mathcal{L}_{\text{TKD},t}$ in (\ref{eq:TKD_loss}), the overall loss for SS-IL becomes:
\begin{align}
    \mathcal{L}_{\text{SS-IL},t}((\bm x, y),\bm\theta) =  \mathcal{L}_{\text{CE-SS},t}((\bm x,  y),\bm\theta) + \mathcal{L}_{\text{TKD},t}(\bm x,\bm\theta),\nonumber \label{eq:SSIL}
    \vspace{-0.3in}
\end{align}
and the mini-batch SGD to minimize the loss is done with RP mini-batches.
Figure \ref{fig:SSIL} and Algorithm \ref{alg:Algorithm} illustrates and summarizes our method, respectively. We show in our experimental results that our SS-IL can significantly correct the prediction bias, without \textit{any} score post-processing as is done in many other baselines, as achieve the state-of-the-art accuracy for various CIL benchmark datasets.

\begin{algorithm}[t]
\caption{Separated Softmax for Incremental Learning (SS-IL)}
\begin{algorithmic}
\REQUIRE $\{\mathcal{D}_t\}_{t=1}^T$: Training dataset
\REQUIRE $\mathcal{M}\leftarrow \{\}$: Memory buffer
\REQUIRE $E$: The number of epochs per task.
\REQUIRE $N_{\mathcal{D}_t}, N_{\mathcal{M}}$: Training \& replay batch sizes
\REQUIRE $\alpha$: Learning rate
\REQUIRE $\bm\theta$ : Network parameters
\STATE \textcolor{blue}{\# Start class incremental learning}
\STATE Randomly initialize $\bm\theta$
\FOR{$t=1,...,T$}
\FOR{$e=1,...,E$}
\STATE \textcolor{blue}{\# Sample a mini-batch of size $N_{\mathcal{D}_t}$}
\FOR{$B_{\mathcal{D}_t} \sim \mathcal{D}_t$}
\STATE \textcolor{blue}{\# Sample a mini-batch of size $N_{\mathcal{M}}$}
\STATE $B_{\mathcal{M}} \sim \mathcal{M}$
\STATE $\mathcal{L}_t(\bm \theta) = \sum_{(\bm x, \bm y)\in B_{\mathcal{D}_t} \cup B_{\mathcal{M}}} \mathcal{L}_{\text{SS-IL},t}((\bm x, \bm y), \bm \theta)$
\STATE $\bm\theta \leftarrow \bm\theta - \frac{\alpha}{N_{\mathcal{D}_t}+N_{\mathcal{M}}} \cdot \nabla_{\bm\theta}\mathcal{L}_t(\bm \theta)$
\ENDFOR
\ENDFOR
\STATE $\mathcal{M}\leftarrow$ UpdateMemory($\mathcal{D}_t$, $\mathcal{M}$)
\ENDFOR
\end{algorithmic}
\label{alg:Algorithm}
\end{algorithm}

\section{Experiments}
In this section, we compare our SS-IL with other state-of-the-art CIL methods with various experimental scenarios. For evaluation, we use two large scale datasets: ILSVRC 2012 (ImageNet) \cite{(Imagenet)deng2009} and Google Landmark Dataset v2 (Landmark-v2) \cite{(Landmark-v2)}. In addition, extensive analyses are carried out to show the effectiveness of SS-IL, and the importance of each component consisting the proposed method is analyzed through ablation study. Also, in detailed analyses about the distillation methods, we show the excellence of $\mathcal{L}_{\text{TKD}}$ by comparing $\mathcal{L}_{\text{TKD}}$ and $\mathcal{L}_{\text{GKD}}$.
\subsection{Datasets and evaluation protocol}
For ImageNet and Landmark-v2 datasets, we use all classes in ImageNet dataset, and choose 1,000 and 10,000 classes in Landmark-v2 dataset to make two variations. The detailed explanation on each dataset is as follows:

\noindent\textbf{ImageNet and Landmark-v2}:\ \
ILSVRC 2012 dataset consists of 1,000 classes, which has nearly 1,300 images per class. Google Landmark Dataset v2 consists of 203,094 classes, and each class has $1\sim10,247$ images. We construct Landmark-v2-1K and Landmark-v2-10K which are composed of 1,000 and 10,000 classes respectively.

\noindent\textbf{Evaluation protocol}:\ \
By following the benchmark protocol in \cite{(icarl)rebuffi17}, we arrange the classes of each dataset in a fixed random order. To construct various training scenarios, we vary the total number of incremental tasks as $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ in 1K datasets and $m=\{2000,1000,500\}$ in 10K dataset, respectively. For the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$ for 1K datasets and $|\mathcal{M}|=\{20k,40k,60k\}$ for 10K dataset, respectively. We use the Ringbuffer approach used in \cite{(ER)Chaudhry2019} for constructing the exemplars. For the evaluation of CIL models, we use ILSVRC 2012 validation set for ImageNet-1K, and we randomly selected test samples that are not in the training set.

The details on constructing the datasets and the evaluation protocol are explained in Supplementary Materials.

\subsection{Implementation detail}
The Resnet-18 \cite{(Resnet)HEZhang15} architecture is used in all experiments, and all the implementations are done with the Pytorch framework\cite{(PyTorch)Paszke2017}. For training the neural network, we always use the stochastic gradient descent (SGD) with learning rate 0.1, weight decay 0.0001, and Nesterov momentum 0.9. The batch size used for $\mathcal{D}_t$, $N_{\mathcal{D}_t}$, is 128, and we use different replay batch size, $N_{\mathcal{M}}$, depending on the number of different incremental tasks; \ie,  $N_{\mathcal{M}}=16 / 32 / 64$ for $T=20 / 10 / 5$, respectively. Thus, the ratio of $N_{\mathcal{D}_t}$ over $N_{\mathcal{M}}$ is $8 /4/2$, respectively. The number of epochs for training incremental task is 100, and the learning rate is divided by 10 at epochs 40 and 80.

We compare our SS-IL with iCaRL\cite{(icarl)rebuffi17}, vanilla Fine-Tuning (FT) proposed in \cite{(IL2M)Belouadah}, IL2M\cite{(IL2M)Belouadah}, EEIL\cite{(End2EndIL)Castro2018}, and BiC\cite{(LargeScaleIL)Wu2019}. For iCaRL, as proposed in \cite{(Revisiting)Javed2018}, instead of using binary cross entropy loss for each class output, we use multi-class cross entropy loss for both classification loss and KD loss, which achieves much higher accuracy than the original paper. In Supplementary Materials, all the training details for baselines are explained. For data pre-processing, the random re-sized cropping and horizontal flipping is adopted to all datasets as data augmentation, and normalization with mean and standard deviation is performed only for the ImageNet dataset.

\subsection{Results}\label{subsec:results}
Table \ref{table:Overall_results} shows the results on Average Top-1 and Top-5 accuracy. The left half of the table reports the results of fixed $T=10$ with varying exemplar-memory size $|\mathcal{M}|$, and the right half shows the results of fixed $|\mathcal{M}|$ with varying $T$.
\begin{figure*}[ht]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/ablation.pdf}

    \vspace{-.07in}
    \caption{(a) Ablation study on SS-IL. (b)$\sim$(e) Confusion matrix of four different variations: (b) SS-IL, (c) $\mathcal{L}_{\text{TKD}}$ w/SS, (d) $\mathcal{L}_{\text{TKD}}$ w/RP, (e) $\mathcal{L}_{\text{TKD}}$}\label{fig:Ablation}
    \vspace{-.15in}

\end{figure*}

From the table, we can make the following observations. Firstly, among the baselines, there is no clear winner; EEIL tends to excel for small $|\mathcal{M}|$, while BiC achieves much higher accuracy than other baselines  for large $|\mathcal{M}|$. Secondly, SS-IL consistently dominates other baselines for all the large-scale datasets throughout every possible scenarios. In particular, SS-IL outperforms other state-of-the-art baselines with significant margin in ImageNet-1K and Landmark-v2-10K. Lastly, when smaller $|\mathcal{M}|$ is used, the accuracy drop is marginal in all datasets, which indicates SS-IL is somewhat robust to $|\mathcal{M}|$.

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/Top-5.pdf}
    \vspace{-.15in}
    \caption{Incremental learning results on ImageNet-1K, Landmark-1K, and Landmark-10K datasets for $T=10$. The exemplar size is $|\mathcal{M}|=20k$ in ImageNet-1K and Landmark-1K datasets, and $|\mathcal{M}|=60k$ in Landmark-v2-10K dataset.}\label{fig:Top_5}
    \vspace{-.13in}
\end{figure}

Figure \ref{fig:Top_5} shows the overall result on each dataset with respect to the incremental task, when $|\mathcal{M}|=20k$ and $T=10$, and the tasks are denoted as classes. In this figure, we denote jointly trained approach as the Upper-bound. Note that SS-IL again mostly dominates the baselines, and the performance gap over the baselines widens as the incremental task increases. Especially, in ImageNet-1K, compared with other baselines which have more performance degradation from the Upper-bound, our SS-IL is less affected by catastrophic forgetting. Furthermore, we observe that iCaRL and EEIL achieves lower accuracy in the first incremental task. Weak Nearest Exemplar Mean (NEM) classifier in iCaRL and inefficient training schedule in EEIL are the main reasons of low accuracy.

\begin{comment}
Figure \ref{fig:Analysis}(a) shows the misclassified error ratios $e_t(p,n)$ and $e_t(n,p)$ more in details for ImageNet-1K dataset with $T=10$ and $|\mathcal{M}|=10k$. From the figure, which essentially adds the results of iCaRL and SS-IL to Figure \ref{fig:Bias_motivation}, we first observe that $e_t(p,n)$ of FT and IL2M drastically increases as the incremental task increases, while $e_t(n,p)$ is significantly small due to the classification score bias. Namely, the most of errors they make are simply due to predicting most of the old classes as the new classes. We note IL2M does improve upon FT after the post-processing, but it still suffers from the heavy bias.
On the other hand, for iCaRL, we observe that $e_t(p,n)$ is significantly lower than those of FT and IL2M, while $e_t(n,p)$ is now much higher than those of FT and IL2M. This suggests that the knowledge distillation of iCaRL does induce to make more predictions toward the old classes, but the low accuracy shows that it hinders the accurate predictions within the old classes.
Furthermore, we clearly observe that our SS-IL shows the most balanced error ratios among the methods. That is, we note $e_t(p,n)$ for SS-IL is significantly lower than those of FT and IL2M (similar to that of iCaRL) and $e_t(n,p)$ also stays more or less similar to $e_t(p,n)$ throughout the incremental tasks. This confirms that our SS layer learned with RP mini-batch makes much more balanced classification decisions, resulting in much fewer confusions between the old and new classes, hence, higher prediction accuracy as shown in Figure \ref{fig:Top_5}.
\end{comment}

\subsection{Ablation study}\label{subsec:ablation}
In this section, we perform various detailed analyses to show the effectiveness of replay batch sizes while varying the total number of incremental tasks. Also, we do ablation study on the components of our SS-IL method and demonstrate their impact.

\noindent\textbf{Ablation study on SS and RP}
In this section, we validate our approach by ablating each component of SS-IL. Figure \ref{fig:Ablation} shows the ablation study results for ImageNet-1K with $|\mathcal{M}|=10k$, $T=10$. In this figure, ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' stands for the model that selects mini-batches for SGD as in our SS-IL, but does not have the separated softmax layer, ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' stands for the model that has the separated softmax layer as our SS-IL but randomly selects mini-batches from $\mathcal{D}_t\cup\mathcal{M}$, and ``$\mathcal{L}_{\text{TKD}}$'' stands for the model that only uses $\mathcal{L}_{\text{TKD}}$ without SS and RP. In Figure \ref{fig:Ablation} (a), thanks to the effectiveness of SS, ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' achieves higher accuracy than ``$\mathcal{L}_{\text{TKD}}$'', and by comparing Figure \ref{fig:Ablation} (c) and (e), using SS makes much more balanced decisions. Furthermore, we observe that ``$\mathcal{L}_{\text{TKD}}$ w/ SS'' achieves almost same accuracy as SS-IL. One may think that using RP has no effect on CIL problem. However, a comparison of Figure (b) and (c) shows that SS-IL makes more balanced decisions than ``$\mathcal{L}_{\text{TKD}}$ w/ SS'', which means using SS and RP together achieves the highest accuracy and makes more balanced decisions. Interestingly, ``$\mathcal{L}_{\text{TKD}}$'' largely outperforms ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' which uses RP only, and the decision of ``$\mathcal{L}_{\text{TKD}}$ w/ RP'' is highly biased toward new classes.

\begin{table}[h]
\small
\centering
\vspace{-.0in}
\caption{Results on ImageNet-1K with varying $N_\mathcal{M}$ and $T$.
}\label{table:batch_state_results}
\vspace{.0in}
\resizebox{1.0\columnwidth}{!}{\begin{tabular}{cc|c}
\hline
\multicolumn{1}{c|}{$T$/$N_\mathcal{M}$} & 16 / 32 / 64        & 16 / 32 / 64        \\ \hline\hline
                             & Average Top-1 accuracy    & Average Top-5 accuracy   \\ \hline\hline
\multicolumn{1}{c|}{20}      & 58.8 / 59.0 / 58.9 & 82.9 / 82.6 / 82.4 \\
\multicolumn{1}{c|}{10}      & 64.3 / 64.5 / 64.1 & 86.6 / 86.4 / 86.0 \\
\multicolumn{1}{c|}{5}       & 68.4 / 68.4 / 68.2 & 88.8 / 88.6 / 88.4 \\ \hline
\end{tabular}}
\vspace{-.05in}
\end{table}

\noindent\textbf{Analysis on $N_{\mathcal{M}}$}\ \ Table \ref{table:batch_state_results} shows the results on Average Top-1 and Top-5 accuracy with respect to varying replay batch size, $N_\mathcal{M}$, and the total number of incremental tasks, $T$, for ImageNet-1K on $|\mathcal{M}|=10k$. From Table \ref{table:batch_state_results}, we observe that no matter what $N_{\mathcal{M}}$ is being used, the accuracy difference is negligible. This indicates that, using RP is still effective
regardless of the ratio between old and new class samples in the mini-batch,
if the old class examples are guaranteed to some extent.

\subsection{Analyses on KD}\label{subsec:kd}

In this section, given different bias correction schemes, we carry out several experiments to compare TKD with GKD and FT. We use three different bias correction schemes: balanced fine-tuning (BFT) \cite{(End2EndIL)Castro2018}, score correction \cite{(LargeScaleIL)Wu2019}, and SS+RP (Ours). For a fair comparison, same training settings are used for TKD and GKD. Note that we use same FT as we described in section \ref{subsec:results}. All the training details are explained in Supplementary Materials.

\noindent\textbf{Comparison of $\mathcal{L_{\text{TKD}}}$ and  $\mathcal{L_{\text{GKD}}}$} Table \ref{table:local_global_acc} shows the  Average Top-1 and Top-5 accuracy with respect to varying KD loss and bias correction method.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\columnwidth]{figures/global_to_local.pdf}
    \vspace{-.1in}
    \caption{Top-1 accuracy for $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$ on two different bias correction schemes at $t=3,6,9$}\label{fig:global_to_local}
    \vspace{-.12in}
\end{figure}

\begin{table}[h]
\small
\centering
\vspace{.1in}
\caption{Results on ImageNet-1K with varying the KD losses and bias correction schemes
}\label{table:local_global_acc}
\vspace{.05in}
\resizebox{1.0\columnwidth}{!}{\begin{tabular}{ccc}
\hline
\multicolumn{1}{c|}{Schemes / KD losses} & \multicolumn{1}{c|}{FT / GKD / TKD}      & FT / GKD / TKD       \\ \hline\hline
                                       & Average Top-1 accuracy                     & Average Top-5 accuracy \\ \hline\hline
\multicolumn{1}{c|}{No bias correction}          & \multicolumn{1}{c|}{53.6 / 54.7 / \textbf{56.8}} & 78.8 / 77.3 / \textbf{81.8}  \\ \hline
\multicolumn{1}{c|}{Balanced Fine-Tuning}    & \multicolumn{1}{c|}{61.3 / 63.6 / \textbf{64.8}} & 83.6 / 85.3 / \textbf{86.2}  \\ \hline
\multicolumn{1}{c|}{Score Correction }            & \multicolumn{1}{c|}{53.6 / 63.3 / \textbf{63.8}} & 78.8 / 84.3 / \textbf{85.3}  \\ \hline
\multicolumn{1}{c|}{SS + RP}              & \multicolumn{1}{c|}{59.3 / 64.5 / \textbf{65.3}} & 82.9 / 86.5 / \textbf{86.7}  \\ \hline
\end{tabular}}
\vspace{-.1in}
\end{table}

Models are trained with GKD, TKD, and without KD (\ie FT) while applying different bias correction schemes for ImageNet-1K on $|\mathcal{M}|=20k$, $T=10$. As shown in Table \ref{table:local_global_acc}, compared to FT, TKD achieves the highest accuracy in every case while GKD is not always effective. According to \cite{(IL2M)Belouadah,(scail)belouadah2020}, using distillation in CIL which inherently includes class imbalance causes a drop of performance. However, we observe better performance of TKD over FT even in the absence of bias correction method, which implies that using TKD can be effective. Similarly, \cite{(UnlabeledIL)Lee2019} mention that using TKD misses the knowledge about discrimination between old tasks. Meanwhile, our result implies that TKD may be a better way of using KD in CIL. Overall, TKD is an effective way to prevent catastrophic forgetting in exemplar-memory based CIL.

\noindent\textbf{Existence of bias on $\mathcal{L}_{\text{GKD}}$} In this section, for further analysis on $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$, we carry out another experiment with newly designed training scenario. The scenario is as follows:
\begin{enumerate}
    \item Train a model using $\mathcal{L}_{\text{GKD}}$ until incremental task $t-1$, then we obtain $\bm \theta_{t-1}$.
    \vspace{-.07in}
    \item At incremental task $t$, train $\bm \theta_{t-1}$ using two different KD losses, $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$. As a result, we obtain two different models, $\bm \theta_{\text{GKD},t}$ and $\bm \theta_{\text{TKD},t}$.
\end{enumerate}

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/prediction_map.pdf}
    \vspace{-.15in}
    \caption{Task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$}\label{fig:prediction_map}
    \vspace{-.15in}
\end{figure}

We apply above scenario to the models that use BFT and SS+RP. As a result, two different KD models are obtained per each bias correction method, and we evaluate each model on same test data. Note that since identical $\bm \theta_{t-1}$ is used for KD, we can directly compare $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$.

Figure \ref{fig:global_to_local} shows the Top-1 accuracy of $\bm \theta_{\text{GKD},t}$ and $\bm \theta_{\text{TKD},t}$ on BFT and SS+RP at $t=3,6,9$, and Figure \ref{fig:prediction_map} shows the task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$.

In Figure \ref{fig:global_to_local}, for the same $\bm \theta_{t-1}$, the accuracy of $\bm \theta_{\text{TKD},t}$ is higher than that of $\bm \theta_{\text{GKD},t}$ at task 3, 6, 9. We assume that the main reason for the accuracy difference is due to the bias preservation caused by the score bias of $\bm \theta_{t-1}$ on $\bm x \in \mathcal{D}_t$. As shown in Figure \ref{fig:prediction_map}, the ratio of the prediction on the latest old task is higher than the ratio of any other tasks. For example, at incremental task 9, the predictions for $\bm x \in \mathcal{D}_9$ are predominantly made toward task 8. Therefore, since $\bm p^{\tau}_{1:t-1}(\bm x, \bm \theta_{t-1})$ used in  (\ref{eq:GKD_loss}) can be heavily biased due to the biased score, $\mathcal{L}_{\text{GKD}}$ rather induces catastrophic forgetting. On the other hand, similar to the intuition of SS, TKD performs distillation using task specific separated softmax which is not affected by score bias between tasks. As a result, using TKD in CIL can be a better choice compared to using GKD.

\section{Datasets and evaluation protocol}

\noindent\textbf{ImageNet}:\ \
ILSVRC 2012 dataset consists of 1,000 classes, which has nearly 1,300 images per class. By following the benchmark protocol in \cite{(icarl)rebuffi17}, we arrange the classes of each dataset in a fixed random order. We experiment with varied total number of incremental tasks, $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ per task, and for the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$. When constructing exemplar-memory, we use Ringbuffer approach proposed in \cite{(ER)Chaudhry2019}, which simply samples random data from old classes.
We always maintain balanced number of exemplars across all the old classes. Thus, as the incremental task increases, we delete equal number of exemplars from the old classes and add exemplars for the newly learned classes. For the evaluation of CIL models, we use ILSVRC 2012 validation set for testing.

\noindent\textbf{Landmark-v2}:\ \
Google Landmark Dataset v2 consists of 203,094 classes, and each class has $1\sim10,247$ images. Since the dataset is highly imbalanced, we sample 1,000 and 10,000 classes in the order of largest number of samples per class. We denote Landmark-v2 dataset with 1,000 and 10,000 classes as Landmark-v2-1K and Landmark-v2-10K, respectively. After sampling the classes, we arrange the classes in a fixed random order. Similarly as in ImageNet, we vary the total number of incremental tasks as $T=\{5,10,20\}$, which corresponds to $m=\{200,100,50\}$ in Landmark-v2-1K and $m=\{2000,1000,500\}$ in Landmark-v2-10K, respectively. For the exemplar-memory size, we use $|\mathcal{M}|=\{5k,10k,20k\}$ for Landmark-v2-1K and $|\mathcal{M}|=\{20k,40k,60k\}$ for Landmark-v2-10K, respectively. Same as in ImageNet, we use the Ringbuffer approach for constructing the exemplars.
For evaluation, we randomly select 50 and 10 images per each class in Landmark-v2-1K and Landmark-v2-10K that are not in the training set for testing.

\section{Implementation details}\label{sec:Implementation details}
All the baselines use the Resnet-18\cite{(Resnet)HEZhang15} architecture and are implemented using Pytorch framework\cite{(PyTorch)Paszke2017}. We always use the stochastic gradient descent (SGD) with weight decay 0.0001 and momentum 0.9. We planned to use WA\cite{(WA)Zhao2020} as one of our baselines for comparison. However we could not compare our method with it since it did not publish its official code and we failed to reproduce it. Including SS-IL and all the other baselines, the code implementations will be publicly available.

\noindent\textbf{iCaRL}:\ \
iCaRL is implemented based on \cite{(Revisiting)Javed2018}. We use multi-class cross entropy loss for both classification loss and KD loss instead of binary cross entropy. The number of epochs for training incremental task is 60. The learning rate starts at 0.1 and is divided by 5 at 20, 30, 40 and 50 epochs. The size of mini-batches is 128.

\noindent\textbf{FT and IL2M}:\ \
FT and IL2M are implemented based on \cite{(IL2M)Belouadah}. The first incremental task consists of 100 epochs with initial learning rate 0.1 and the learning rate is divided by 10 at 40 and 80 epochs. The rest incremental tasks consists of 25 epochs with initial learning rate lr=$\frac{0.1}{\text{t}}$, where t is the incremental task. The learning rate is divided by 10 at 10 and 20 epochs. The size of mini-batches is 128.

\noindent\textbf{EEIL}:\ \
EEIL is implemented based on \cite{(End2EndIL)Castro2018}. Each incremental step consists of 40 epochs for training and an additional 30 epochs for balanced fine-tuning. For the first 40 epochs, the learning rate starts at 0.1 and is divided by 10 at 10, 20 and 30 epochs. For balanced fine tuning, the learning rate starts at 0.01 and is divided by 10 at 10 and 20 epochs. The size of mini-batches is 128.

\noindent\textbf{BiC}:\ \
BiC is implemented based on \cite{(LargeScaleIL)Wu2019}. The number of epochs for training incremental task is 100. The learning rate starts at 0.1 and is divided by 10 at 30, 60 and 80 epochs. In addition, the number of epochs for training bias correction layer at incremental task is 200. The learning rate for bias correction layer starts at 0.001 and is divided by 10 at 60, 120, 180 epochs. The size of mini-batches is 256.

\section{Additional results}
Figure \ref{fig:results_top_1} and \ref{fig:results_top_5} show the detailed results used to generate (Table 1, manuscript). Overall, SS-IL achieves much higher accuracy than other baselines for most of scenarios. Especially, for the smallest $|\mathcal{M}|$, SS-IL significantly outperforms other baselines, and the accuracy is not much smaller than for the largest $|\mathcal{M}|$, which means it is sufficient for SS-IL to use less memory in CIL.

\begin{figure}[ht]
    \centering
    \subfigure[$T=10$ and $|\mathcal{M}|=5k$(1K), $20k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_20000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=20k$(1K), $60k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_60000_1000.pdf}}
    \subfigure[$T=20$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_500.pdf}}
    \subfigure[$T=5$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-1_40000_2000.pdf}}
    \caption{Top-1 accuracy on ImageNet-1K, Landmark-1K and, Landmark-10K.}\label{fig:results_top_1}
\end{figure}

\begin{figure}[ht]
    \centering
    \subfigure[$T=10$ and $|\mathcal{M}|=5k$(1K), $20k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_20000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_1000.pdf}}
    \subfigure[$T=10$ and $|\mathcal{M}|=20k$(1K), $60k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_60000_1000.pdf}}
    \subfigure[$T=20$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_500.pdf}}
    \subfigure[$T=5$ and $|\mathcal{M}|=10k$(1K), $40k$(10K)]{
    \includegraphics[width=1.0\columnwidth]{figures/Top-5_40000_2000.pdf}}
    \caption{Top-5 accuracy on ImageNet-1K, Landmark-1K and, Landmark-10K.}\label{fig:results_top_5}
\end{figure}

\section{Analyses on KD}

\subsection{Training Details}\label{subsec:training_details}

We use the Resnet-18\cite{(Resnet)HEZhang15} architecture, implemented on Pytorch\cite{(PyTorch)Paszke2017} framework. For training the neural network with TKD and GKD, we use the stochastic gradient descent (SGD) with learning rate 0.1, weight decay 0.0001, and momentum 0.9. The batch size used for $\mathcal{D}_t$ is 128. The number of epochs for training incremental task is 100, and the learning rate is divided by 10 at epochs 30, 60, 90. For temperature scaling parameter $\tau$ in (Eq.(2) and Eq.(3), manuscript), we set $\tau=2$. The training schedule for FT is same as mentioned in \ref{sec:Implementation details}. After learning the model, we correct the bias by some additional post processing: Balanced Fine-Tuning (BFT) and Score correction, respectively. In case of SS+RP, we carry out the bias correction step on the model without any additional post-processing steps.

\noindent\textbf{Balanced Fine-Tuning}:\ \
BFT is implemented based on \cite{(End2EndIL)Castro2018,(IL2M)Belouadah}. After training incremental task $t$, we perform additional 30 epochs for BFT. Learning rate is set to $\frac{0.001}{t}$ which is proposed in \cite{(IL2M)Belouadah} when applying to FT, GKD, and TKD.

\noindent\textbf{Score correction}:\ \
Score correction is implemented based on \cite{(LargeScaleIL)Wu2019}. The number of epochs for training bias correction layer at incremental task $t$ is 200 for TKD, GKD and 50 for FT. The learning rate starts at 0.001 and is divided by 10 at 60, 120, 180 epochs for TKD, GKD and at 20, 40 epochs for FT. The size of mini-batches is 128.

\noindent\textbf{SS+RP}:\ \
SS+RP is implemented based on (Eq.(7), manuscript). The size of RP mini-batches is 32, and all the other training schemes are same as specified in manuscript.

\subsection{Existence of bias on $\mathcal{L}_{\text{GKD}}$}\label{subsec:bias_existence}

\begin{figure}[h]
    \centering
    \includegraphics[width=1\columnwidth]{figures/global_to_local_supplementary.pdf}
    \vspace{-.1in}
    \caption{Top-1 accuracy for $\mathcal{L}_{\text{GKD}}$ and $\mathcal{L}_{\text{TKD}}$ on two different bias correction schemes at each task $t$ ($t = 3 \sim 10$)}
    \label{fig:global_to_local_supplementary}
    \vspace{-.12in}
\end{figure}

\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\columnwidth]{figures/prediction_map_supplementary.pdf}
    \vspace{-.15in}
    \caption{Task ratio of $\hat{t}$ on $\bm \theta_{t-1}$ for input $\bm x \in \mathcal{D}_t$ at each task $t$ ($t = 3 \sim 10$)}\label{fig:prediction_map_supplementary}
    \vspace{-.15in}
\end{figure}

Figure \ref{fig:global_to_local_supplementary} and Figure \ref{fig:prediction_map_supplementary} show the detailed versions of (Figure 7 and Figure 8, manuscript), respectively. For both BFT and SS+RP, the results are presented at task $t = 3 \sim 10$.
Observation starts from $t = 3$ since training at least 2 tasks is needed for the bias on $\mathcal{L}_{\text{GKD}}$ to occur.

In Figure \ref{fig:global_to_local_supplementary}, for the same $\bm \theta_{t-1}$, the Top-1 accuracy of $\bm \theta_{\text{TKD},t}$ is higher than that of $\bm \theta_{\text{GKD},t}$ at every task $t$ in the range of $t = 3 \sim 10$. We assume that the main reason for the accuracy difference is due to the bias preservation caused by the score bias of $\bm \theta_{t-1}$ on $\bm x \in \mathcal{D}_t$. As shown in Figure \ref{fig:prediction_map_supplementary}, the ratio of the prediction on the latest old task is higher than the ratio of any other tasks.
Especially, for the case of BFT, this significantly increases as the incremental task $t$ grows.
It leads to the result that the accuracy difference between $\bm \theta_{\text{TKD},t}$ and $\bm \theta_{\text{GKD},t}$ for BFT is large in comparison with that for SS+RP.

\clearpage

{\small
\bibliographystyle{ieee_fullname}

redpajama_set_name: RedPajamaArXiv