Compact Memory for Continual Logistic Regression
Yohan Jung, Hyungi Lee, Wenlong Chen, Thomas Möllenhoff, Yingzhen Li, Juho Lee, Mohammad Emtiyaz Khan
Despite recent progress, continual lifelong learning algorithms still do not match the performance of batch training. A major challenge is to represent old knowledge and reuse it to learn while avoiding forgetting. It remains unclear how to compactly represent past memories, and there is no clear answer even in shallow neural networks with just one or two layers.In this paper, we take a step toward building compact memory. We build upon a result by Khan and Swaroop [2021], who show the existence of optimal memory for logistic regression. We show that finding the memory can be formulated as Hessian matching and propose a Probabilistic principal component analysis (PPCA)-based method to obtain the memory. Our approach can drastically reduce memory size without sacrificing performance, for instance, we can obtain $30\%$  improvement in accuracy over Experience Replay that uses $1\%$ of the data for memory when adapting it with the the pre-trained Vision transformer as a feature extractor. Our work takes the first step to build compact memory which can be useful for deeper models in the future.