text:
The Delta 4000 series was an American expendable launch system which was used to conduct two orbital launches in 1989 and 1990. It was a member of the Delta family of rockets. Although several variants were put forward, only the Delta 4925 was launched. The designations used a four digit numerical code to store information on the configuration of the rocket. It was built from a combination of spare parts left over from earlier Delta rockets, which were being retired, and parts from the Delta II 6000-series, which was just entering service.

The first stage was the MB-3-III powered Extended Long Tank Thor, previously flown on the 1000-series. Nine Castor-4A solid rocket boosters were attached to increase thrust at lift-off, replacing the less powerful Castor-4 boosters used on the 3000 series. The Delta-K was used as a second stage. A Star-48B PAM-D was used as a third stage, to boost payloads into geosynchronous transfer orbit.

Both Delta 4000 launches occurred from Launch Complex 17B at Cape Canaveral. The first launched Marco Polo 1 for BSkyB, and the second launched INSAT 1D for the Indian Space Research Organisation. Both were successful.

References

Delta (rocket family)


My foray into getting my intentions, thoughts, musings, meditations and ideas out into the universe!
this is what came to me this morning....my breath is a carpet, an amazing and beautiful persian rug.
when you get a chance, take a ride on my little magic carpet, or rather your very own!
So, last night at a lovely Seder dinner the discussion of an orange on a Seder plate came up. Without going into deep discuss it symbolizes the right of anyone to practice what they believe in no one can be left out due to race, religion , sexual identity, disability or difference. This morning after my mantra meditation I used this analogy to do my own little pit spitting...spitting out my seeds of judgement, the pits that block me....give it a try and see what you can spit out!
simple breath and listening is what came to me this morning....after chanting, i decided to just listen, first to the sounds of my own breath, then the sounds in my room, then the sounds outside my room....i find that each time i try this practice i am able to go a little deeper or you might say a little further with my listening!
HOW FAR CAN YOU GO WITH YOUR LISTENING?


text:
Hill-Rom (NYSE:HRC) är en amerikansk medicinteknikkoncern med säte i Batesville, IN, USA. Sedan 2008 äger Hill-Rom det svenska vårdföretaget Liko AB, med huvudkontor i Alvik utanför Luleå.

Externa länkar
Officiell webbplats

Referenser

Svenska medicintekniska företag

redpajama_set_name: RedPajamaWikipedia


Mission Assertion: We develop dynamic communicators who think critically and communicate, listen, and write strive to embody this mission in several methods. In this current day and age, communications, each digital and physical, current a optimistic power in our society by advancing our "Information Age" exponentially and allowing individual-to-individual contact that heretofore seemed inconceivable, although this societal saturation of communication can seem overwhelming at instances on an individual degree.
With proximity to New York State's capital and state authorities, in the midst of New York State's burgeoning Tech Valley, and with entry to the intellectual sources that solely a large public analysis establishment can provide, the Division of Communication at the College at Albany is an exciting place to review for both undergraduate and graduate students.
This program provides students a foundation in communications, digital media, social media and digital journalism. Non-verbal behaviours are multifunctional." 6 Many different non-verbal channels are engaged at the identical time in communication acts, and permit the chance for simultaneous messages to be sent and acquired.
Hold observe of the time and different ways you spend communicating, akin to talking individual-to-individual, listening to academics, the radio, or podcasts, watching tv, studying books and different print media, and utilizing any digital communication gadget.
Good communication is crucial in nearly each business, helping to promote merchandise to the general public, keep strong relationships with buyers, purchasers and prospects, and – not all the time as distinguished however just as important – to make sure everybody within the enterprise and out of doors are working on the same web page.


Finding a real psychic isn't as hard as you might think. All that is required, really, is a discerning mind. Here are some easy tips on how to find a real psychic and avoid psychics who might be unethical, disreputable, or taking advantage of you.
Cheap psychic readings or free psychic readings — you might get taken.
Or, "free" anything! Didn't mama tell you that nothing in life is truly free?
Free is something that will draw most of us in, because who doesn't want something for free? However, a free psychic reading invariably has some really BIG hooks attached and you may not be getting a real psychic or a real psychic reading for that matter.
With a lot of psychic network websites; the ones with a hundred or more psychics on staff, you'll often see the words "free", "free live psychics" or "free psychic readings" somewhere in a headline. This will be their first hook, because in all likelihood, you'll click on their site to get that free psychic reading.
In the interest of "truth in advertising", once there, you'll discover that you can get a free psychic reading, but it's not with a real psychic; it's computer generated!
Since you are already actively within their site, you'll also notice that instead of a computer generated free psychic reading, they offer "live" psychic readers as well. And that is the next hook. You don't want an impersonal computerized psychic reading, you want to speak to a real psychic!
Yes, the first few minutes are free, but the psychics listed charge anywhere from $4.00 to $13.00 per minute beyond that. That's right, as much as $13.00 per minute! For a brief 15 minute reading; suddenly that enticing "free" psychic reading done by a "real" live psychic is going to cost you $195.00! Should you lose sight of the time during your psychic reading; after all, the psychic is purposely and actively engaging you, you could end up paying hundreds.
Another ploy is the "package deal" of say $30.00 for 30 minutes for your psychic reading and at a fixed price, it sounds fantastic and reasonable. Well, it isn't fantastic or reasonable, because it isn't a fixed price. If you also notice, the psychics listed with this same company will advertise at a charge per minute ($3.95 and up) and this is what you could actually end up paying; therefore, the fixed price is misleading.
Often, these "real psychics" are specifically trained to keep you on the phone for as long as possible. For obvious reasons, this makes the company more money and the psychic too. It's hardly a good sign that you've found a real or true psychic who will actually care about you, if making the most money from you is the motivating factor.
And, since you are probably in a vulnerable state of mind, you will not likely notice that a 30 minute psychic reading has turned into 60 minutes or more. The packaged deal of $30.00 for 30 minutes has morphed into 60 minutes and the total cost levied you will be at the additional rate per minute−or another $118 and likely, far more!
How do you like your "free psychic reading" so far? Did you find a real psychic?
Best psychic directory or best psychic medium directory — can you find a real psychic that way?
Thinking you might find a real psychic by going through a psychic directory?
Listed at the top of a Google search, you might also find a handy-dandy list of the top 10 psychic networks, best psychic directory or the best psychic medium directory, all compiled in one place for your ease to choose from.
The thing with these lists is that whoever compiled them, also receives monetary kickbacks from the very companies listed (every time you click through) or the company who compiled the list, also happens to have these "best" psychics. Of course, they will say they are the top! They have a vested interest in your believing as much. You won't necessarily be viewing it this way; you'll think how nice of them to make it so easy for you—but that is what makes this more like a bait and switch.
While some of these lists may truly be helpful or legitimate; personally, I question any person or company who utilizes these sorts of marketing tactics, especially where human vulnerabilities are concerned. I would not feel as though I had found a real psychic or one who had my best interests in mind and heart. At the very least, I would approach with more skepticism or awareness.
Whatever you decide is best for you; a private psychic, psychic network or going through a psychic directory, to find a real and true psychic, do your homework. Investigate company/personal policies, read about the psychic so you can get a feel for him or her and ascertain what you will actually be charged. If you go with a per minute based system, be sure that YOU end the call after you have reached your time and dollar limit.
Don't give the psychic any information. Find a real psychic who will be able to connect to you and give you information that is accurate and true and you will know it, without your providing them information.
If you find a real psychic or one that you feel is, be sure to check them out first. Be very careful of websites that were free to create or appear as though they were put up in a day, because they can disappear just as easily.
Check Icann Whois for information on who owns the domain url; when it was created and when it is due to expire. Although the registrant's personal information may be private (understandably), you can still ascertain the creation date and the date the domain is due to expire (although it might be renewed). This will give you an idea of whether the person and/or website are well established. A real psychic has likely been around for a while and intends to stay.
It goes without saying to be very careful with your credit card information at all times no matter who you choose. Make sure the website is secure and uses https and/or double check the url to make sure you are at a legitimate payment processor, such as PayPal.
Find a real psychic by noting that they don't ask questions at all or very few: watch out for any psychic who asks too many questions up front or asks leading questions that require you to expand upon the information you provide them. Keep your answers as simple and vague as possible, respond with a yes or no and preferably, say nothing at all.
A real psychic reading is energy based. In my opinion, either a psychic or Tarot reader can read one's energy in that moment or they can't. Why would they need direction from you? Find a real psychic − one who doesn't ask a lot of questions or leads you for more information.
Many Tarot card readers or psychics can't read for someone unless they have your question(s) in mind first or know the purpose of your visit. Again, this can be another form of direction. If not a real psychic, possessing even this much information could enable them to tailor your psychic reading based upon what you hoped to hear. They could literally provide a "scripted" psychic reading.
Although it may be a genuine need for very few psychics to know your questions upfront or the purpose of your visit, to me, it raises a BIG red flag.
Over the years, I have seen many psychics and Tarot card readers. At first sitting, most would ask what questions I had in mind and I would say, "I am completely open, tell me what you see or feel." This would completely fluster them and what would generally follow was an inability to read me at all. In other words, the so-called psychic reading was way too general or completely off. If a psychic becomes flustered by not having a basis point to work from, I highly doubt you have found a real psychic.
There are times when a real psychic is connecting, but the information presented isn't in your consciousness as yet. However, in an instance such as this, it should still be a smaller piece of the whole psychic reading pie. Most of the psychic reading should resonate with you and you should not need to give overt pointers in the process.
At the onset of a psychic reading, the most information I get from my clients is their first name and astrological sign or month/day of birth. This allows me to focus in on them (their energy) and the energies around them and then off I go. I do most of the talking and prefer that they not give me any information before or during their psychic reading. If they do, I ask that they only confirm or deny what I have already told them first; however, doing so will still not change what I see or what I feel. My philosophy is such that I prefer the purity of information that I will receive, uncolored by personal perceptions or perspectives. This allows me to provide the most accurate psychic reading possible and for me to be the most objective and helpful.
There are times when a psychic reading can be all over the place. This is due to many factors, such as a lot going on in one's life or complicated issues. In this case, I will do my best to piece together what I am seeing or feeling, but in order to make even more sense of it, it may require some input from my client's too.
I have a good friend who went to see a very high-profile and famous psychic. This woman had written several books and was seen regularly on television shows. She also charged $750.00 dollars for a 30-minute reading!
My friend's experience was horrible and she suspected that this woman was not only far from being a legitimate or real psychic, but a complete fraud. I have since heard and read about several other people having similar experiences with this same psychic. An expensive lesson to learn for all of them!
The best, most accurate psychic reading is not necessarily received via the most expensive, famous or top-rated psychic. Sometimes the truly great and real psychics in this world are almost complete unknowns and you can find these real psychics, you just may need to search a bit more beyond the third or fourth page of search engines or get referrals from friends or family.
While this adage usually holds true—you get what you pay for—there is no substantive difference between a pair of Jimmy Choo shoes that cost $3,000.00 and a well-made pair of Italian shoes that cost $300.00. In this case, there is no difference in quality; rather, what someone is willing to pay and what motivates them to pay it. The key point here is quality.
One can also purchase something inexpensive and at lesser quality and find that they need to replace this item repeatedly; at a far greater cost than it might have been to buy quality in the first place. Psychic readings are no different. Getting a cheap psychic reading one after the other that is neither helpful nor accurate can add up—a lot.
Finding quality is king and balance is the answer: going to one extreme (expensive psychic readings) will not garner any better than going to the other extreme (cheap or free psychic readings). Find a real psychic who has your best interests at heart and who has a track record of being a legitimate psychic and provides accurate psychic readings.
Finding a real psychic is well-worth any time and effort you put into it. A real psychic will likely leave you feeling empowered, more knowledgeable about your life direction or clearer about your feelings—those things that are far more important and life-changing than hearing something truly unhelpful, insignificant or fleeting.
Remember, real psychic ability emanates from a higher place and has spiritual goals for personal growth and a life well and happily lived.


text:
Q: How do I find the largest negative value in an array with both positive and negative values? I need to return the greatest negative value, and if there are no negative values, I need to return zero.
Here is what I have:
public int greatestNegative(int[] list) {

    for (int i = 0; i < list.length; i++) {

       if (list[i] < 0)
           negativeNumbers ++;
    }

    int j = list.length - 1;

    while (j >= 0) {
       if (list[j - negativeNumbers] < 0) {
        list[j] = 0;
        list[j - 1] = list[j - negativeNumbers];
        negativeNumbers--;
        j--;
       }
       else{
        list[j] = list[j - negativeNumbers];
        j--;
     }
  }

}

A: You just need to think of this problem as 2 steps:

*

*Only consider negative values in list[].

*In the loop within negative values, update current result if (result == 0) or (value > result).

Code:
public int greatestNegative(int[] list) {
    int result = 0;
    for (int i = 0; i < list.length; i++) {
        if (list[i] < 0) {
            if (result == 0 || list[i] > result) {
                result = list[i];
            }
        }
    }
    return result;
}

A: Just go about finding the max number with an added condition.
public static int greatestNegative(int[] list) {
    int max = Integer.MIN;
    boolean set = false;
    for (int i = 0; i < list.length; i++) {
        if (list[i] < 0 && list[i] > max) {
             max = arr[i];
             set = true;
        }
    }
    if (!set)
        max = 0;
    return max;
}

A: Here is the code which return the smallest negative number
public static int greatestNegative(int[] list) {
        int negativeNumbers = 0;
        for (int i = 0; i < list.length; i++) {
           if (list[i] < 0 && list[i] < negativeNumbers)
               negativeNumbers  = list[i];
        }

        return negativeNumbers;
    }

Input : 1, 2, -3, 5, 0, -6
Output : -6
Input : 1, 2, 3, 5, 0, 6
Output : 0

A: You have to try this....
public int greatestNegative(int[] list) {
    int negNum = 0;
    for(int i=0; i<list.length; i++) {
        if(list[i] < negNum){
            negNum = list[i];
        }
    }
    return negNum;
}

public int largNegative(int[] list) {
    int negNum = 0;
    boolean foundNeg = false;
    for(int i=0; i<list.length; i++) {
        if(list[i] < negNum && !foundNeg){
            foundNeg = true;
            negNum = list[i];
        } else if(foundNeg && list[i] < 0 && negNum < list[i]) {
            negNum = list[i];
        }
    }
    return negNum;
}

A: If you need the greatest negative number then sort array thin search for first negative number
import java.util.Arrays;

class Main {

    public static void main(String[] args) {
        int arr[] = { 2, 4, 1, 7,2,-3,5,-20,-4,5,-9};
        System.out.println(greatestNegative(arr));
    }

    private static int greatestNegative(int[] arr) {
        Arrays.sort(arr);
        for (int i = arr.length - 1; i >= 0; i--) {
            if (isNegative (arr[i])) {
                return arr[i];
            }
        }
        return 0;
    }

    private static boolean isNegative (int i) {
        return i < 0;
    }
}

Output : -3

A: Please check following code, which will
first calculate small number from  array,
then check is it positive? if yes return 0 else return negative.
public static int greatestNegative(int[] list)
{
    int negativeNumbers = Integer.MAX_VALUE;
    for (int i = 0; i < list.length; i++) {
        if (list[i] < negativeNumbers)
                negativeNumbers  = list[i];
    }

    if(negativeNumbers  >=0)
         return 0;
    else
         return negativeNumbers;

}

A: Start by setting your "maxNegative" value to 0.  Then assign the first negative number you come across.  After that, only assign negative numbers that are higher.  If there are no negative numbers, then your "maxNegative" will still be zero.
public static void main(String[] args) {
    int arr[] = {2, -1, 4, 1, 0, 7, 2, -3, 5, 9, -4, 5, -9};
    int maxNegative = 0;
    for (int i = 0; i < arr.length; i++) {
        if (maxNegative == 0 && arr[i] < maxNegative) {
            // Set the first negative number you come across
            maxNegative = arr[i];
        } else if (maxNegative < arr[i] && arr[i] < 0) {
            // Set greater negative numbers
            maxNegative = arr[i];
        }
    }
    System.out.println(maxNegative);
}

Results:
-1

Java 8
Then there are streams, that allow you to do this with one line of code.
public static void main(String[] args) {
    int arr[] = {2, 4, 1, 0, 7, 2, -3, 5, 9, -4, 5, -9};
    int maxNegative = Arrays.stream(arr).filter(a -> a < 0).max().orElse(0);
    System.out.println(maxNegative);
}

Results:
-3

redpajama_set_name: RedPajamaStackExchange


text:
Q: Edit magento 2 order 'print' layout So when u go to My Account -> My Orders  and click on view order and then click Print Order then u see a PDF of the order that u just placed. Mine looks all messy :

So I want to edit this.
Does someone know how to? It's not in the email templates section as far as I can see.
Thanks!


text:
What does self-guilt mean?
Definitions for self-guilt
self-guilt
Here are all the possible meanings and translations of the word self-guilt.
Editors Contribution(5.00 / 3 votes)Rate this definition:
A feeling of responsibility or remorse for some offense, crime, wrong, etc., whether real or imagined.
I'm so self-guilt after leaving her.
Submitted by anonymous on July 12, 2019
Guilt for one's own actions.
I felt so self-guilt for leaving my family while moving out.
How to pronounce self-guilt?
How to say self-guilt in sign language?
Chaldean Numerology
The numerical value of self-guilt in Chaldean Numerology is: 9
Pythagorean Numerology
The numerical value of self-guilt in Pythagorean Numerology is: 3
Images & Illustrations of self-guilt
Find a translation for the self-guilt definition in other languages:
Discuss these self-guilt definitions with the community:
Would you like us to send you a FREE new word definition delivered to your inbox daily?
"self-guilt." Definitions.net. STANDS4 LLC, 2019. Web. 17 Jul 2019. <https://www.definitions.net/definition/self-guilt>.
Are we missing a good definition for self-guilt? Don't keep it to yourself...
Submit Definition
The fingerspelling provided here is most commonly used for proper names of people and places; it is also used in some languages for concepts for which no sign is available at that moment.
There are obviously specific signs for many words available in sign language that are more appropriate for daily usage.
self-governing colony
self-gratification
self-gratulation
self-harmer
self-hating
self-hatred
self-heal
Alternative searches for self-guilt:
Search for Synonyms for self-guilt
Search for Anagrams for self-guilt
Quotes containing the term self-guilt
Search for Phrases containing the term self-guilt
Search for Poems containing the term self-guilt
Search for Scripts containing the term self-guilt
Search for Abbreviations containing the term self-guilt
What rhymes with self-guilt?
Search for Song lyrics containing the word self-guilt
Search for self-guilt on Amazon
Search for self-guilt on Google

redpajama_set_name: RedPajamaCommonCrawl


text:
Racing Broken Mast Costs TNZ Race 4
Broken Mast Costs TNZ Race 4
By Stuart Streuli
Stuart Streuli
After a 10-day layoff and a significant crew change, Team New Zealand hoped Race 4 would be a new beginning in their star-crossed attempt to defend the America's Cup. But instead of things getting better, they got significantly worse for skipper Dean Barker and his crew as they lost their rig on the second beat and now trail Alinghi by a daunting 4-0 score in the best-of-nine series.
After three days of strong southeasterly and easterly breezes, the Hauraki Gulf was rougher than it had been during any of the first three races, with swells reaching six feet. While the breeze did peak at 26 knots soon after the boats rounded the leeward mark, it was a pair of monster waves that did in Team New Zealand. The starboard shroud parted at the second spreader and the mast broke 25 feet off the deck. According to syndicate head Tom Schnackenberg, the tip cup, which ties together the discontinuous rod rigging used on America's Cup Class boats, on the second windward spreader failed and caused the rig to come down.
Since there is no structural damage to the rest of the boat–something Schnackenberg confirmed at the post-race press conference–the team's shore crew shouldn't have much trouble replacing the rig with the one from NZL-81 and getting the boat ready for Race 5, which is still scheduled for Saturday. Whether the team's morale can be repaired as quickly is probably the more pressing question. Team New Zealand came into this regatta brimming with confidence, both in its boat and its crew. Neither has proven itself up to the task in the first four races and the team's confidence has taken a beating. "We certainly didn't expect to be racing in conditions like that [26 to 30 knots] during the America's Cup," said Barker, when asked whether the team pushed hard enough during training. "It's always hard with new boats knowing how much to push them."
| Stuart Streuli|
| Alinghi nailed the start, carrying a half-boatlength lead off the line, and was able to turn that into an eight-second advantage by the first windward mark* * *|
While the race was officially won when Team New Zealand retired with the broken rig, things started going wrong for Team New Zealand before the five minute gun sounded as the foredeck crew struggled to get the jib up in time and Barker was 12 seconds late entering the starting box. The delay allowed Alinghi skipper Russell Coutts, who started at the pin, to dive low, cross in front, and claim the starboard advantage. The rest of the pre-start was decidedly dull as the two boats never came close together, both seemingly happy to avoid any close confrontations in the sloppy sea. Alinghi lead TNZ back toward the line on starboard and hit the middle of the line just as the gun sounded with Barker a few seconds late. Alinghi was able to build that lead up to 50 meters and hold that advantage for the first leg, rounding the first mark eight seconds ahead.
With both boats surging down the rolling waves–they averaged 13.4 knots for the run–the lead varied quite a bit on the second leg, going from as much as 80 meters to less than 40. In the end the run produced a nine-second gain for Alinghi. As the boats headed upwind a rain squall moved through and the wind picked up from the high teens into the mid 20s. Team New Zealand once again appeared to be shipping an inordinate amount of water in the heaviest puffs, but when the rig fell down the wind was starting to ease and the sun breaking through the clouds. By the time Alinghi finished the race the wind had dropped to less than 15 knots.


text:
\section{Discussion and Conclusion}
\vspace{-10pt}
\label{sec:conclusion}

\looseness=-1
In this paper, we define a simple yet effective method \method{} and the KKT knowledge transfer module that leverages sparsity to learning tasks while resolving the shortcomings of supermasks by performing Exclusive Subnetwork Training. \method{} improves performance, prevents forgetting, accumulates and transfers knowledge across tasks, and works for both vision and language domains.

\section*{Acknowledgements}
We would like to thank Marc'Aurelio Ranzato, Xiang Zhou, Swarnadeep Saha, and Archiki Prasad for their helpful discussions.
This work was supported by ARO Award W911NF2110220, NSF-CAREER Award 1846185, DARPA Machine-Commonsense (MCS) Grant N66001-19-2-4031. The views contained in this article are those of the authors and not of the funding agency.

\section{Appendix for \method{}}
\label{sec:appendix}

\subsection{Additional Experiments and Details}

\begin{table}[tbh!]
\centering
\small

\begin{minipage}{0.48\textwidth}
    \resizebox{0.95\linewidth}{!}{
    \begin{tabular}{cl}
    \toprule
    \textbf{ID} &  \textbf{Task Sequence} \\
    \midrule

    S1 & mnli \Arrow{.2cm} qqp \Arrow{.2cm} qnli \Arrow{.2cm} sst2 \Arrow{.2cm} cola  (Dec. data Size) \\
    \midrule
    S2 & ag \Arrow{.2cm} yelp \Arrow{.2cm} amazon \Arrow{.2cm} yahoo \Arrow{.2cm} dbpedia \\
    S3 & yelp \Arrow{.2cm} yahoo \Arrow{.2cm} amazon \Arrow{.2cm} dbpedia \Arrow{.2cm} ag \\
    S4 & dbpedia \Arrow{.2cm} yahoo \Arrow{.2cm} ag \Arrow{.2cm} amazon \Arrow{.2cm} yelp \\
    S5 & yelp \Arrow{.2cm} ag \Arrow{.2cm} dbpedia \Arrow{.2cm} amazon \Arrow{.2cm} yahoo\\
    S6 & ag \Arrow{.2cm} yelp \Arrow{.2cm} yahoo \\
    S7 & yelp \Arrow{.2cm} yahoo \Arrow{.2cm} ag \\
    S8 & yahoo \Arrow{.2cm} ag \Arrow{.2cm} yelp \\

    \bottomrule
    \end{tabular}
    }
    \captionsetup{margin=0.25cm}
    \captionof{table}{\label{tab:task-order}Task sequences used in text experiments. For the GLUE dataset, we use order corresponding to decreasing train data size. Sequence S2-S8 are from \citep{huang-etal-2021-continual-idbr,d2019episodic,sun2019lamol}.}

\end{minipage}
\begin{minipage}{0.48\textwidth}
    \centering
    \resizebox{0.95\linewidth}{!}{
    \begin{tabular}{cccccc}
    \toprule
    \textbf{Dataset}  & \textbf{Class} & \textbf{Type} & \textbf{Train}& \textbf{Validation}& \textbf{Test} \\
    \midrule
    AGNews  & 4 & News & 8k & 8k & 7.6k  \\
    Yelp & 5 & Sentiment& 10k & 10k & 7.6k \\
    Amazon & 5 & Sentiment & 10k &  10k & 7.6k \\
    DBPedia & 14 & Wikipedia & 28k &  28k & 7.6k \\
    Yahoo & 10 & Q\&A & 20k &  20k & 7.6k \\
    \bottomrule
    \end{tabular}
    }
    \caption{\label{app:tab:sampled-stats}
    Statistics for sampled data used from \citet{huang-etal-2021-continual-idbr} for hyperparameter tuning. The validation set is the same size as the train set. Class means the number of output classes for the text classification task. Type is the domain of text classification.
    }

\end{minipage}
\end{table}

\subsubsection{Experimental setup and hyperparameters}
\label{sec:additional-details}

\textbf{Implementation Details:} Unless otherwise specified, we obtain supermasks with a mask density of 0.1. In our CNN models, we use non-affine batch normalization to avoid storing their means and variance parameters for all tasks \citep{supsup}. Similar to \citep{supsup}, bias terms in our model are 0 and we randomly initialize the model parameters using \textit{signed kaiming constant} \citep{ramanujan2019s}.
We use Adam optimizer \citep{kingma2014adam} along with cosine decay \citep{cosine} and conduct our experiments on GPUs with 12GB of memory. For our main experiment, we run three independent runs for each experiment and report the averages for all the metrics and experiments. For natural language tasks, unless specified otherwise we initialize the token embedding for our methods using a frozen BERT-base-uncased \citep{devlin2018bert} model's representations using Huggingface \citep{wolf2020transformers}. We use a static CNN model from \citet{kim-2014-convolutional} as our text classifier over BERT representations. The model employs 1D convolutions along with \textit{Tanh} activation. Following \citet{sun2019lamol,huang-etal-2021-continual-idbr}, we evaluate our model on various task sequences as provided in Appendix Table \ref{tab:task-order}, while limiting the maximum number of tokens to 256. Following \citep{supsup}, we use LeNet \citep{lecun1998lenet} for SplitMNIST dataset, a Resnet-18 model with fewer channels \citep{supsup} for SplitCIFAR100 dataset, a ResNet50 model \citep{He2016resnet} for TinyImageNet dataset. Unless specified, we randomly split all the vision datasets to obtain five tasks with disjoint classes. We use the codebase of DER \citep{buzzega2020der} to obtain the vision baselines. In all our experiments, all methods perform an equal number of epochs over the datasets.

For the ablation experiment on natural language data, following \citet{huang-etal-2021-continual-idbr}, we use a sampled version of the WebNLP datasets due to limited resources. The reduced dataset contains 2000 training and validation examples from each output class. The test set is the same as the main experiments. The dataset statistics are summarized in Table \ref{app:tab:sampled-stats}. For WebNLP datasets, we tune the learning rate on the validation set across the values $\{0.01, 0.001, 0.0001\}$, for GLUE datasets we use the default learning rate of the BERT model. For our vision experiments, we use the default learning rate for the dataset provided in their original implementation. For TinyImageNet, SplitCIFAR100, SplitMNIST dataset, we run for 30, 100, and 30 epochs respectively. We store 0.1\% of our vision datasets for replay while for our language experiments we use 0.01\% of the data because of the large number of datasets available for them.

\begin{table}[t]
\begin{minipage}{0.5\linewidth}

\centering
    \captionsetup{margin=0.25cm}
    \captionof{table}{\label{tab:overlap} We report the \textcolor{BrickRed}{average sparse overlap} for all method and dataset combinations reported in Table \ref{tab:knn-results}.}
    \resizebox{0.9\linewidth}{!}{
    \begin{tabular}{clll}
        \toprule
        \textbf{Method} & \textbf{S-MNIST} &  \textbf{S-CIFAR100} & \textbf{S-TinyImageNet} \\
        \midrule
        \textbf{SupSup} & 22.6 & 18.9 & 18.1 \\
        \textbf{+ KKT} & 46.4 & 48.3 & 52.4 \\
        \midrule
        \textbf{\modelorg{}} & 22.5 & 17.6 & 18.6 \\
        \textbf{+ KKT} & 52.7 & 49.9 & 52.4\\
        \midrule
        \textbf{\method{}} & 22.5 & 17.3 & 18.5 \\
        \textbf{+ KKT} & 47.8 & 48.8 & 52.4 \\
        \bottomrule
    \end{tabular}
    }
\end{minipage}
\begin{minipage}{0.5\linewidth}
\centering
\includegraphics[width=0.9\textwidth]{figures/cifar_avgacc.pdf}
\vspace{5pt}
\captionof{figure}{\label{fig:acc_evolution}Average Accuracy of all seen tasks as a function of the number of learned classes for the Split-CIFAR100 dataset.}
\end{minipage}
\end{table}

\subsubsection{Sparse Overlap Numbers}
In Table \ref{tab:overlap}, we report the sparse overlap numbers for SupSup, \modelorg{}, and \method{} with and without the KKT knowledge transfer module. This table corresponds to the results in main paper Table \ref{tab:knn-results}.

\subsubsection{Average Accuracy Evolution}
\label{sec:acc-evolution}

In Figure \ref{fig:acc_evolution}, we plot $\sum_{i \leq t} A_{ti}$ vs $t$, that is the average accuracy as a function of observed classes. This plot corresponds to the SplitCIFAR100 results provided in the main paper Table \ref{tab:vision-main}. We can observe from these results that Supsup and ExSSNeT performance does not degrade when we learn new tasks leading to a very stable curve whereas for other methods the performance degrades as we learn new tasks indicating some degree of forgetting.

\vspace{15pt}
\begin{table}[tbh!]
\centering
\large
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{ccccccccc}
\toprule
\textbf{Model ($\downarrow$)} &  \multicolumn{4}{c}{\textbf{Length-5 WebNLP}} & \multicolumn{4}{c}{\textbf{Length-3 WebNLP}}\\
\cmidrule(lr){2-5} \cmidrule(lr){6-9}
\textbf{Order ($\rightarrow$)} & \textbf{S2} & \textbf{S3} & \textbf{S4} & \textbf{Average}  & \textbf{S6} & \textbf{S7} & \textbf{S8} & \textbf{Average} \\
\midrule
\textbf{\textit{Random}} & \textit{7.14} & \textit{7.14} & \textit{7.14} & \textit{7.14} & \textit{10.0} & \textit{10.0} & \textit{10.0} & \textit{10.0} \\
\textbf{\textit{MTL}} & \textit{75.09} & \textit{75.09} & \textit{75.09} & \textit{75.09} & \textit{74.16} & \textit{74.16} & \textit{74.16} & \textit{74.16} \\
\midrule
\textbf{Finetune}$\dagger$   & 32.37 & 32.22 & 26.44 & 30.34 & 25.79 & 36.56 & 41.01 & 34.45  \\
\textbf{Replay}$\dagger$ & 68.25 & 70.52 & 70.24 & 69.67 & 69.32 & 70.25 & 71.31 & 70.29  \\
\textbf{Regularization}$\dagger$ & 72.28 & 73.03 & 72.92 & 72.74 & 71.50 & 70.88 & 72.93 & 71.77  \\
\textbf{AdaptBERT}  & 30.49 & 20.16 & 23.01 & 24.55 & 24.48 & 31.08 & 26.67 & 27.41  \\
\textbf{AdaptBERT + Replay} & 69.30 & 67.91 & 71.98 & 69.73 & 66.12 & 69.15 & 71.62 & 68.96  \\
\textbf{IDBR}$\dagger$   &  72.63 & 73.72 & 73.23 & 73.19 & 71.80 & 72.72 &  73.08 & 72.53  \\
\textbf{SupSup} & 74.01 & 74.04 & 74.18 & 74.08 & 72.01 & 72.35 & 72.53 & 72.29 \\
\midrule
\textbf{\modelorg{}} & 74.5 & 74.5 & 74.65 & 74.55 & 73.1 & 72.92 & 73.07 & 73.03 \\
\textbf{\method{}} & \textbf{74.78} & \textbf{74.72} & \textbf{74.71} & \textbf{74.73} & \textbf{72.67} & \textbf{72.99} & \textbf{73.24} & \textbf{72.97} \\
\bottomrule
\end{tabular}
}
\captionsetup{margin=0.25cm}
\captionof{table}{\label{tab:text-seq-len}Average test accuracy reported over task sequences for three independent runs on sub-sampled data. Results with $\dagger$ are taken from \citet{huang-etal-2021-continual-idbr}.}
\end{table}

\subsubsection{Effect of Task Order and Number of Tasks}
\label{sec:task-order}

Following \citet{huang-etal-2021-continual-idbr}, we conduct experiments to study the effect of task length and order in the language domain. We use task sequences of lengths three and five, with multiple different task orders on the sampled data (Section \ref{sec:setup}, Table \ref{tab:task-order}, and Appendix) to characterize the impact of these variables on the performance. In Table \ref{tab:text-seq-len}, we present the average test accuracy averaged over three different random seeds. We observe that across all six different settings our method performs better compared to all the baseline methods. Our methods bridge the gap toward multitask methods' performance, leaving a gap of  0.36\% and 1.19\% for lengths three and five sequences, respectively.

\begin{table}[t]
\begin{minipage}{0.4\linewidth}
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{cc}
    \toprule
    \textbf{Method} & \textbf{Runtime (in minutes)} \\
    \midrule
    \textbf{\textit{Multitask}} & 200 \\
    \textbf{Finetune} & 175 \\
    \textbf{Replay} & 204 \\
    \textbf{AdapterBERT + FT} & 170 \\
    \textbf{AdapterBERT + Replay} & 173 \\
    \textbf{MultiAdaptBERT} & 170 \\
    \textbf{Regularization} &  257 \\
    \textbf{IDBR} & 258 \\
    \textbf{SupSup} & \textbf{117} \\
    \midrule
    \textbf{\modelorg{}} & \textbf{117} \\
    \textbf{\method{}} & \textbf{117} \\
    \bottomrule
    \end{tabular}
    }
    \caption{\label{app:tab:runtime}Runtime comparison of different methods used in the text experiments.}

\end{minipage}
\begin{minipage}{0.6\linewidth}
    \centering
    \footnotesize
    \vspace{5pt}
    \resizebox{0.9\linewidth}{!}{
    \begin{tabular}{llllllllll}
    \toprule
    \textbf{Method ($\downarrow$)} & \multicolumn{1}{c}{\textbf{GLUE}}  & \multicolumn{5}{c}{\textbf{WebNLP}} \\
    \cmidrule(lr){2-2} \cmidrule(lr){3-7}
    \textbf{Order ($\rightarrow$)}& \textbf{S1} & \textbf{S2} & \textbf{S3} & \textbf{S4} & \textbf{S5} & \textbf{Average}\\
    \midrule
    \textbf{\textit{Random}} & \textit{33.3} & \textit{7.14} & \textit{7.14} & \textit{7.14} & \textit{7.14} & \textit{7.14} \\
    \textbf{\textit{Multitask}} & \textit{80.6} & \textit{77.4} & \textit{77.5} & \textit{76.9} & \textit{76.8} & \textit{77.1} \\
    \midrule
    \textbf{FT} & 14.0 & 27.0 & 22.9 & 30.4 & 15.6 & 24.0 \\
    \textbf{Replay} & 79.7 & 75.2 & 74.5 & 75.2 & 75.5 & 75.1 \\
    \textbf{AdaptBERT + FT} & 25.1 & 20.8 & 19.1 & 23.6 & 14.6 & 19.5 \\
    \textbf{AdaptBERT + Replay} & 78.6 & 73.3 & 74.3 & 74.7 & 74.6 & 74.2 \\
    \textbf{MultiAdaptBERT} & 83.6 & 76.7 & 76.7 & 76.7 & 76.7 & 76.7\\
    \textbf{Regularization} & 75.5 & 75.9 & 75.0 & 76.5 & 76.3 & 75.9 \\
    \textbf{IDBR} & 77.5 & 75.8 & 75.4 & 76.4 & 76.4 & 76.0 \\
    \textbf{SupSup} & 78.1 & 75.7 & 76.0 & 76.0 & 75.9 & 75.9 \\
    \midrule
    \textbf{\modelorg{}} & 77.2 & 76.3 & 76.3 & 77.0 & 76.1 & 76.4 \\
    \textbf{\method{}} & 80.1 & 77.1 & 77.3 & 77.2 & 77.1 & 77.2 \\

    \bottomrule
    \end{tabular}
    }
    \captionof{table}{\label{app:tab:text-main-val} Average validation accuracy ($\uparrow$) for multiple tasks and sequence orders with previous state-of-the-art (SotA) methods.}

\end{minipage}

\end{table}

\subsubsection{Runtime Comparison across methods}
\label{sec:runtime}

In this Section, we provide the result to compare the runtime of various methods used in the paper. We ran each method on the sampled version of the WebNLP dataset for the \textit{S2} task order as defined in Table \ref{tab:task-order}. We report the runtime of methods for four epochs over each dataset in Table \ref{app:tab:runtime}. Note that the masking-based method, SupSup, \modelorg{}, \method{} takes much lower time because they are not updating the BERT parameters and are just finding a mask over a much smaller CNN-based classification model using pretrained representation from BERT. This gives our method an inherent advantage that we are able to improve performance but with significantly lower runtime while learning a mask over much fewer parameters for the natural language setting.

\subsubsection{Validation results}
\label{sec:val-results}
In Table \ref{app:tab:text-main-val}, we provide the average validation accuracies for the main natural language results presented in Table \ref{tab:text-main}. We do not provide the validation results of LAMOL \citep{sun2019lamol} and MBPA++ \citep{d2019episodic} as we used the results provided in their original papers. For the vision domain, we did not use a validation set because no hyperparameter tuning was performed as we used the experimental setting and default parameters from the original source code from \citep{supsup,wen2020batchensemble}.

\subsection{Additional Model Details}

\subsubsection{Algorithm for \method{}}
\label{sec:exessnet-algorithm}

In Algorithm \ref{alg:training}, we provide a pseudo-code for our method \method{} for easier reference and understanding. We also attach our working code as supplementary material to encourage reproducibility.

\begin{algorithm}[H]
\scriptsize
\caption{\label{alg:training} \method{} training procedure.}
\DontPrintSemicolon
\KwIn{Tasks $\mathcal{T}$, a model
$\mathcal{M}$, mask sparsity $k$, exclusive=True}
\KwOut{Trained model}
\Comment{\scriptsize Initialize model weights $W^{(0)}$}
initialize\_model\_weights($\mathcal{M}$)

\ForAll{$i \in range(|\mathcal{T}|)$}
{
     \Comment{ \scriptsize Set the mask $M_i$ corresponding to task $t_i$ for optimization.}
    mask\_opt\_params = $M_i$

    \Comment{\scriptsize Learn the supermask $M_i$ using edge-popup}
    \ForAll{$em \in \text{mask\_epochs}$}
    {
        $M_i$ = learn\_supermask(model, mask\_opt\_params, $t_i$)
    }

    \Comment{\scriptsize Model weight at this point are same as the last iteration $W^{(i-1)}$}
    \uIf{$i > 1$ and exclusive}{
        \Comment{\scriptsize Find mask for all the weights used by previous tasks.}
        $M_{1:i-1} = \lor_{j=1}^{i-1} (M_{j})$

        \Comment{\scriptsize Get mask for weights in $M_i$ which are not in $\{M_i\}_{j=1}^{i-1}$}
        $M_i^{free} = M_i \land \lnot M_{1:i-1}$

        \Comment{\scriptsize Find non-overlapping weight for updating.}
        $W^{(i)}_{free} = M_i^{free} \odot W^{(i-1)}$
    }\;

    \ElseIf{not exclusive}{
    $W^{(i)}_{free} = W^{(i-1)}$
    }
    weight\_opt\_params = $W^{(i)}_{free}$

    \Comment{\scriptsize Learn the free weight in the supermask $M_i$}
    \ForAll{$em \in \text{weight\_epochs}$}
    {
        $W^{(i)}$ = update\_weights(model, weight\_opt\_params, $t_i$)
    }
}
\end{algorithm}

\subsubsection{Model Diagram for Supsup}
\label{sec:supsup}

In Figure \ref{fig:supsup}, we provide the canonical model diagram for SupSup. Please read the figure description for more details regarding the distinctions between SupSup and ExSSNeT.

\begin{figure}[tbh]
    \centering
    \includegraphics[width=0.6\linewidth]{figures/SupSup.pdf}
    \captionof{figure}{This is a canonical model diagram for SupSup. In SupSup, the model weights are always fixed at the random initialization $W^{(0)}$. For each task SupSup learns a new mask (in this case $M_1, M_2, M_3$) over the weights $W^{(0)}$. A mask selectively activates a subset of weights for a particular task. This subset of selected weights forms a subnetwork inside the full model which we refer to as the supermask subnetwork. For example, when learning Task 2, SupSup learns the mask $M_2$ (the weights activated by the mask are highlighted in green) over the fixed weight $W^{(0)}$. These highlighted weights along with the participating nodes are the subnetwork formed by mask $M_2$. Whenever a prediction is made for Task 2 samples, this mask is selected and used to obtain the predictions. Please note that the model weights $W^{(0)}$ are never updated after their random initialization. Hence, for SupSup there is no learned knowledge sharing across tasks. This is in contrast to our setup in Figure \ref{fig:method}, where for the first task the mask is learned over the weights $W^{(0)}$ but once the mask is selected the weights of the corresponding subnetwork are also updates to obtain new weight $W^{(1)}$. Then the next task's mask is learned over these new set of weights $W^{(1)}$ and so on. Also note that in Figure \ref{fig:method}, we do not show the KKT knowledge transfer module here to avoid confusion.}
    \label{fig:supsup}
\end{figure}

\section{Experiments}
\label{sec:experiments}

\subsection{Experimental Setup and Training Details}
\label{sec:setup}

\textbf{Datasets:} For natural language domain, we follow the shared text classification setup of IDBR \citep{huang-etal-2021-continual-idbr}, LAMOL \citep{sun2019lamol}, and MBPA++ \citep{de2019continual} to sequentially learn five text classification tasks;
(1) Yelp Sentiment analysis \citep{zhang2015character};
(2) DBPedia for Wikipedia article classification \citep{dbpedia_dataset};
(3) Yahoo! Answer for Q\&A classification \citep{yahoo_dataset};
(4) Amazon sentiment analysis \citep{amazon_dataset};
(5) AG News for news classification \citep{zhang2015character}.
We call them WebNLP classification tasks for easier reference.
While comparing with the previous state-of-the-art text methods, we use the same training and test set as IDBR and LAMOL containing 115,000/500/7,600 Train/Val/Test examples. For our ablation studies, we follow IDBR and use a sampled dataset, please see Appendix Table \ref{app:tab:sampled-stats} for statistics. Additionally, we create a CL benchmark using the popular \textit{GLUE classification} tasks \citep{wang-etal-2018-glue} consisting of more than 5k train samples. We use the official validation split as test data and use $0.1\%$ of the train data to create a validation set. Our final benchmark includes five tasks; MNLI \textcolor{arsenic}{(353k/39k/9.8k)}, QQP \textcolor{arsenic}{(327k/36k/40k)}, QNLI \textcolor{arsenic}{(94k/10k/5.4k)}, SST-2 \textcolor{arsenic}{(60k/6.7k/872)}, CoLA \textcolor{arsenic}{(7.6k/856/1k)}.
For vision experiments, we follow SupSup and use three CL benchmarks,
SplitMNIST \citep{zenke2017continual}
SplitCIFAR100 \citep{chaudhry2018efficient}, and SplitTinyImageNet \citep{buzzega2020der} datasets with 10, 100 and 200 total classes respectively.

\looseness=-1
\textbf{Metrics:} We follow \citet{chaudhry2018efficient} and evaluate our model after learning task $t$ on all the tasks, denoted by $\mathcal{T}$. This gives us an accuracy matrix $A \in \mathbb{R}^{n \times n}$, where $a_{i,j}$ represents the classification accuracy on task $j$ after learning task $i$.
We want the model to perform well on all the tasks it has been learned. This is measured by the \textit{average accuracy}, $\mathcal{A}(\mathcal{T}) = \frac{1}{N} \sum_{k=1}^{N} a_{N,k}$, where $N$ is the number of tasks. Next, we want the model to retain performance on the previous tasks when learning multiple tasks. This is measured by the \textit{forgetting metric} \citep{lopez2017gradient}, $F(\mathcal{T}) = \frac{1}{N-1} \sum_{t=1}^{N-1} (\max_{k \in \{1, \hdots, N-1\}} a_{k,t} - a_{N,t})$. This is the average difference between the maximum accuracy obtained for task $t$ and its final accuracy. Higher accuracy and lower forgetting are desired.

\looseness=-1
\textbf{Sparse Overlap to Quantify Parameter Interference:}
Next, we propose a measure to quantify parameter interference for a task $i$, i.e., the fraction of the parameters in mask $M_i$ that are already updated by some previous task. We define \textit{sparse overlap} as the difference between the number of parameters selected by mask $M_i$ and $M^{free}_i$ divided by the total parameters selected by $M_i$.
Formally, we define \textit{sparse overlap} (SO) between current supermask $M_i$ and supermasks for previous tasks $\{M_j\}_{j=1}^{i-1}$ as,
\begin{equation}
    \small
    \text{SO}(M_i, \{M_j\}_{j=1}^{i-1}) = \frac{sum(M_i) - sum(M^{free}_i)}{sum(M_i)}~~;~~ M^{free}_i = M_i \land \lnot (\lor_{j=1}^{i-1} (M_{j}))
    \label{eqn:sp_overlap}
\end{equation}
where $\land, \lor, \lnot$ are logical \textit{and}, \textit{or}, and \textit{not} operations.

\begin{table*}[t!]
\centering
\large
\vspace{-5pt}
\resizebox{0.8\linewidth}{!}{
\begin{tabular}{llllllllll}
\toprule
\textbf{Method ($\downarrow$)} & \multicolumn{1}{c}{\textbf{GLUE}}  & \multicolumn{5}{c}{\textbf{WebNLP}} \\
\cmidrule(lr){2-2} \cmidrule(lr){3-7}
\textbf{Order ($\rightarrow$)} & \textbf{S1} & \textbf{S2} & \textbf{S3} & \textbf{S4} & \textbf{S5} & \textbf{Average}\\

\midrule

\rowcolor{mygray}
\textbf{\textit{Random}} & \textit{33.3 (-)} & \textit{7.14 (-)} & \textit{7.14 (-)} & \textit{7.14 (-)} & \textit{7.14 (-)} & \textit{7.14 (-)} \\
\rowcolor{mygray}
\textbf{\textit{Multitask}} & \textit{79.9 (0.0)} & \textit{77.2 (0.0)} & \textit{77.2 (0.0)} & \textit{77.2 (0.0)} & \textit{77.2 (0.0)} & \textit{77.2 (0.0)} \\
\rowcolor{mygray}
\textbf{\textit{Individual}} & \textit{87.7 (0.0)} & \textit{79.5 (0.0)} & \textit{79.5 (0.0)} & \textit{79.5 (0.0)} & \textit{79.5 (0.0)} & \textit{79.5 (0.0)} \\

\midrule

\textbf{FT} & 14.1 (86.0) & 26.9 (62.1) & 22.8 (67.6) & 30.6 (55.9) & 15.6 (76.8) & 24.0 (65.6)  \\
\textbf{AdaptBERT + FT} & 24.7 (53.4) & 20.8 (68.4) & 19.1 (70.9) & 23.6 (64.5) & 14.6 (76.0) & 19.6 (70.0) \\
\textbf{AdaptBERT + Replay} & 76.8 (3.8) & 73.2 (3.0) & 74.5 (2.0) & 74.5 (2.0) & 74.6 (2.0) & 74.2 (2.3) \\
\textbf{MultiAdaptBERT} & 78.5 (0.0) & 76.7 (0.0) & 76.7 (0.0) & 76.7 (0.0) & 76.7 (0.0) & 76.7 (0.0) \\
\textbf{Prompt Tuning} & 76.3 (0.0) & 66.3 (0.0) & 66.3 (0.0) & 66.3 (0.0) & 66.3 (0.0) & 66.3 (0.0) \\
\textbf{Regularization} & 72.5 (8.8) & 76.0 (2.8) & 74.9 (3.8) & 76.4 (1.8) & 76.5 (2.0) & 76.0 (2.6) \\
\textbf{Replay} & 77.7 (4.8) & 75.1 (3.1) & 74.6 (3.5) & 75.2 (2.2) & 75.7 (3.1) & 75.1 (3.0) \\
\textbf{MBPA++}$\dagger$ & - & 74.9 (-) & 73.1 (-) & 74.9 (-) & 74.1 (-) & 74.3 (-) \\
\textbf{LAMOL}$\dagger$ & - &  76.1 (-) & 76.1 (-) & \textbf{77.2 (-)} & 76.7 (-) & 76.5 (-)  \\
\textbf{IDBR} & 73.0 (6.8) & 75.9 (2.7) & 75.4 (3.5) & 76.5 (1.6) & 76.4 (1.9) & 76.0 (2.4) \\
\textbf{SupSup} & 78.3 (0.0) & 75.9 (0.0) & 76.1 (0.0) & 76.0 (0.0) & 75.9 (0.0) & 76.0 (0.0) \\
\midrule
\textbf{\modelorg{}} & 78.4 (3.6) & 76.3 (0.8) & 76.3 (0.8) & 76.4 (0.3) & 76.3 (0.3) & 76.3 (0.6)  \\
\textbf{\method{}} & \textbf{80.5 (0.0)} & \textbf{77.0 (0.0)} & \textbf{77.1 (0.0)} & 76.7 (0.0) & \textbf{76.9 (0.0)} & \textbf{76.9 (0.0)} \\

\bottomrule
\end{tabular}
}
\captionof{table}{\label{tab:text-main} Comparing average test accuracy $\uparrow$ \textcolor{BrickRed}{(\textbf{and forgetting metric} $\downarrow$}\textbf{)} for multiple tasks and sequence orders with state-of-the-art (SotA) methods. Results with $\dagger$ are taken from \citep{huang-etal-2021-continual-idbr}.
}
\vspace{-10pt}
\end{table*}

\looseness=-1
\textbf{Previous Methods and Baselines:}
For both vision and language (\textbf{VL}) tasks, we compare with:
\textbf{(VL.1) Naive Training/Finetuning} \citep{yogatama2019learning}: where for the vision domain we train all model parameters from scratch whereas for the language domain we finetune BERT model weights for each task sequentially.
\textbf{(VL.2) Experience Replay (ER)} \citep{d2019episodic}: we replay previous tasks examples when we train new tasks;
\textbf{(VL.3) Multitask Learning} \citep{Crawshaw2020MultiTaskLW}: where all the tasks are used jointly to train the model and have strong performance;
\textbf{(VL.4) Individual Models}: where we train a separate model for each task. This is considered an upper bound for CL;
\textbf{(VL.5) Supsup} \citep{supsup}.
For natural language (\textbf{L}), we further compare with the following methods:
\textbf{(L.6) Regularization} \citep{huang-etal-2021-continual-idbr}: Along with the Replay method, we regularize the hidden states of the BERT classifier with an L2 loss term;
We show three Adapter BERT \citep{pmlr-v97-houlsby19adapter} variants,
\textbf{(L.7) AdaptBERT + FT} where we have single adapter which is finetuned for all task;
\textbf{(L.8) AdaptBERT + ER} where a single adapter is finetuned with replay;
\textbf{(L.9) MultiAdaptBERT} where a separate adapter is finetuned for each task;
\textbf{(L.10) Prompt Tuning} \citep{li-liang-2021-prefix} that learns 50 different continuous prompt tokens for each task.
\textbf{(L.11) MBPA++} \citep{d2019episodic} perform replay with random examples during training and does local adaptation during inference to select replay example;
\textbf{(L.12) LAMOL} \citep{sun2019lamol} uses a language model to generate pseudo-samples for previous tasks for replay;
\textbf{(L.13) IDBR} \citep{huang-etal-2021-continual-idbr} disentangles hidden representations into generic and task-specific representations and regularizes them while also performing replay.
For vision task (\textbf{V}), we additionally compare with two popular regularization based methods, \textbf{(V.6) Online EWC} \citep{schwarz2018progress},
\textbf{(V.7) Synaptic Intelligence (SI)} \citep{zenke2017continual};
one knowledge distillation method,
\textbf{(V.8) Learning without Forgetting (LwF)} \citep{li2017learning},
three additional experience replay method,
\textbf{(V.9) AGEM} \citep{chaudhry2018efficient},
\textbf{(V.10) Dark Experience Replay (DER)} \citep{buzzega2020der},
\textbf{(V.11) DER++} \citep{buzzega2020der}.

\textbf{Implementation Details:}
We focus on the task incremental setting \citep{hsu20clscenario}
and unless otherwise specified, we obtain supermasks with a mask density of 0.1. For language tasks, unless specified otherwise we initialize the token embedding for our methods using a frozen BERT-base-uncased \citep{devlin2018bert} model's representations using Huggingface \citep{wolf2020transformers}. We use a static CNN model from \citet{kim-2014-convolutional} as our text classifier. Following \citet{sun2019lamol,huang-etal-2021-continual-idbr}, for WebNLP datasets we learn different task orders S1-S5\footnote{\label{foot:order}For example, in S2 order the model learns the task in this order, ag \Arrow{.2cm} yelp \Arrow{.2cm} amazon \Arrow{.2cm} yahoo \Arrow{.2cm} dbpedia} that are provided in Appendix Table \ref{tab:task-order}.
Following \citep{supsup}, we use LeNet \citep{lecun1998lenet} for SplitMNIST, a Resnet-18 model \citep{supsup} for SplitCIFAR100, a ResNet50 model \citep{He2016resnet} for TinyImageNet datasets. Unless specified, we randomly split all the vision datasets to obtain five tasks with disjoint classes. In all our experiments, all methods perform an equal number of epochs over the datasets. For additional details on implementation and hyperparameters refer to Appendix \ref{sec:additional-details}.

\subsection{Main Results}
\label{sec:results}

\begin{figure*}
\centering
  \begin{minipage}{.55\textwidth}
    \centering
    \large
    \resizebox{0.9\linewidth}{!}{
    \begin{tabular}{cccc}
    \toprule
    \textbf{Method} & \textbf{S-MNIST} & \textbf{S-CIFAR100} & \textbf{S-TinyImageNet} \\
    \midrule
    \rowcolor{mygray}
    \textbf{\textit{Multitask}} & \textit{96.5 (0.0)} & \textit{53.0 (0.0)} & \textit{45.9 (0.0)} \\
    \rowcolor{mygray}
    \textbf{\textit{Individual}} & \textit{99.7 (0.0)} & \textit{75.5 (0.0)} & \textit{53.7 (0.0)} \\
    \midrule
    \textbf{Naive Sequential} & 49.6 (25.0) & 19.3 (73.7) & 11.5 (43.9) \\
    \textbf{EWC}	& 96.1 (4.5) & 32.4 (60.5) & 20.5 (52.1) \\
    \textbf{SI}	& 99.2 (0.6) & 46.1 (47.8) & 19.5 (46.2) \\
    \textbf{LwF}	& 99.2 (0.8) & 29.5 (70.2) & 18.1 (56.5) \\
    \textbf{AGEM} & 98.3 (1.9) & 52.1 (42.0) & 21.6 (54.9) \\
    \textbf{ER} & 99.2 (0.6) & 60.1 (27.5) & 35.6 (36.0) \\
    \textbf{DER} & 98.9 (1.2) & 62.5 (28.4) & 35.9 (37.7) \\
    \textbf{DER++} & 98.3 (1.8) & 62.5 (27.5) & 36.2 (35.7) \\

    \textbf{SupSup} & 99.6 (0.0) & 62.1 (0.0) & 50.6 (0.0) \\
    \midrule
    \textbf{\modelorg{}} & \textbf{99.7 (0.0)} & 23.9 (54.4) & 49.6 (1.9) \\
    \textbf{\method{}} & \textbf{99.7 (0.0)} & \textbf{67.3 (0.0)} & \textbf{52.0 (0.0)} \\
    \bottomrule
    \end{tabular}
    }
    \captionsetup{margin=0.25cm}
    \captionof{table}{\label{tab:vision-main}Average accuracy $\uparrow$ \textcolor{BrickRed}{\textbf{(Forgetting metric} $\downarrow$\textbf{)}} on all tasks for vision. For our method we report the results are averaged over three random seeds.}
\end{minipage}
\begin{minipage}{.44\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/kkt_speed_w5.pdf}
    \captionof{figure}{We plot validation accuracy vs Epoch for \method{} and \method{} + KKT. We observe that KKT helps to learn the subsequent tasks faster and improves performance.
    }
    \label{fig:knn_speed}
\end{minipage}
\end{figure*}

\looseness=-1
\textbf{Q1. Does Supermask Subnetwork Training Help?}

In these experiments, we show that \method{} outperforms multiple strong baseline methods including SupSup. For our main language experiments in Table \ref{tab:text-main}, we sequentially learn multiple task orders, S1 - S5\textsuperscript{\ref{foot:order}} corresponding to the GLUE and WebNLP benchmarks. These task orders are listed in Appendix Table \ref{tab:task-order}. We report the average test accuracy (and forgetting in parentheses).
For natural language, we perform better than previous state-of-the-art CL methods in four out of five cases, across multiple task orders, and in aggregate. Specifically, on the GLUE benchmark, \method{} is at least $2.0$\% better than other methods while avoiding CF. Furthermore, \method{} either outperforms or is close to the performance of the multitasking baseline which is a strong baseline for CL methods.

\looseness=-1
For vision tasks, we split the MNIST, CIFAR100, and TinyImageNet datasets into \textit{five different tasks} with an equal number of disjoint classes and report results.
We report these results in Table \ref{tab:vision-main} and observe that \method{} leads to a $4.8$\% and $1.4$\% improvement over the strongest baseline for Split-CIFAR100 and Split-TinyImageNet datasets.
Furthermore, both \method{} and SupSup outperform the multitask baseline. Moreover, \method{} bridges the gap to individually trained models significantly, for TinyImageNet we reach within $1.7$\% of individual models' performance. The average sparse overlap of \method{} is $19.4$\% across all three datasets implying that there is a lot more capacity in the model. See appendix Table \ref{tab:overlap} for sparse overlap of other methods.

We note that past methods employ specific tricks like local adaptation in MBPA++, a generative replay module in LAMOL, and experience replay in AGEM, DER, and ER. In contrast, \method{} does not require replay and simply trains different subnetworks for tasks.

\begin{table}[t!]
    \begin{minipage}{.5\textwidth}
        \resizebox{1.0\linewidth}{!}{
        \begin{tabular}{clll}
            \toprule
            \textbf{Method} & \textbf{S-MNIST} &  \textbf{S-CIFAR100} & \textbf{S-TinyImageNet} \\
            \midrule
            \textbf{SupSup} & 99.6 & 62.1 & 50.6 \\
            \textbf{+ KKT} & 99.6 [+0.0] & \textbf{67.1} [\textcolor{color2}{+5.0}] & \textbf{53.3} [\textcolor{color2}{+2.7}] \\
            \midrule
            \textbf{\modelorg{}} & \textbf{99.7} & \textbf{23.9} & 49.6 \\
            \textbf{+ KKT} & 99.3 [\textcolor{BrickRed}{-0.4}] & 23.5 [\textcolor{BrickRed}{-0.4}] & \textbf{51.8} [\textcolor{color2}{+2.2}] \\
            \midrule
            \textbf{\method{}} & 99.7 & 67.3 & 52.0 \\
            \textbf{+ KKT} & 99.7 [+0.0] & \textbf{70.5} [\textcolor{color2}{+3.2}] & \textbf{54.0} [\textcolor{color2}{+2.0}] \\
            \bottomrule
        \end{tabular}
        }
        \vspace{-5pt}
        \captionsetup{margin=0.25cm}
        \captionof{table}{\label{tab:knn-results} We report average test accuracies $\uparrow$ \textcolor{color2}{\textbf{[and gains from KKT]}} when using the KKT knowledge sharing module for Vision datasets. The overall best method is highlighted in gray.
        }
    \end{minipage}
    \begin{minipage}{.5\textwidth}
         \centering
        \includegraphics[width=0.8\linewidth]{figures/sparsity_log.pdf}
        \vspace{-5pt}
        \captionsetup{margin=0.25cm}
        \captionof{figure}{\label{fig:sparsity}Average test accuracy versus mask density on SplitCIFAR100 dataset.}
    \end{minipage}
\end{table}

\textbf{Q2. Can KKT Knowledge Transfer Module Share Knowledge Effectively?}

In Section \ref{sec:kkt_module}, we presented our KKT module for enabling knowledge sharing across tasks. In Table \ref{tab:knn-results}, we show that adding the KKT module to \method{}, \modelorg{}, and SupSup improves performance on vision benchmarks. The experimental setting here is similar to Table \ref{tab:vision-main}. We observe across all methods and datasets that the KKT module improves average test accuracy. Specifically, for the Split-CIFAR100 dataset, the KKT module results in $5.0$\%, and $3.2$\% improvement for SupSup and \method{} respectively; while for Split-TinyImageNet, \method{} + KKT outperforms the individual models.
We observe a performance decline for \modelorg{} when using KKT because KKT promotes sharing of parameters across tasks which can lead to worse performance for \modelorg{}.
Furthermore, \method{} + KKT outperforms all other methods on both the Split-CIFAR100 and Split-TinyImageNet datasets.
For \method{} + KKT, the average sparse overlap is $49.6$\% across all three datasets (see appendix Table \ref{tab:overlap}). These results indicate that the benefits of weight training and the KKT module can be coupled together to further improve performance.

\textbf{Q3. Can KKT Knowledge Transfer Module Improve Learning Speed of Subsequent Tasks?}

\looseness=-1
Next, we show that the KKT module enables us to learn new tasks faster. To demonstrate this, in Figure \ref{fig:knn_speed} we plot the running mean of the validation accuracy vs epochs for different tasks from the Split-CIFAR100 experiment in Table \ref{tab:knn-results}. We show curves for \method{} with and without the KKT module and omit the first task as both these methods are identical for Task 1 because there is no previous task to transfer knowledge.
For all the subsequent tasks (Task 2,3,4,5), we observe that -- (1) \method{} + KKT starts off with a much better initial performance compared to \method; (2) given a fixed number of epochs for training, \method{} + KKT always learns the task better because it has a better accuracy at all epochs; and (3) \method{} + KKT can achieve similar performance as \method{} in much fewer epochs as shown by the green horizontal arrows.
This clearly illustrates that using the KKT knowledge-transfer module not only helps to learn the tasks better (see Table \ref{tab:knn-results}) but it also learns them faster.

\subsection{Additional Results and Analysis}

\textbf{Q4. Effect of Mask Density on Performance:} Next, we show the advantage of using \method{} when the mask density is low. In Figure \ref{fig:sparsity}, we show the average accuracy for the Split-CIFAR100 dataset as a function of mask density. We observe that \method{} obtains $7.9$\%, $18.4$\%, $8.4$\%, and $4.7$\% improvement over SupSup for mask density values $0.02, 0.04, 0.06, 0.08$ respectively.
This is a highly desirable property as tasks select fewer parameters which inherently reduces sparse overlap allowing \method{} to learn a large number of tasks.

\looseness=-1
\textbf{Q5. Can \method{} Learn a Large Number of Tasks?}
SupSup showed that it can scale to a large number of tasks. Next, we show that this property is preserved by \method{}. We perform experiments to learn 100 tasks created by splitting the TinyImageNet dataset. Note that, as the number of task increase, the sparse overlap between the masks also increases resulting in fewer free model weights. In the extreme case where there are no free weights, \method{} by design reduces to SupSup because there will be no weight training. From Table \ref{tab:large-exp}, we conclude that \method{} can learn 100 tasks while still improving performance over SupSup and preventing forgetting.

\begin{table}[t!]
    \begin{minipage}{.5\textwidth}
        \centering
        \scriptsize
        \resizebox{0.9\linewidth}{!}{
        \begin{tabular}{ccc}
                \toprule
                \textbf{Method} & \textbf{S-TinyImageNet} & \textbf{Avg. Sparse Overlap} \\
                \midrule
                \textbf{SupSup} & 90.34 (0.0) & 90.1 \\
                \textbf{\modelorg{}} & 89.02 (2.2) & 90.0 \\
                \textbf{\method{}} & \textbf{91.21 (0.0)} & 90.0 \\
                \bottomrule
            \end{tabular}
        }
        \vspace{-5pt}
        \captionsetup{margin=0.25cm}
        \captionof{table}{\label{tab:large-exp}
        Average accuracy $\uparrow$ \textcolor{BrickRed}{(\textbf{forgetting metric} $\downarrow$)} and average sparse overlap when learning 100 tasks.}

    \end{minipage}
    \begin{minipage}{.5\textwidth}

        \centering
        \resizebox{0.9\linewidth}{!}{
        \begin{tabular}{clll}
        \toprule
        \textbf{Method} & \textbf{FastText} & \textbf{Glove} & \textbf{BERT}\\
        \midrule
        \textbf{SupSup} & 54.01 & 55.52 & 74.0 \\
        \textbf{\modelorg{}} & 60.41 [\textcolor{color2}{+6.4}] & 59.78 [\textcolor{color2}{+4.3}] & 74.5 [\textcolor{color2}{+0.5}] \\
        \textbf{\method{}} & \textbf{62.52} [\textcolor{color2}{+8.5}] & \textbf{62.81} [\textcolor{color2}{+7.3}] & \textbf{74.8} [\textcolor{color2}{+0.8}]\\
        \bottomrule
        \end{tabular}
        }
        \vspace{-5pt}
        \captionsetup{margin=0.25cm}
        \captionof{table}{\label{tab:text-ablation-init} Ablation result for token embeddings. We report average accuracy $\uparrow$ \textcolor{color2}{\textbf{[and gains over SupSup]}}}
    \end{minipage}
\end{table}

\looseness=-1
\textbf{Q6. Effect of Token Embedding Initialization for NLP:}
For our language experiments, we use a pretrained BERT model \citep{devlin-etal-2019-bert} to obtain the initial token representations. Using the powerful representations from a multitask model like BERT overshadows the superiority of \method{} over SupSup in terms of performance. Hence, we perform ablations on the token embedding initialization to further highlight these differences.
In Table \ref{tab:text-ablation-init}, we present the average test accuracy on the S2\textsuperscript{\ref{foot:order}} task-order sequence of the sampled version of WebNLP dataset (see Section \ref{sec:setup}, Datasets). We initialize the token representations using \textit{FastText} \citep{bojanowski2016enriching-fasttext}, \textit{Glove} \citep{pennington-etal-2014-glove}, and \textit{BERT} embeddings.
From Table \ref{tab:text-ablation-init}, we observe that -- (1) the performance gap between \method{} and SupSup increases from $0.8\% \rightarrow 7.3\%$ and $0.8\% \rightarrow 8.5\%$ when moving from BERT to Glove and FastText initializations respectively.
These gains imply that it is even more beneficial to use \method{} in absence of good initial representations, and (2) the performance trend, \method{} > \modelorg{} > SupSup is consistent across initialization.

\section{Introduction}
\label{sec:intro}
In artificial intelligence, the overarching goal is to develop autonomous agents that can learn to accomplish a set of tasks. Continual Learning (CL) \citep{ring1998child,thrun1998lifelong,kirkpatrick2017overcoming} is a key ingredient for developing agents that can accumulate expertise on new tasks.
However, when a model is sequentially trained on tasks $t_1$ and $t_2$ with different data distributions, the model's ability to extract meaningful features for the previous task $t_1$ degrades. This loss in performance on the previously learned tasks, is referred to as \textit{catastrophic forgetting} (CF) \citep{mccloskey1989catastrophic,zhao1996incremental,thrun1998lifelong,goodfellow2013empirical}.
Forgetting is a consequence of two phenomena happening in conjunction: (1) not having access to the data samples from the previous tasks, and (2) multiple tasks sequentially updating shared model parameters resulting in conflicting updates, which is called as \emph{parameter interference} \citep{mccloskey1989catastrophic}.

\looseness=-1
Recently, some CL methods avoid parameter interference by taking inspiration from the \textit{Lottery Ticket Hypothesis} \citep{frankle2018lottery} and \textit{Supermasks} \citep{zhou2019deconstructing} to exploit the expressive power of sparse subnetworks.
\citet{zhou2019deconstructing} observed that the number of sparse subnetwork combinations is large enough (combinatorial) that even within randomly weighted neural networks, there exist \textit{supermasks} that achieve good performance. A supermask is a sparse binary mask that selectively keeps or removes each connection in a fixed and randomly initialized network to produce a subnetwork with good performance on a given task. We call this the subnetwork as \textit{supermask subnetwork} that is shown in Figure \ref{fig:method}, highlighted in red weights.
Building upon this idea, \citet{supsup} proposed a CL method, \textit{SupSup}, which initializes a network with fixed and random weights and then learns a different supermask for each new task.
This allows them to prevent catastrophic forgetting (CF) as there is no parameter interference (because the model weights are fixed).

\begin{figure}
    \centering
    \includegraphics[scale=0.5]{figures/main_horizontal1.pdf}
    \captionof{figure}{\label{fig:method}\method{} diagram.
    We start with random weights $W^{(0)}$. For task 1, we first learn a supermask $M_1$ (the corresponding subnetwork is marked by red color, column 2 row 1) and then train the weight corresponding to $M_1$ resulting in weights $W^{(1)}$ (bold red lines, column 1 row 2). Similarly, for task 2, we learn the mask $M_2$ over fixed weights $W^{(1)}$. If mask $M_2$ weights overlap with $M_1$ (marked by bold dashed green lines in column 3 row 1), then only the non-overlapping weights (solid green lines) of the task 2 subnetwork are updated (as shown by bold and solid green lines column 3 row 2). These already trained weights (bold lines) are not updated by any subsequent task. Finally, for task 3, we learn the mask $M_3$ (blue lines) and update the solid blue weights.
    }
    \vspace{-15pt}
\end{figure}

Although SupSup \citep{supsup} prevents CF, there are some problems to using supermasks for CL:
(1) Fixed random model weights in SupSup limits the supermask subnetwork's representational power resulting in sub-optimal performance.
As shown in Figure \ref{fig:single_task}, for a single task the test accuracy of SupSup is approximately $10$\% worse compared to a \textit{fully trained model} where all model weights are updated.
As a possible remedy, one could try to naively train the weights corresponding to supermask subnetworks of different tasks; however, it can lead to CF as shown in Figure \ref{fig:sp_overlap}. This happens because subnetworks for different tasks can overlap and training subnetworks weights might result in parameter interference.
(2) When learning a task, there is no mechanism for transferring knowledge from previously learned tasks to better learn the current task. Moreover, the model is not accumulating knowledge over time as the weights are not being updated.

\looseness=-1
We overcome the aforementioned issues, we propose our method, \method{} (\textsc{E}xclusive \textsc{S}upermask \textsc{S}ub\textsc{Ne}twork \textsc{T}raining), pronounced as \textit{`excess-net'}, which
first learns a mask for a task and then selectively trains a subset of weights from the supermask subnetwork.
We train the weights of this subnetwork via \textit{exclusion} that avoids updating parameters from the current subnetwork that have already been updated by any of the previous tasks. This helps us to prevent forgetting. This procedure is demonstrated in Figure \ref{fig:method} for learning three tasks sequentially.
Training the supermask subnetwork's weights increases its representational power and allows \method{} to encode task-specific knowledge inside the subnetwork.
This solves the first problem and allows \method{} to perform comparably to a fully trained network on individual tasks; and when learning multiple tasks, the exclusive subnetwork training improves the performance of each task while still preventing forgetting.

To address the second problem of knowledge transfer, we propose a $k$-nearest neighbors-based knowledge transfer (KKT) module that transfers relevant information from the previously learned tasks to improve performance on new tasks while learning them faster.
Our KKT module uses KNN classification to select a subnetwork from the previously learned tasks that have better than random predictive power for the current task and use it as a starting point to learn the new tasks.

Next, we show our method's advantage by experimenting with both natural language and vision tasks. For natural language, we evaluate on WebNLP classification tasks \citep{d2019episodic,huang-etal-2021-continual-idbr} and GLUE benchmark tasks \citep{wang-etal-2018-glue}, whereas, for vision, we evaluate on SplitMNIST \citep{zenke2017continual,De_Lange_2021_cope}, SplitCIFAR100 \citep{chaudhry2018efficient,De_Lange_2021_cope}, and SplitTinyImageNet \citep{buzzega2020der} datasets.
We show that for both language and vision domains, \method{} outperforms multiple strong and recent continual learning methods based on replay, regularization, distillation, and parameter isolation.
For the vision domain, \method{} outperforms the strongest baseline by $4.8$\% and $1.4$\% on SplitCIFAR and SplitTinyImageNet datasets respectively, while surpassing multitask model and bridging the gap to training \textit{individual} models for each task.
In addition, for GLUE datasets, \method{} is 2\% better than the strongest baseline methods and surpasses the performance of multitask learning (that has access to all the tasks at once and does not suffer from CF).
Moreover, \method{} obtains an average improvement of $8.3$\% over SupSup for sparse masks with $2-10$\% of the model parameters and scales to a large number of tasks (100). Furthermore, \method{} with the KKT module learns new tasks in as few as 30 epochs compared to 100 epochs without it, while achieving 3.2\% higher accuracy on the SplitCIFAR100 dataset.
In summary, our contributions are listed below:
\begin{itemize}[topsep=0pt,leftmargin=0.8cm]
    \item We propose a simple and novel method to improve mask learning by combining it with exclusive subnetwork weight training to improve CL performance while preventing CF.

    \item We propose a \textsc{K}NN-based \textsc{K}nowledge \textsc{T}ransfer (KKT) module for supermask initialization that dynamically identifies previous tasks to transfer knowledge to learn new tasks better and faster.
    \item Extensive experiments on language and vision tasks show that \method{} outperforms strong baselines and is comparable to multitask model for language tasks while surpassing it for vision tasks. Moreover, \method{} works well for sparse masks and scales to a large number of tasks.
\end{itemize}
\section{Method}
\label{sec:method}

As shown in Figure \ref{fig:method}, when learning a new task $t_i$, \method{} follows three steps: (1) We learn a supermask $M_i$ for the task;
(2) We use all the previous tasks' masks $M_1, \hdots, M_{i-1}$ to create a free parameter mask $M^{free}_i$, that finds the parameters selected by the mask $M_i$ that were not selected by any of the previous masks;
(3) We update the weights corresponding to the mask $M^{free}_i$ as this avoids parameter interference.
Now, we formally describe all the step of our method \method{} (\textsc{E}xclusive \textsc{S}upermask \textsc{S}ub\textsc{Ne}twork \textsc{T}raining) for a Multi-layer perceptron (MLP).

\textbf{Notation:}
When finding supermasks and training subnetwork's weight, we can treat each layer $l$ of an MLP network separately. An intermediate layer $l$ has $n_l$ nodes denoted by $\mathcal{V}^{(l)} = \{v_1, \hdots, v_{n_l}\}$. For a node $v$ in layer $l$, let $\mathcal{I}_v$ denote its input and $\mathcal{Z}_v = \sigma(\mathcal{I}_v)$ denote its output, where $\sigma(.)$ is the activation function. Given this notation, $\mathcal{I}_v$ can be written as $\mathcal{I}_v = \sum_{u \in \mathcal{V}^{(l-1)}} w_{uv}\mathcal{Z}_u$, where $w_{uv}$ is the network weight connecting node $u$ to node $v$. The complete network weights for the MLP are denoted by $W$.
When training the task $t_i$, we have access to the supermasks from all previous tasks $\{M_{j}\}_{j=1}^{i-1}$ and the model weights $W^{(i-1)}$ obtained after learning task $t_{i-1}$.

\subsection{\method{}: \textsc{E}xclusive \textsc{S}upermask \textsc{S}ub\textsc{Ne}twork \textsc{T}raining}
\label{sec:exssnet}

\textbf{Finding Supermasks:} Following \citet{supsup}, we use the algorithm of \citet{ramanujan2019s} to learn a supermask $M_i$ for the current task $t_i$. The supermask $M_i$ is learned with respect to the underlying model weights $W^{(i-1)}$ and the mask selects a fraction of weights that lead to good performance on the task without training the weights.
To achieve this, we learn a score $s_{uv}$ for each weight $w_{uv}$, and once trained, these scores are thresholded to obtain the mask.
Here, the input to a node $v$ is
$\mathcal{I}_v = \sum_{u \in \mathcal{V}^{(l-1)}} w_{uv}\mathcal{Z}_u m_{uv}$,
where $m_{uv} = h(s_{uv})$ is the binary mask value and $h(.)$ is a function which outputs 1 for top-$k\%$ of the scores in the layer with $k$ being the mask density.
Next, we use a straight-through gradient estimator \citep{bengio2013straightthrough} and iterate over the current task's data samples to update the scores for the corresponding supermask $M_i$ as follows,
\begin{equation}
    \footnotesize
    s_{uv} = s_{uv} - \alpha \hat{g}_{s_{uv}}~;~\hat{g}_{s_{uv}} = \frac{\partial\mathcal{L}}{\partial \mathcal{I}_v}\frac{\partial \mathcal{I}_v}{\partial s_{uv}} = \frac{\partial\mathcal{L}}{\partial \mathcal{I}_v} w_{uv} \mathcal{Z}_u\nonumber
\end{equation}
\looseness=-1

\textbf{Finding Exclusive Mask Parameters:}
Given a learned mask $M_i$, we use all the previous tasks' masks $M_1, \hdots, M_{i-1}$ to create a free parameter mask $M^{free}_i$, that finds the parameters selected by the mask $M_i$ that were not selected by any of the previous masks.
We do this by -- (1) creating a new mask $M_{1:i-1}$ containing all the parameters already updated by any of the previous tasks by taking a union of all the previous masks $\{M_j\}_{j=1}^{i-1}$ by using the logical \textit{or} operation, and (2) Then we obtain a mask $M^{free}_i$ by taking the intersection of all the network parameters not used by any previous task which is given by the negation of the mask $M_{1:i-1}$ with the current task mask $M_i$ via a logical \textit{and} operation. Next, we use this mask $M^{free}_i$ for the exclusive supermask subnetwork weight training.

\looseness=-1
\textbf{Exclusive Supermask Subnetwork Weight Training:}
For training the subnetwork parameters for task $t_i$ given the free parameter mask $M^{free}_i$, we perform the forward pass on the model after applying the mask $M^{free}_i$ as $model(x, W \odot M^{free}_i)$, where $\odot$ is the element-wise multiplication.
Hence, during the weight training, only the parameters corresponding to the mask $M^{free}_i$ are updated because the gradient value is 0 for all the weights $w_{uv}$ where $m^{free}_{uv} = 0$. While during the inference on task $t_i$ we use the mask $M_i$.
In contrast, \modelorg{} uses the task mask $M_i$ both during the training and inference as $model(x, W^{(i-1)} \odot M_i)$. This updates all the parameters in the mask including the parameters that are already updated by previous tasks that result in CF. Therefore, in cases where the sparse overlap is high, \method{} is preferred over \modelorg{}.
To summarize, \method{} circumvents the CF issue of \modelorg{} while benefiting from the subnetwork training to improve overall performance as shown in Figure \ref{fig:sp_overlap}.

\subsubsection{Space, Time, and Memory Complexity of \method{}}
\looseness=-1
For training, we store an additional set of scores on GPU with size as the model weight. The additional GPU memory required is a small fraction because the model activations account for a huge fraction of the total GPU memory. Our runtime is similar to training the weight of a model with $<5\%$ overhead due to the logical operations on masks and masking weight during the forward passes. For training time comparisons refer to Appendix Table \ref{app:tab:runtime}. On the disk, we need to store $k*|W|$ updated weights of 32-bits and boolean mask which takes 1-bit for each parameter. Hence, we take  $max(|W|*k*t, |W|) * 32 + |W| * 1$ bits in total as in the worst case we need to store all $|W|$ model weights.

\subsection{KKT: Knn-Based Knowledge Transfer Module}
\label{sec:kkt_module}
\looseness=-1
When learning multiple tasks, it is a desired property to transfer information learned by the previous tasks to achieve better performance on new tasks and to learn them faster \citep{biesialska-etal-2020-continualsurvery}.
Hence, we propose a K-Nearest Neighbours (KNN) based knowledge transfer (KKT) module that uses KNN classification to dynamically find the most relevant previous task \citep{veniat2021efficient} to initialize the supermask for the current task.
To be more specific, before learning the mask $M_i$ for the current task $t_i$, we randomly sample a small fraction of data from task $t_i$ and split it into a train and test set. Next, we use the trained subnetworks of each previous task $t_1, \hdots, t_{i-1}$ to obtain features on this sampled data. Then we learn $i-1$ independent KNN-classification models using these features. Then we evaluate these $i-1$ models on the sampled test set to obtain accuracy scores which denote the predictive power of features from each previous task for the current task. Finally, we select the previous task with the highest accuracy on the current task. If this accuracy is better than random then we use its mask to initialize the current task's supermask. This enables \method{} to transfer information from the previous task to learn new tasks better and faster.

\section{Motivation}
\label{sec:motivation}

\begin{figure}
\centering
\begin{minipage}[t]{0.45\textwidth}
\centering
    \includegraphics[width=0.95\linewidth]{figures/motivation_wssnet1.pdf}
    \captionsetup{margin=0.25cm}
    \captionof{figure}{\label{fig:single_task}Test accuracy versus the mask density for 100-way CIFAR100 classification. Results are averaged over three seeds.}
\end{minipage}
\begin{minipage}[t]{0.45\textwidth}
\centering
    \includegraphics[width=0.95\linewidth]{figures/sp_overlap.pdf}
    \captionof{figure}{\label{fig:sp_overlap}Average Test accuracy on five 20-way tasks from SplitCIFAR100 as a function of sparse overlap. Results are averaged over three seeds.}
\end{minipage}
\end{figure}

Using sparsity for CL is an effective technique to learn multiple tasks, i.e., by encoding them in different subnetworks inside a single model.
SupSup \citep{supsup} is an instantiation of this that initializes the network weights randomly and then learns a separate supermask for each task as shown in the SupSup diagram in Appendix Figure \ref{fig:supsup}.
They prevent CF because the weights of the network are fixed and never updated. However, this is a crucial problem as discussed below.

\textbf{Problem 1 - Sub-Optimal Performance of Supermask:} Although fixed network weights in SupSup prevent CF, this also restricts the network's representational capacity, leading to worse performance compared to a fully trained network.
We demonstrate this in Figure \ref{fig:single_task}, where we report the test accuracy of different methods with respect to the fraction of network parameters selected by the mask, i.e., the \textit{mask density} for an underlying ResNet18 model \citep{supsup} on a \textit{single 100-way classification} on CIFAR100 dataset.
The fully trained ResNet18 model (dashed green line) achieves an accuracy of $63.9\%$.
Similar to \citet{zhou2019deconstructing}, we observe that the performance of SupSup (yellow dashed line) is at least $8.3$\% worse compared to a fully trained model.
As a possible \emph{partial} remedy, we propose a simple solution, \modelorg{} (\textsc{S}upermask \textsc{S}ub\textsc{Ne}twork \textsc{T}raining), that first finds a subnetwork for a task and then trains the subnetwork's weights.
This increases the representational capacity of the subnetwork because there are more trainable parameters.
For a single task, the test accuracy of \modelorg{} is better than SupSup for all mask densities and matches the performance of the fully trained model beyond a density threshold. But as shown below, when learning multiple tasks sequentially, \modelorg{} gives rise to parameter interference that results in CF.

\textbf{Problem 2 - Parameter Interference Due to Subnetwork Weight Training for Multiple Tasks:}
Next, we demonstrate that when learning multiple tasks sequentially, \modelorg{} can still lead to CF.
In Figure \ref{fig:sp_overlap}, we report the average test accuracy versus the fraction of overlapping parameters between the masks of different tasks, i.e., the \textit{sparse overlap} (see Equation \ref{eqn:sp_overlap}) for five different 20-way classification tasks from SplitCIFAR100 dataset with ResNet18 model.
We observe that \modelorg{} outperforms SupSup for lower sparse overlap but as the sparse overlap increases, the performance declines because the supermask subnetworks for different tasks have more overlapping (common) weights (bold dashed lines in Figure \ref{fig:method}). This leads to higher parameter interference resulting in increased forgetting which suppresses the gain from subnetwork weight training.

\looseness=-1
To address this second problem, our final proposed method, \method{} selectively trains a subset of the weights in the supermask subnetwork to prevent parameter interference. When learning multiple tasks, this prevents CF resulting in strict improvement in performance over SupSup as shown in Figure \ref{fig:sp_overlap} while having the representational power to bridge the gap with fully trained models, see Figure \ref{fig:single_task}.

\section{Problem Statement}
\vspace{-5pt}
\label{sec:problem_statement}

\looseness=-1
\textbf{Continual Learning Setting:}
We focus on task incremental setting \citep{hsu20clscenario} where for any pair of tasks $t_i, t_j$ and $i \neq j$: (1) the input distributions are not equal, $P(X_{t_i}) \neq P(X_{t_j})$; (2) the output distributions are not equal, $P(Y_{t_i}) \neq P(Y_{t_j})$; (3) the output label spaces are not shared across tasks $\{Y_{t_i}\} \neq \{Y_{t_j}\}$; (4) the task identification labels are provided both during training and inference.

\textbf{Metrics:} We follow \citep{chaudhry2018efficient} and evaluate our model after learning task $t$ on all the tasks, denoted by $\mathcal{T}$. This gives us an accuracy matrix $A \in \mathbb{R}^{n \times n}$, where $a_{i,j}$ represents the classification accuracy on task $j$ after learning task $i$.
Given $A$, we can assess any CL method for our desired properties.
First, once the model is trained it should perform well on all the learned tasks. This is measured by the \textit{average accuracy}, $\mathcal{A}(\mathcal{T}) = \frac{1}{N} \sum_{k=1}^{N} a_{N,k}$, where $N$ is the number of tasks.
Second, we want the model to retain performance on the previous tasks while learning the current task.
This is measured using the \textit{forgetting metric} \citep{lopez2017gradient}, $F(\mathcal{T}) = \frac{1}{N-1} \sum_{t=1}^{N-1} (\max_{k \in \{1, \hdots, N-1\}} a_{k,t} - a_{N,t})$. This is the average difference between the maximum accuracy obtained for task $t$ and its final accuracy.
Higher accuracy and lower forgetting are desired.

\section{Related Work}
\vspace{-7pt}
\label{sec:related}

Continual Learning methods fall into three main categories: \textit{Regularization}, \textit{Replay}, and \textit{Architecture} based methods.
We point the readers to \citet{delange2021clsurvey,biesialska-etal-2020-continualsurvery} for a comprehensive survey of all continual learning methods. Next, we discuss these main categories.

\looseness=-1
\textbf{Regularization-based methods} estimate the importance of model components and add importance regularization terms to the loss function.
\citet{zenke2017continual} regularize based on the distance of weights from their initialization, whereas \citet{kirkpatrick2017overcoming,schwarz2018progress} use an approximation of the Fisher information matrix \citep{pascanu2013revisiting} to regularize the parameters.
In NLP, regularization methods \citep{han2020continual,wang2019sentence} are used to constrain the relevant information from the huge amount of knowledge inside large language models (LLM).
\citet{huang-etal-2021-continual-idbr} first identifying hidden spaces that need to be updated versus retained via information disentanglement \citep{fu2017style,li2020complementary} and then regularize these hidden spaces separately.
Our method inherently avoids the need for regularization as it avoids parameter interference.

\textbf{Replay based methods} maintain a small memory buffer of data samples \citep{de2019continual} or their relevant proxies \citep{rebuffi2017icarl} from the previous tasks and retrain for them later to prevent CF. \citet{lopez2017gradient,chaudhry2018efficient} use the buffer during optimization to constrain parameter gradients.
\citet{shin2017continual,kemker2018fearnet} uses a generative model to sample and replay pseudo-data during training, whereas
\citet{rebuffi2017icarl} replay distilled knowledge from the previous tasks.
\citet{d2019episodic} employ episodic memory along with local adaptation for CL in the NLP domain, whereas \citet{sun2019lamol} trains a language model to generate a pseudo-sample for replay. These methods usually have associated memory and runtime costs whereas our method works without a replay buffer.

\looseness=-1
\textbf{Architecture based methods} can be divided into two categories: (1) methods that add new modules over time \citep{sodhani2019toward,veniat2021efficient,douillard2021dytox}; and (2) methods that isolate the network's parameters for different tasks \citep{Fernando2017pathnet,mallya2018packnet,mallya2018piggyback}. \citet{rusu2016progressive} introduce a new network for each task that is connected to all the previous tasks resulting in super-linear growth in network size. \citet{schwarz2018progress} fix this issue by distilling the new network after each task into the original one.
Recent prompt learning-based CL models for vision \citep{wang2022dualprompt,wang2022l2p} assume access to a pre-trained model to learn a set of prompts that can potentially be shared across tasks to perform CL this is orthogonal to our method that trains from scratch.
Methods like \citet{Fernando2017pathnet} initialize a fixed-size model and reuse a subset of modules for each task by finding a path in the graph of neural network modules.
\citet{mallya2018packnet} allocates parameters to specific tasks and then trains them in isolation which limits the number of tasks that can be learned.
In contrast, \citet{mallya2018piggyback} use a frozen pretrained model and learns a new mask for each task but a pretrained model is crucial for their method's good performance. \citet{supsup} removes the pretrained model dependence and learns a mask for each task over a fixed randomly initialized network.
\method{} avoids the shortcomings of \citet{mallya2018packnet,mallya2018piggyback} and performs supermask subnetwork training to increase the representational capacity compared to \citep{supsup} while performing knowledge transfer and avoiding CF.

redpajama_set_name: RedPajamaArXiv


text:
Viola cuneata S. Watson wedge leaf violet - native
Violaceae
Natl. For. Dev. Rd. 4201, 2.1 mi. W of Hwy 199 (SW of Eight Dollar Mt.), c. 5.8 mi. SW of Selma, Josephine Co., OR, 4/10/2007 Natl. For. Dev. Rd. 4201, 2.1 mi. W of Hwy 199 (SW of Eight Dollar Mt.), c. 5.8 mi. SW of Selma, Josephine Co., OR, 4/10/2007 Natl. For. Dev. Rd. 4201, 2.1 mi. W of Hwy 199 (SW of Eight Dollar Mt.), c. 5.8 mi. SW of Selma, Josephine Co., OR, 4/10/2007 Natl. For. Dev. Rd. 4201, 2.1 mi. W of Hwy 199 (SW of Eight Dollar Mt.), c. 5.8 mi. SW of Selma, Josephine Co., OR, 4/10/2007
Darlingtonia fen off NFD Rd. 4201, 1.7 mi. SW of Illinois River Bridge, SW of 8 Dollar Mt., 1720 ft. elev., N42.23084, W123.70367, Josephine Co., OR, 5/12/2010 Darlingtonia fen off NFD Rd. 4201, 1.7 mi. SW of Illinois River Bridge, SW of 8 Dollar Mt., 1720 ft. elev., N42.23084, W123.70367, Josephine Co., OR, 5/12/2010 Darlingtonia fen off NFD Rd. 4201, 1.7 mi. SW of Illinois River Bridge, SW of 8 Dollar Mt., 1720 ft. elev., N42.23084, W123.70367, Josephine Co., OR, 5/12/2010
CR5070, 2 mi. W of Selma (N of Eight Dollar Mt.), N42.28167, W123.65278, Josephine Co., OR, 4/10/2007 CR5070, 2 mi. W of Selma (N of Eight Dollar Mt.), N42.28167, W123.65278, Josephine Co., OR, 4/10/2007
CR5070, 2 mi. W of Selma (N of Eight Dollar Mt.), N42.28167, W123.65278, Josephine Co., OR, 4/10/2007 CR5070, 2 mi. W of Selma (N of Eight Dollar Mt.), N42.28167, W123.65278, Josephine Co., OR, 4/10/2007 CR5070, 2 mi. W of Selma (N of Eight Dollar Mt.), N42.28167, W123.65278, Josephine Co., OR, 4/10/2007