SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in
  LLMs
We propose SPARC, a lightweight continual learning framework for large
language models (LLMs) that enables efficient task adaptation through prompt
tuning in a lower-dimensional space. By leveraging principal component analysis
(PCA), we identify a compact subspace of the training data. Optimizing prompts
in this lower-dimensional space enhances training efficiency, as it focuses
updates on the most relevant features while reducing computational overhead.
Furthermore, since the model's internal structure remains unaltered, the
extensive knowledge gained from pretraining is fully preserved, ensuring that
previously learned information is not compromised during adaptation. Our method
achieves high knowledge retention in both task-incremental and
domain-incremental continual learning setups while fine-tuning only 0.04% of
the model's parameters. Additionally, by integrating LoRA, we enhance
adaptability to computational constraints, allowing for a tradeoff between
accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate
that our PCA-based prompt tuning combined with LoRA maintains full knowledge
retention while improving accuracy, utilizing only 1% of the model's
parameters. These results establish our approach as a scalable and
resource-efficient solution for continual learning in LLMs.
67aa91fd5f845ebfe01d7769
Dinithi Jayasuriya
67aa91fd5f845ebfe01d776a
Sina Tayebati
claimed_verified
2025-02-13T08:25:43.230Z
655ec30b12fb73960ceb048f
https://cdn-avatars.huggingface.co/v1/production/uploads/655ec30b12fb73960ceb048f/q7zVSStJWBywrtPoL2ChO.png
Sina Tayebati
user
sinatayebati
67aa91fd5f845ebfe01d776b
Davide Ettori
67aa91fd5f845ebfe01d776c
Ranganath Krishnan
extracted_pending
2025-02-10T23:55:43.302Z
647a45aeccb84c6180b41b54
/avatars/cd0db59a1b7f49f53f65751a8efc1033.svg
Ranganath Krishnan
user
ranganathkrishnan
67aa91fd5f845ebfe01d776d
Amit Ranjan Trivedi
SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in
  LLMs
We propose SPARC, a lightweight continual learning framework for large
language models (LLMs) that enables efficient task adaptation through prompt
tuning in a lower-dimensional space. By leveraging principal component analysis
(PCA), we identify a compact subspace of the training data. Optimizing prompts
in this lower-dimensional space enhances training efficiency, as it focuses
updates on the most relevant features while reducing computational overhead.
Furthermore, since the model's internal structure remains unaltered, the
extensive knowledge gained from pretraining is fully preserved, ensuring that
previously learned information is not compromised during adaptation. Our method
achieves high knowledge retention in both task-incremental and
domain-incremental continual learning setups while fine-tuning only 0.04% of
the model's parameters. Additionally, by integrating LoRA, we enhance
adaptability to computational constraints, allowing for a tradeoff between
accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate
that our PCA-based prompt tuning combined with LoRA maintains full knowledge
retention while improving accuracy, utilizing only 1% of the model's
parameters. These results establish our approach as a scalable and
resource-efficient solution for continual learning in LLMs.
A lightweight continual learning framework for LLMs uses PCA-based prompt tuning in a lower-dimensional space to enhance efficiency and retain knowledge while adapting to new tasks.
SPARC continual learning large language models LLMs prompt tuning principal component analysis PCA task-incremental domain-incremental fine-tuning LoRA knowledge retention SuperGLUE