CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting
  Mitigation
This paper introduces CURLoRA, a novel approach to fine-tuning large language
models (LLMs) that leverages CUR matrix decomposition in the context of
Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM
fine-tuning: mitigating catastrophic forgetting during continual learning and
reducing the number of trainable parameters. We propose a unique modification
to the CUR decomposition process, utilizing inverted probabilities for column
and row selection which acts as an implicit regularization, and initializing
the U matrix as a zero matrix, and only fine-tuning it. We demonstrate
through experiments on multiple datasets that CURLoRA outperforms standard LoRA
in mitigating catastrophic forgetting. It maintains model stability and
performance across tasks while significantly reducing the number of trainable
parameters. Our results show that CURLoRA achieves very good and stable task
accuracy while maintaining base model's perplexity scores fixed compared to
LoRA upon continual fine-tuning, particularly in scenarios with limited data.
66d104cab66342762f60228a
Muhammad Fawi
claimed_verified
2024-08-30T07:18:07.049Z
6629743aee8fc0fa81e184b7
https://cdn-avatars.huggingface.co/v1/production/uploads/6629743aee8fc0fa81e184b7/pgjTipnBVt0uo5kwQzddx.png
Muhammad Fawi
user
mnoorfawi
CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting
  Mitigation
This paper introduces CURLoRA, a novel approach to fine-tuning large language
models (LLMs) that leverages CUR matrix decomposition in the context of
Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM
fine-tuning: mitigating catastrophic forgetting during continual learning and
reducing the number of trainable parameters. We propose a unique modification
to the CUR decomposition process, utilizing inverted probabilities for column
and row selection which acts as an implicit regularization, and initializing
the U matrix as a zero matrix, and only fine-tuning it. We demonstrate
through experiments on multiple datasets that CURLoRA outperforms standard LoRA
in mitigating catastrophic forgetting. It maintains model stability and
performance across tasks while significantly reducing the number of trainable
parameters. Our results show that CURLoRA achieves very good and stable task
accuracy while maintaining base model's perplexity scores fixed compared to
LoRA upon continual fine-tuning, particularly in scenarios with limited data.
CURLoRA, a method that uses CUR matrix decomposition in Low-Rank Adaptation (LoRA), effectively mitigates catastrophic forgetting and reduces trainable parameters in fine-tuning large language models.
CUR matrix decomposition Low-Rank Adaptation (LoRA) catastrophic forgetting continual learning