KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
The increasing sizes of large language models (LLMs) result in significant
computational overhead and memory usage when adapting these models to specific
tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have
been devised to mitigate these challenges by training a small set of parameters
for the task-specific updates of the model weights. Among PEFT methods, LoRA
stands out for its simplicity and efficiency, inspiring the development of a
series of variants. However, LoRA and its successors disregard the knowledge
that is noisy or irrelevant to the targeted task, detrimentally impacting model
performance and leading to suboptimality. To address this limitation, we
introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that
leverages singular value decomposition (SVD) with knowledge-aware singular
values to dynamically activate knowledge based on its relevance to the task at
hand. We conduct extensive experiments across a range of LLMs on tasks spanning
natural language understanding (NLU), generation (NLG), instruction following,
and commonsense reasoning. The experimental results demonstrate that KaSA
consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks
and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
The source code of our method is available at
https://github.com/juyongjiang/KaSA.
675ad3a39d2eead3eb196816
Fan Wang
675ad3a39d2eead3eb196817
Juyong Jiang
claimed_verified
2024-12-30T19:37:01.554Z
64b63712b3b69063d9aff341
https://cdn-avatars.huggingface.co/v1/production/uploads/64b63712b3b69063d9aff341/nP6GyBKBtnWWoA0TvQF3K.jpeg
John
user
juyongjiang
675ad3a39d2eead3eb196818
Chansung Park
claimed_verified
2024-12-12T13:40:54.230Z
60d3b57ad7b174177faabd6e
https://cdn-avatars.huggingface.co/v1/production/uploads/1659971187637-60d3b57ad7b174177faabd6e.jpeg
chansung park
user
chansung
675ad3a39d2eead3eb196819
Sunghun Kim
675ad3a39d2eead3eb19681a
Jing Tang
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
The increasing sizes of large language models (LLMs) result in significant
computational overhead and memory usage when adapting these models to specific
tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have
been devised to mitigate these challenges by training a small set of parameters
for the task-specific updates of the model weights. Among PEFT methods, LoRA
stands out for its simplicity and efficiency, inspiring the development of a
series of variants. However, LoRA and its successors disregard the knowledge
that is noisy or irrelevant to the targeted task, detrimentally impacting model
performance and leading to suboptimality. To address this limitation, we
introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that
leverages singular value decomposition (SVD) with knowledge-aware singular
values to dynamically activate knowledge based on its relevance to the task at
hand. We conduct extensive experiments across a range of LLMs on tasks spanning
natural language understanding (NLU), generation (NLG), instruction following,
and commonsense reasoning. The experimental results demonstrate that KaSA
consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks
and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
The source code of our method is available at
https://github.com/juyongjiang/KaSA.
Knowledge-aware Singular-value Adaptation (KaSA) enhances parameter-efficient fine-tuning of large language models by dynamically activating relevant knowledge using singular value decomposition.
large language models parameter-efficient fine-tuning PEFT LoRA singular value decomposition SVD natural language understanding generation instruction following commonsense reasoning Knowledge-aware Singular-value Adaptation KaSA