GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
democratized the usage of Large Language Models (LLMs). Recent studies have
shown that a small subset of weights significantly impacts performance. Based
on this observation, we introduce a novel PEFT method, called Gaussian noise
Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
salient columns, while injecting Gaussian noise into non-salient ones. To
identify these columns, we developeda generalized sensitivity metric that
extends and unifies metrics from previous studies. Experiments with LLaMA
models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
methods under the same computational budget. Moreover, GIFT-SW offers practical
advantages to recover performance of models subjected to mixed-precision
quantization with keeping salient weights in full precision.
66d44c5c8a438492b073a251
Maxim Zhelnin
66d44c5c8a438492b073a252
Viktor Moskvoretskii
66d44c5c8a438492b073a253
Egor Shvetsov
claimed_verified
2024-09-02T07:17:05.907Z
65afde6ba0b4bf3b0e95b4e8
/avatars/e9b97040b0a619bf6609465d1678705c.svg
Egor Shvetsov
user
dalime
66d44c5c8a438492b073a254
Egor Venediktov
66d44c5c8a438492b073a255
Mariya Krylova
66d44c5c8a438492b073a256
Aleksandr Zuev
66d44c5c8a438492b073a257
Evgeny Burnaev
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
democratized the usage of Large Language Models (LLMs). Recent studies have
shown that a small subset of weights significantly impacts performance. Based
on this observation, we introduce a novel PEFT method, called Gaussian noise
Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
salient columns, while injecting Gaussian noise into non-salient ones. To
identify these columns, we developeda generalized sensitivity metric that
extends and unifies metrics from previous studies. Experiments with LLaMA
models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
methods under the same computational budget. Moreover, GIFT-SW offers practical
advantages to recover performance of models subjected to mixed-precision
quantization with keeping salient weights in full precision.
GIFT-SW, a new PEFT method, fine-tunes only salient weights and injects Gaussian noise into non-salient ones, outperforming full fine-tuning and other PEFT methods while maintaining model performance post-quantization.
Parameter Efficient Fine-Tuning PEFT Large Language Models LLMs Gaussian noise salient weights generalized sensitivity metric full fine-tuning mixed-precision quantization