Residual Prompt Tuning: Improving Prompt Tuning with Residual
  Reparameterization
Prompt tuning is one of the successful approaches for parameter-efficient
tuning of pre-trained language models. Despite being arguably the most
parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
it typically performs worse than other efficient tuning methods and is quite
sensitive to hyper-parameters. In this work, we introduce Residual Prompt
Tuning - a simple and efficient method that significantly improves the
performance and stability of prompt tuning. We propose to reparameterize soft
prompt embeddings using a shallow network with a residual connection. Our
experiments show that Residual Prompt Tuning significantly outperforms prompt
tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
improvement over prompt tuning with T5-Base and allows to reduce the prompt
length by 10x without hurting performance. In addition, we show that our
approach is robust to the choice of learning rate and prompt initialization,
and is effective in few-shot settings.
6459a4663b6fafd9664711e2
Anastasia Razdaibiedina
6459a4663b6fafd9664711e3
Yuning Mao
admin_assigned
2023-06-02T14:16:57.213Z
62bf46a630a1888a761f8619
/avatars/de6c656066b1e88d37882ce68a0fd25c.svg
Yuning Mao
user
yuningm
6459a4663b6fafd9664711e4
Rui Hou
admin_assigned
2023-06-02T14:17:13.225Z
6407d236a7bc7c3865adcc3a
/avatars/8b75cc07fd049acde5363d268c5b88d4.svg
Rui Hou
user
rayhou
6459a4663b6fafd9664711e5
Madian Khabsa
6459a4663b6fafd9664711e6
Mike Lewis
admin_assigned
2023-06-02T14:17:40.021Z
5e67eda71009063689407484
/avatars/6d7688faba888d463584eaa02ad011e3.svg
Mike Lewis
user
mikelewis0
6459a4663b6fafd9664711e7
Jimmy Ba
6459a4663b6fafd9664711e8
Amjad Almahairi
Residual Prompt Tuning: Improving Prompt Tuning with Residual
  Reparameterization
Prompt tuning is one of the successful approaches for parameter-efficient
tuning of pre-trained language models. Despite being arguably the most
parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
it typically performs worse than other efficient tuning methods and is quite
sensitive to hyper-parameters. In this work, we introduce Residual Prompt
Tuning - a simple and efficient method that significantly improves the
performance and stability of prompt tuning. We propose to reparameterize soft
prompt embeddings using a shallow network with a residual connection. Our
experiments show that Residual Prompt Tuning significantly outperforms prompt
tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
improvement over prompt tuning with T5-Base and allows to reduce the prompt
length by 10x without hurting performance. In addition, we show that our
approach is robust to the choice of learning rate and prompt initialization,
and is effective in few-shot settings.
Residual Prompt Tuning enhances prompt tuning for parameter-efficient pre-trained language models, improving performance and stability while reducing prompt length.
prompt tuning parameter-efficient tuning soft prompts reparameterization residual connection SuperGLUE benchmark few-shot settings