Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
The performance of large language models in domain-specific tasks
necessitates fine-tuning, which is computationally expensive and technically
challenging. This paper focuses on parameter-efficient fine-tuning using soft
prompting, a promising approach that adapts pre-trained models to downstream
tasks by learning a small set of parameters. We propose a novel Input Dependent
Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that
generates soft prompts based on the input tokens and attends different tokens
with varying importance. Our method is simple and efficient, keeping the number
of trainable parameters small. We show the merits of the proposed approach
compared to state-of-the-art techniques on various tasks and show the improved
zero shot domain transfer capability.
68464b273ec10bdd8ab4da86
Ananth Muppidi
extracted_pending
2025-06-09T02:47:03.971Z
64a6518132cf858d6386ac52
/avatars/4cabf3dab8b1ba06245ad8024f334181.svg
Ananth Muppidi
user
ananthmuppidi
68464b273ec10bdd8ab4da87
Abhilash Nandy
extracted_confirmed
2025-06-09T03:49:29.446Z
5f89da6c5d083370c711f37c
/avatars/c2f17a4a636973817fd5da2ae6dbaac3.svg
Abhilash Nandy
user
abhi1nandy2
68464b273ec10bdd8ab4da88
Sambaran Bandyopadhyay
extracted_confirmed
2025-06-10T07:04:13.158Z
65238ea295df08170c93933d
/avatars/8364301e324274a550d12f2b184ea10e.svg
Sambaran Bandyopadhyay
user
sambaran
Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
The performance of large language models in domain-specific tasks
necessitates fine-tuning, which is computationally expensive and technically
challenging. This paper focuses on parameter-efficient fine-tuning using soft
prompting, a promising approach that adapts pre-trained models to downstream
tasks by learning a small set of parameters. We propose a novel Input Dependent
Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that
generates soft prompts based on the input tokens and attends different tokens
with varying importance. Our method is simple and efficient, keeping the number
of trainable parameters small. We show the merits of the proposed approach
compared to state-of-the-art techniques on various tasks and show the improved
zero shot domain transfer capability.
A new method using input-dependent soft prompting with a self-attention mechanism improves parameter-efficient fine-tuning for large language models, enhancing zero-shot domain transfer.
soft prompting parameter-efficient fine-tuning pre-trained models downstream tasks Input Dependent Soft Prompting technique self-Attention Mechanism zero shot domain transfer