NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient
  Fine-Tuning
Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into
two categories: addition-based and selective in-situ adaptation. The former,
such as LoRA, introduce additional modules to adapt the model to downstream
tasks, offering strong memory efficiency. However, their representational
capacity is often limited, making them less suitable for fine-grained
adaptation. In contrast, the latter directly fine-tunes a carefully chosen
subset of the original model parameters, allowing for more precise and
effective adaptation, but at the cost of significantly increased memory
consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT
method that enables fine-grained model finetuning while maintaining high memory
efficiency. Our approach first identifies important parameters (i.e.,
connections within the network) as in selective adaptation, and then introduces
bypass connections for these selected parameters. During finetuning, only the
bypass connections are updated, leaving the original model parameters frozen.
Empirical results on 23+ tasks spanning both natural language generation and
understanding demonstrate that NeuroAda achieves state-of-the-art performance
with as little as leq 0.02% trainable parameters, while reducing
CUDA memory usage by up to 60%. We release our code here:
https://github.com/FightingFighting/NeuroAda.git.
68f998bdb9b2e4ae0467382f
Zhi Zhang
claimed_verified
2025-10-23T14:48:14.251Z
62b8e6adaf1addc58cfcc578
/avatars/3f2b190eea73bd2bf2b4ddec1014edf5.svg
ZhiZhang
user
ZhiZhang
68f998bdb9b2e4ae04673830
Yixian Shen
68f998bdb9b2e4ae04673831
Congfeng Cao
68f998bdb9b2e4ae04673832
Ekaterina Shutova
NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient
  Fine-Tuning
Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into
two categories: addition-based and selective in-situ adaptation. The former,
such as LoRA, introduce additional modules to adapt the model to downstream
tasks, offering strong memory efficiency. However, their representational
capacity is often limited, making them less suitable for fine-grained
adaptation. In contrast, the latter directly fine-tunes a carefully chosen
subset of the original model parameters, allowing for more precise and
effective adaptation, but at the cost of significantly increased memory
consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT
method that enables fine-grained model finetuning while maintaining high memory
efficiency. Our approach first identifies important parameters (i.e.,
connections within the network) as in selective adaptation, and then introduces
bypass connections for these selected parameters. During finetuning, only the
bypass connections are updated, leaving the original model parameters frozen.
Empirical results on 23+ tasks spanning both natural language generation and
understanding demonstrate that NeuroAda achieves state-of-the-art performance
with as little as leq 0.02% trainable parameters, while reducing
CUDA memory usage by up to 60%. We release our code here:
https://github.com/FightingFighting/NeuroAda.git.
NeuroAda is a parameter-efficient fine-tuning method that combines selective adaptation with bypass connections to achieve high performance with minimal trainable parameters and reduced memory usage.
parameter-efficient fine-tuning PEFT LoRA addition-based selective in-situ adaptation NeuroAda important parameters bypass connections natural language generation natural language understanding