Knowledge Composition using Task Vectors with Learned Anisotropic
  Scaling
Pre-trained models produce strong generic representations that can be adapted
via fine-tuning. The learned weight difference relative to the pre-trained
model, known as a task vector, characterises the direction and stride of
fine-tuning. The significance of task vectors is such that simple arithmetic
operations on them can be used to combine diverse representations from
different domains. This paper builds on these properties of task vectors and
aims to answer (1) whether components of task vectors, particularly parameter
blocks, exhibit similar characteristics, and (2) how such blocks can be used to
enhance knowledge composition and transfer. To this end, we introduce aTLAS, an
algorithm that linearly combines parameter blocks with different learned
coefficients, resulting in anisotropic scaling at the task vector level. We
show that such linear combinations explicitly exploit the low intrinsic
dimensionality of pre-trained models, with only a few coefficients being the
learnable parameters. Furthermore, composition of parameter blocks leverages
the already learned representations, thereby reducing the dependency on large
amounts of data. We demonstrate the effectiveness of our method in task
arithmetic, few-shot recognition and test-time adaptation, with supervised or
unsupervised objectives. In particular, we show that (1) learned anisotropic
scaling allows task vectors to be more disentangled, causing less interference
in composition; (2) task vector composition excels with scarce or no labeled
data and is less prone to domain shift, thus leading to better
generalisability; (3) mixing the most informative parameter blocks across
different task vectors prior to training can reduce the memory footprint and
improve the flexibility of knowledge transfer. Moreover, we show the potential
of aTLAS as a PEFT method, particularly with less data, and demonstrate that
its scalibility.
668dd783a697b69d668feaed
Frederic Z. Zhang
claimed_verified
2024-07-10T08:19:56.816Z
668ca1b3ed63008dfa687990
/avatars/b3ae0e7ff60c60db605288c9cf5ae6f3.svg
Fred Zhang
user
fredzzhang
668dd783a697b69d668feaee
Paul Albert
claimed_verified
2024-07-10T08:19:54.961Z
65f26ed18404ac0e4cfe7d83
/avatars/9c72744836f86a8e355a45700b10e393.svg
Paul Albert
user
PAlbert31
668dd783a697b69d668feaef
Cristian Rodriguez-Opazo
claimed_verified
2024-07-10T10:20:24.326Z
66456873dd82c5792f4fd53a
/avatars/0dcdc7243a81920e7448230541590195.svg
Cristian Rodriguez-Opazo
user
crodriguezo
668dd783a697b69d668feaf0
Anton van den Hengel
668dd783a697b69d668feaf1
Ehsan Abbasnejad
Knowledge Composition using Task Vectors with Learned Anisotropic
  Scaling
Pre-trained models produce strong generic representations that can be adapted
via fine-tuning. The learned weight difference relative to the pre-trained
model, known as a task vector, characterises the direction and stride of
fine-tuning. The significance of task vectors is such that simple arithmetic
operations on them can be used to combine diverse representations from
different domains. This paper builds on these properties of task vectors and
aims to answer (1) whether components of task vectors, particularly parameter
blocks, exhibit similar characteristics, and (2) how such blocks can be used to
enhance knowledge composition and transfer. To this end, we introduce aTLAS, an
algorithm that linearly combines parameter blocks with different learned
coefficients, resulting in anisotropic scaling at the task vector level. We
show that such linear combinations explicitly exploit the low intrinsic
dimensionality of pre-trained models, with only a few coefficients being the
learnable parameters. Furthermore, composition of parameter blocks leverages
the already learned representations, thereby reducing the dependency on large
amounts of data. We demonstrate the effectiveness of our method in task
arithmetic, few-shot recognition and test-time adaptation, with supervised or
unsupervised objectives. In particular, we show that (1) learned anisotropic
scaling allows task vectors to be more disentangled, causing less interference
in composition; (2) task vector composition excels with scarce or no labeled
data and is less prone to domain shift, thus leading to better
generalisability; (3) mixing the most informative parameter blocks across
different task vectors prior to training can reduce the memory footprint and
improve the flexibility of knowledge transfer. Moreover, we show the potential
of aTLAS as a PEFT method, particularly with less data, and demonstrate that
its scalibility.
aTLAS algorithm leverages parameter blocks from pre-trained models to enhance knowledge composition and transfer, achieving better generalization and flexibility with limited data.
pre-trained models fine-tuning task vector parameter blocks anisotropic scaling linear combinations intrinsic dimensionality few-shot recognition test-time adaptation disentangled domain shift knowledge transfer PEFT scalibility