Delta Activations: A Representation for Finetuned Large Language Models
The success of powerful open source Large Language Models (LLMs) has enabled
the community to create a vast collection of post-trained models adapted to
specific tasks and domains. However, navigating and understanding these models
remains challenging due to inconsistent metadata and unstructured repositories.
We introduce Delta Activations, a method to represent finetuned models as
vector embeddings by measuring shifts in their internal activations relative to
a base model. This representation allows for effective clustering by domain and
task, revealing structure in the model landscape. Delta Activations also
demonstrate desirable properties: it is robust across finetuning settings and
exhibits an additive property when finetuning datasets are mixed. In addition,
we show that Delta Activations can embed tasks via few-shot finetuning, and
further explore its use for model selection and merging. We hope Delta
Activations can facilitate the practice of reusing publicly available models.
Code is available at https://github.com/OscarXZQ/delta_activations.
68ba6fd7736018af705e8ed5
Zhiqiu Xu
claimed_verified
2025-09-05T07:58:07.308Z
648f276cdb6279384d1b3231
/avatars/9137b367cf2da707cf78363e9f3af73b.svg
Zhiqiu (Oscar) Xu
user
OscarXZQ
68ba6fd7736018af705e8ed6
Amish Sethi
68ba6fd7736018af705e8ed7
Mayur Naik
68ba6fd7736018af705e8ed8
Ser-Nam Lim
Delta Activations: A Representation for Finetuned Large Language Models
The success of powerful open source Large Language Models (LLMs) has enabled
the community to create a vast collection of post-trained models adapted to
specific tasks and domains. However, navigating and understanding these models
remains challenging due to inconsistent metadata and unstructured repositories.
We introduce Delta Activations, a method to represent finetuned models as
vector embeddings by measuring shifts in their internal activations relative to
a base model. This representation allows for effective clustering by domain and
task, revealing structure in the model landscape. Delta Activations also
demonstrate desirable properties: it is robust across finetuning settings and
exhibits an additive property when finetuning datasets are mixed. In addition,
we show that Delta Activations can embed tasks via few-shot finetuning, and
further explore its use for model selection and merging. We hope Delta
Activations can facilitate the practice of reusing publicly available models.
Code is available at https://github.com/OscarXZQ/delta_activations.
Delta Activations represent fine-tuned models as vector embeddings based on internal activation shifts, enabling effective clustering and model reuse.
Large Language Models LLMs Delta Activations vector embeddings internal activations base model clustering domain task robust additive property few-shot finetuning model selection model merging