Activation-Informed Merging of Large Language Models
Model merging, a method that combines the parameters and embeddings of
multiple fine-tuned large language models (LLMs), offers a promising approach
to enhance model performance across various tasks while maintaining
computational efficiency. This paper introduces Activation-Informed Merging
(AIM), a technique that integrates the information from the activation space of
LLMs into the merging process to improve performance and robustness. AIM is
designed as a flexible, complementary solution that is applicable to any
existing merging method. It aims to preserve critical weights from the base
model, drawing on principles from continual learning~(CL) and model
compression. Utilizing a task-agnostic calibration set, AIM selectively
prioritizes essential weights during merging. We empirically demonstrate that
AIM significantly enhances the performance of merged models across multiple
benchmarks. Our findings suggest that considering the activation-space
information can provide substantial advancements in the model merging
strategies for LLMs with up to 40\% increase in benchmark performance.
67a4fc2450641c7d60cead58
Amin Heyrani Nobari
extracted_pending
2025-02-06T18:15:01.447Z
64d516ba80d47a6b76fc1015
/avatars/e520825f2ac9ff047844496ae2dad7d6.svg
Amin Heyrani Nobari
user
ahn1376
67a4fc2450641c7d60cead59
Kaveh Alimohammadi
67a4fc2450641c7d60cead5a
Ali ArjomandBigdeli
67a4fc2450641c7d60cead5b
Akash Srivastava
67a4fc2450641c7d60cead5c
Faez Ahmed
67a4fc2450641c7d60cead5d
Navid Azizan
Activation-Informed Merging of Large Language Models
Model merging, a method that combines the parameters and embeddings of
multiple fine-tuned large language models (LLMs), offers a promising approach
to enhance model performance across various tasks while maintaining
computational efficiency. This paper introduces Activation-Informed Merging
(AIM), a technique that integrates the information from the activation space of
LLMs into the merging process to improve performance and robustness. AIM is
designed as a flexible, complementary solution that is applicable to any
existing merging method. It aims to preserve critical weights from the base
model, drawing on principles from continual learning~(CL) and model
compression. Utilizing a task-agnostic calibration set, AIM selectively
prioritizes essential weights during merging. We empirically demonstrate that
AIM significantly enhances the performance of merged models across multiple
benchmarks. Our findings suggest that considering the activation-space
information can provide substantial advancements in the model merging
strategies for LLMs with up to 40\% increase in benchmark performance.
A novel activation-informed merging technique for large language models enhances performance across tasks by selectively prioritizing essential weights during the merging process.
Activation-Informed Merging AIM merging methods fine-tuned large language models LLMs continual learning CL model compression task-agnostic calibration set