GPTailor: Large Language Model Pruning Through Layer Cutting and
  Stitching
Large language models (LLMs) have shown remarkable capabilities in language
understanding and generation. However, such impressive capability typically
comes with a substantial model size, which presents significant challenges in
deployment and inference. While structured pruning of model parameters offers a
promising way to reduce computational costs at deployment time, current methods
primarily focus on single model pruning. In this work, we develop a novel
strategy to compress models by strategically combining or merging layers from
finetuned model variants, which preserves the original model's abilities by
aggregating capabilities accentuated in different finetunes. We pose the
optimal tailoring of these LLMs as a zero-order optimization problem, adopting
a search space that supports three different operations: (1) Layer removal, (2)
Layer selection from different candidate models, and (3) Layer merging. Our
experiments demonstrate that this approach leads to competitive model pruning,
for example, for the Llama2-13B model families, our compressed models maintain
approximately 97.3\% of the original performance while removing sim25% of
parameters, significantly outperforming previous state-of-the-art methods. The
code is available at https://github.com/Guinan-Su/auto-merge-llm.
685d1864696820ba1f28f366
Guinan Su
claimed_verified
2025-06-27T08:57:32.688Z
630ed6cfc9af0163b96a71aa
/avatars/bf984bb1c2f68336bcf042c6fc7e0d9d.svg
Guinan-Su
user
guinansu
685d1864696820ba1f28f367
Li Shen
685d1864696820ba1f28f368
Lu Yin
685d1864696820ba1f28f369
Shiwei Liu
685d1864696820ba1f28f36a
Yanwu Yang
685d1864696820ba1f28f36b
Jonas Geiping
claimed_verified
2025-06-27T08:57:34.794Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
GPTailor: Large Language Model Pruning Through Layer Cutting and
  Stitching
Large language models (LLMs) have shown remarkable capabilities in language
understanding and generation. However, such impressive capability typically
comes with a substantial model size, which presents significant challenges in
deployment and inference. While structured pruning of model parameters offers a
promising way to reduce computational costs at deployment time, current methods
primarily focus on single model pruning. In this work, we develop a novel
strategy to compress models by strategically combining or merging layers from
finetuned model variants, which preserves the original model's abilities by
aggregating capabilities accentuated in different finetunes. We pose the
optimal tailoring of these LLMs as a zero-order optimization problem, adopting
a search space that supports three different operations: (1) Layer removal, (2)
Layer selection from different candidate models, and (3) Layer merging. Our
experiments demonstrate that this approach leads to competitive model pruning,
for example, for the Llama2-13B model families, our compressed models maintain
approximately 97.3\% of the original performance while removing sim25% of
parameters, significantly outperforming previous state-of-the-art methods. The
code is available at https://github.com/Guinan-Su/auto-merge-llm.
A new strategy merges layers from fine-tuned model variants to compress large language models with minimal performance loss.
LLMs structured pruning layer removal layer selection layer merging zero-order optimization Llama2-13B