The LLM Surgeon
State-of-the-art language models are becoming increasingly large in an effort
to achieve the highest performance on large corpora of available textual data.
However, the sheer size of the Transformer architectures makes it difficult to
deploy models within computational, environmental or device-specific
constraints. We explore data-driven compression of existing pretrained models
as an alternative to training smaller models from scratch. To do so, we scale
Kronecker-factored curvature approximations of the target loss landscape to
large language models. In doing so, we can compute both the dynamic allocation
of structures that can be removed as well as updates of remaining weights that
account for the removal. We provide a general framework for unstructured,
semi-structured and structured pruning and improve upon weight updates to
capture more correlations between weights, while remaining computationally
efficient. Experimentally, our method can prune rows and columns from a range
of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance,
and achieve state-of-the-art results in unstructured and semi-structured
pruning of large language models.
658e398711f68f12eae6e23e
Tycho F. A. van der Ouderaa
claimed_verified
2024-04-18T08:11:01.386Z
662046a80e8d8b41b2aaeefb
/avatars/45568fc404a6bc9bf9bc8b0b8ff5ba49.svg
Tycho van der Ouderaa
user
tychovdo
658e398711f68f12eae6e23f
Markus Nagel
658e398711f68f12eae6e240
Mart van Baalen
658e398711f68f12eae6e241
Yuki M. Asano
claimed_verified
2024-10-15T09:12:10.015Z
637d21239a5217b88b7549c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
Yuki Asano
user
yukimasano
658e398711f68f12eae6e242
Tijmen Blankevoort
The LLM Surgeon
State-of-the-art language models are becoming increasingly large in an effort
to achieve the highest performance on large corpora of available textual data.
However, the sheer size of the Transformer architectures makes it difficult to
deploy models within computational, environmental or device-specific
constraints. We explore data-driven compression of existing pretrained models
as an alternative to training smaller models from scratch. To do so, we scale
Kronecker-factored curvature approximations of the target loss landscape to
large language models. In doing so, we can compute both the dynamic allocation
of structures that can be removed as well as updates of remaining weights that
account for the removal. We provide a general framework for unstructured,
semi-structured and structured pruning and improve upon weight updates to
capture more correlations between weights, while remaining computationally
efficient. Experimentally, our method can prune rows and columns from a range
of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance,
and achieve state-of-the-art results in unstructured and semi-structured
pruning of large language models.
Data-driven compression using Kronecker-factored curvature approximations efficiently prunes large language models with minimal performance impact.
Kronecker-factored curvature approximations target loss landscape unstructured pruning semi-structured pruning structured pruning weight updates OPT models Llamav2-7B