SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Large language models have become the cornerstone of natural language
processing, but their use comes with substantial costs in terms of compute and
memory resources. Sparsification provides a solution to alleviate these
resource constraints, and recent works have shown that trained models can be
sparsified post-hoc. Existing sparsification techniques face challenges as they
need additional data structures and offer constrained speedup with current
hardware. In this paper we present SliceGPT, a new post-training sparsification
scheme which replaces each weight matrix with a smaller (dense) matrix,
reducing the embedding dimension of the network. Through extensive
experimentation, we show that SliceGPT can remove up to 25% of the model
parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models
while maintaining 99%, 99% and 90% zero-shot task performance of the dense
model respectively. Our sliced models run on fewer GPUs and run faster without
any additional code optimization: on 24GB consumer GPUs we reduce the total
compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB
A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance
in transformer networks, which enables SliceGPT and we hope it will inspire and
enable future avenues to reduce memory and computation demands for pre-trained
models. Code is available at:
https://github.com/microsoft/TransformerCompression
65b70e8c4839ccc503a60510
Saleh Ashkboos
admin_assigned
2024-01-29T09:32:52.773Z
610d7ec84539844d377af78a
https://cdn-avatars.huggingface.co/v1/production/uploads/610d7ec84539844d377af78a/-9H0bIv5pjI9hJamg9P0x.jpeg
Ashkboos
user
Saleh
65b70e8c4839ccc503a60511
Maximilian L. Croci
claimed_verified
2024-01-29T11:13:59.051Z
63eb657213a3eb9b0dc6d628
/avatars/c4a9f974dd5d2ea67c318d8496e6248d.svg
Max Croci
user
nailimixaM
65b70e8c4839ccc503a60512
Marcelo Gennari do Nascimento
extracted_confirmed
2024-01-29T09:56:30.666Z
6329e25259950c1d279b4333
/avatars/d24a0cf3e09fcfd77bc2a28dd9e2a200.svg
Marcelo Gennari do Nascimento
user
MarceloGennari
65b70e8c4839ccc503a60513
Torsten Hoefler
65b70e8c4839ccc503a60514
James Hensman
extracted_confirmed
2024-04-04T06:18:21.485Z
65af77603db2280ece8a6644
/avatars/37fac7c73e0190f933201398c74bcc50.svg
James Hensman
user
Jameshensman
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
Large language models have become the cornerstone of natural language
processing, but their use comes with substantial costs in terms of compute and
memory resources. Sparsification provides a solution to alleviate these
resource constraints, and recent works have shown that trained models can be
sparsified post-hoc. Existing sparsification techniques face challenges as they
need additional data structures and offer constrained speedup with current
hardware. In this paper we present SliceGPT, a new post-training sparsification
scheme which replaces each weight matrix with a smaller (dense) matrix,
reducing the embedding dimension of the network. Through extensive
experimentation, we show that SliceGPT can remove up to 25% of the model
parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models
while maintaining 99%, 99% and 90% zero-shot task performance of the dense
model respectively. Our sliced models run on fewer GPUs and run faster without
any additional code optimization: on 24GB consumer GPUs we reduce the total
compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB
A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance
in transformer networks, which enables SliceGPT and we hope it will inspire and
enable future avenues to reduce memory and computation demands for pre-trained
models. Code is available at:
https://github.com/microsoft/TransformerCompression
SliceGPT, a novel post-training sparsification method, reduces model parameters by up to 25% without significant performance decrease, optimizing inference speed and GPU usage on large language models.
sparsification post-training weight matrix embedding dimension transformer networks computational invariance