The Unreasonable Ineffectiveness of the Deeper Layers
We empirically study a simple layer-pruning strategy for popular families of
open-weight pretrained LLMs, finding minimal degradation of performance on
different question-answering benchmarks until after a large fraction (up to
half) of the layers are removed. To prune these models, we identify the optimal
block of layers to prune by considering similarity across layers; then, to
"heal" the damage, we perform a small amount of finetuning. In particular, we
use parameter-efficient finetuning (PEFT) methods, specifically quantization
and Low Rank Adapters (QLoRA), such that each of our experiments can be
performed on a single A100 GPU. From a practical perspective, these results
suggest that layer pruning methods can complement other PEFT strategies to
further reduce computational resources of finetuning on the one hand, and can
improve the memory and latency of inference on the other hand. From a
scientific perspective, the robustness of these LLMs to the deletion of layers
implies either that current pretraining methods are not properly leveraging the
parameters in the deeper layers of the network or that the shallow layers play
a critical role in storing knowledge.
6603925f759b2e212d08b6bb
Andrey Gromov
extracted_pending
2024-03-27T03:28:32.198Z
64d198292f92537fbc41fe86
/avatars/1705d81d0b16d992833069150796b1d6.svg
Andrey Gromov
user
gromovand
6603925f759b2e212d08b6bc
Kushal Tirumala
admin_assigned
2024-03-27T09:01:03.657Z
6429dac775bcc24c5e579f5c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GI73-WJ5WjdB-KUUNGkdv.jpeg
Kushal Tirumala
user
kushaltirumala
6603925f759b2e212d08b6bd
Hassan Shapourian
admin_assigned
2024-03-27T09:01:09.596Z
64d3dabaf63b01b7f66b7f98
/avatars/ba34c4fb3b99de36522d0074725750c1.svg
Hassan Shapourian
user
hassansh
6603925f759b2e212d08b6be
Paolo Glorioso
admin_assigned
2024-03-27T09:01:15.478Z
64e40335c0edca443ef8af3e
/avatars/c42a72a448dcb9e8bb3a0b41d1b6397a.svg
Paolo Glorioso
user
pglo
6603925f759b2e212d08b6bf
Daniel A. Roberts
extracted_confirmed
2024-03-27T03:33:10.995Z
64ee5701f5451e99166f50ad
/avatars/aaa21574bb8aacdc58dc412f5292a260.svg
Dan Roberts
user
danintheory
The Unreasonable Ineffectiveness of the Deeper Layers
We empirically study a simple layer-pruning strategy for popular families of
open-weight pretrained LLMs, finding minimal degradation of performance on
different question-answering benchmarks until after a large fraction (up to
half) of the layers are removed. To prune these models, we identify the optimal
block of layers to prune by considering similarity across layers; then, to
"heal" the damage, we perform a small amount of finetuning. In particular, we
use parameter-efficient finetuning (PEFT) methods, specifically quantization
and Low Rank Adapters (QLoRA), such that each of our experiments can be
performed on a single A100 GPU. From a practical perspective, these results
suggest that layer pruning methods can complement other PEFT strategies to
further reduce computational resources of finetuning on the one hand, and can
improve the memory and latency of inference on the other hand. From a
scientific perspective, the robustness of these LLMs to the deletion of layers
implies either that current pretraining methods are not properly leveraging the
parameters in the deeper layers of the network or that the shallow layers play
a critical role in storing knowledge.
Layer pruning of pre-trained LLMs with parameter-efficient finetuning methods shows minimal performance degradation and significant resource savings in both finetuning and inference.
layer-pruning open-weight pretrained LLMs parameter-efficient finetuning PEFT quantization Low Rank Adapters QLoRA shallow layers robustness