LLM Pruning and Distillation in Practice: The Minitron Approach
We present a comprehensive report on compressing the Llama 3.1 8B and Mistral
NeMo 12B models to 4B and 8B parameters, respectively, using pruning and
distillation. We explore two distinct pruning strategies: (1) depth pruning and
(2) joint hidden/attention/MLP (width) pruning, and evaluate the results on
common benchmarks from the LM Evaluation Harness. The models are then aligned
with NeMo Aligner and tested in instruct-tuned versions. This approach produces
a compelling 4B model from Llama 3.1 8B and a state-of-the-art
Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo
12B. We found that with no access to the original data, it is beneficial to
slightly fine-tune teacher models on the distillation dataset. We open-source
our base model weights on Hugging Face with a permissive license.
66c6c723e8e0c0e0a2cafd1b
Sharath Turuvekere Sreenivas
claimed_verified
2025-04-16T09:42:40.199Z
66a01db4d370743fd12531c0
https://cdn-avatars.huggingface.co/v1/production/uploads/66a01db4d370743fd12531c0/de8kFO3PY7nZ2MJd2F_w2.jpeg
Sharath Turuvekere Sreenivas
user
sharathts
66c6c723e8e0c0e0a2cafd1c
Saurav Muralidharan
claimed_verified
2024-08-22T19:50:20.700Z
64caca9b4ca74090b97a8795
/avatars/310f7703ff3c07d283c02aca56fe86df.svg
Saurav Muralidharan
user
srvm
66c6c723e8e0c0e0a2cafd1d
Raviraj Joshi
admin_assigned
2024-08-30T16:28:37.622Z
64b803eb17681d64b1ba446c
/avatars/f49560794044802defd3421d6539a7dc.svg
Raviraj Joshi
user
raviraj-fk
66c6c723e8e0c0e0a2cafd1e
Marcin Chochowski
admin_assigned
2024-08-30T16:28:45.919Z
617a4ee4941993035fbaf294
/avatars/b5d0b14f7cc53dd7f4f7200228a6e342.svg
Marcin Chochowski
user
mchochowski
66c6c723e8e0c0e0a2cafd1f
Mostofa Patwary
admin_assigned
2024-08-30T16:28:51.228Z
630544b09d2531fabd156fd3
/avatars/7b5374244a887577834fb4524ff76d01.svg
Mostofa Patwary
user
mpatwary
66c6c723e8e0c0e0a2cafd20
Mohammad Shoeybi
admin_assigned
2024-08-30T16:28:57.311Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
66c6c723e8e0c0e0a2cafd21
Bryan Catanzaro
admin_assigned
2024-08-30T16:29:02.920Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66c6c723e8e0c0e0a2cafd22
Jan Kautz
66c6c723e8e0c0e0a2cafd23
Pavlo Molchanov
claimed_verified
2024-09-30T08:07:47.182Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
LLM Pruning and Distillation in Practice: The Minitron Approach
We present a comprehensive report on compressing the Llama 3.1 8B and Mistral
NeMo 12B models to 4B and 8B parameters, respectively, using pruning and
distillation. We explore two distinct pruning strategies: (1) depth pruning and
(2) joint hidden/attention/MLP (width) pruning, and evaluate the results on
common benchmarks from the LM Evaluation Harness. The models are then aligned
with NeMo Aligner and tested in instruct-tuned versions. This approach produces
a compelling 4B model from Llama 3.1 8B and a state-of-the-art
Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo
12B. We found that with no access to the original data, it is beneficial to
slightly fine-tune teacher models on the distillation dataset. We open-source
our base model weights on Hugging Face with a permissive license.
Models Llama 3.1 8B and Mistral NeMo 12B are compressed using pruning and distillation, achieving state-of-the-art performance in a reduced parameter setting.
pruning depth pruning joint hidden/attention/MLP pruning LM Evaluation Harness NeMo Aligner instruct-tuned distillation dataset parameter-efficient fine-tuning MN-Minitron-8B