Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
Large language models (LLMs) have demonstrated remarkable capabilities, but
their adoption is limited by high computational costs during inference. While
increasing parameter counts enhances accuracy, it also widens the gap between
state-of-the-art capabilities and practical deployability. We present Puzzle, a
framework to accelerate LLM inference on specific hardware while preserving
their capabilities. Through an innovative application of neural architecture
search (NAS) at an unprecedented scale, Puzzle systematically optimizes models
with tens of billions of parameters under hardware constraints. Our approach
utilizes blockwise local knowledge distillation (BLD) for parallel architecture
exploration and employs mixed-integer programming for precise constraint
optimization.
  We demonstrate the real-world impact of our framework through
Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model
derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference
throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4%
of the original model's capabilities. Nemotron-51B currently stands as the most
accurate language model capable of inference on a single GPU with large batch
sizes. Remarkably, this transformation required just 45B training tokens,
compared to over 15T tokens used for the 70B model it was derived from. This
establishes a new paradigm where powerful models can be optimized for efficient
deployment with only negligible compromise of their capabilities, demonstrating
that inference performance, not parameter count alone, should guide model
selection. With the release of Nemotron-51B and the presentation of the Puzzle
framework, we provide practitioners immediate access to state-of-the-art
language modeling capabilities at significantly reduced computational costs.
674d3de96ab71cc666cce97e
Akhiad Bercovich
admin_assigned
2024-12-02T09:21:17.415Z
66857bd849a4ed9de4c31936
/avatars/f6f016bf36fad5b29f30fbec6cde3e4d.svg
Akhiad Bercovich
user
abercovich
674d3de96ab71cc666cce97f
Tomer Ronen
admin_assigned
2024-12-02T09:21:40.810Z
6671634f1820f293a9995b12
/avatars/50c8f7b4bfb00f2169b808f3c72c7686.svg
Tomer Ronen
user
tomer-nv
674d3de96ab71cc666cce980
Talor Abramovich
admin_assigned
2024-12-02T09:21:55.666Z
668e7a379ab7631fcd434ee7
/avatars/68ddbc5e35125a779655e9033d24afe5.svg
Talor Abramovich
user
talor-abr
674d3de96ab71cc666cce981
Nir Ailon
admin_assigned
2024-12-02T09:22:03.393Z
66bf1cee43a701a83775810f
/avatars/3b0cadcc77905facee1ae3511ec28999.svg
Nir Ailon
user
nailon-nvidia
674d3de96ab71cc666cce982
Nave Assaf
admin_assigned
2024-12-02T09:22:09.140Z
64df9df14a8463e302eb69c6
/avatars/db2151d46aaac236ce371cdd9d3b2302.svg
Nave Assaf
user
nav4
674d3de96ab71cc666cce983
Mohammad Dabbah
admin_assigned
2024-12-02T09:22:15.162Z
63a4182f84a6a25c65bcfc71
/avatars/9e9c700ed5b06a2388a9bb0e93d5beb1.svg
mohammad Dabbah
user
mdabbah
674d3de96ab71cc666cce984
Ido Galil
admin_assigned
2024-12-02T09:22:30.830Z
66867535e44c9931f329ca43
/avatars/ec6fc05b9f596f1ce38ec061d3d1d230.svg
Ido Galil
user
IdoGalilNvidia
674d3de96ab71cc666cce985
Amnon Geifman
admin_assigned
2024-12-02T09:22:39.096Z
65006cac12c1442d993d6d51
/avatars/6700109303b902d453f3d8e2b45a103f.svg
Geifman
user
AmnonGeifman
674d3de96ab71cc666cce986
Yonatan Geifman
admin_assigned
2024-12-02T09:22:56.831Z
66efe3f10989ae1ac1a1795f
/avatars/c798905aeaa0fca51fb0f7b1edeabdc2.svg
Yonatan Geifman
user
ygeifman
674d3de96ab71cc666cce987
Izhak Golan
claimed_verified
2024-12-02T12:52:28.542Z
66cb4b53d9772842192b84c3
/avatars/f4820f60d509c3fa5f94ab72eff854c6.svg
Izik Golan
user
izikg
674d3de96ab71cc666cce988
Netanel Haber
674d3de96ab71cc666cce989
Ehud Karpas
674d3de96ab71cc666cce98a
Itay Levy
admin_assigned
2024-12-02T09:31:54.896Z
668578fdd24e614fec97eac8
https://cdn-avatars.huggingface.co/v1/production/uploads/668578fdd24e614fec97eac8/n5xYnqo5nQbVX2tgaRfEi.jpeg
Itay Levy
user
itlevy
674d3de96ab71cc666cce98b
Shahar Mor
admin_assigned
2024-12-02T09:32:05.610Z
657018a04f3a65f869b10e21
/avatars/715630dc868a728a7866d68fdf9f9d26.svg
Shahar Mor
user
ShaharM98
674d3de96ab71cc666cce98c
Zach Moshe
674d3de96ab71cc666cce98d
Najeeb Nabwani
admin_assigned
2024-12-02T09:32:53.456Z
6685860d852db86b9c91a1f3
/avatars/ea25dd7427375b423af9c7b412826fef.svg
Najeeb Nabwani
user
Najeebnv
674d3de96ab71cc666cce98e
Omri Puny
674d3de96ab71cc666cce98f
Ran Rubin
admin_assigned
2024-12-02T09:34:35.218Z
65059361ffc738079c5d5583
/avatars/03361adea86fa519941a79f8171d867f.svg
Ran Rubin
user
ranrubin
674d3de96ab71cc666cce990
Itamar Schen
admin_assigned
2024-12-02T09:34:29.156Z
665f0a46f065b1d42806000d
/avatars/927f042a3c95c5846621e2a381c66bbf.svg
Itamar Schen
user
ischen-nvidia
674d3de96ab71cc666cce991
Ido Shahaf
admin_assigned
2024-12-02T09:34:21.829Z
666ef13c14f1c262feeb706c
/avatars/7dca59acf5e069d96bdbb98dace9199b.svg
Ido Shahaf
user
ishahaf
674d3de96ab71cc666cce992
Oren Tropp
admin_assigned
2024-12-02T09:34:14.298Z
66b089f14ae4ae811218cdb6
/avatars/a50fe725922dfdbe0e731fade381b22e.svg
Oren Tropp
user
otropp
674d3de96ab71cc666cce993
Omer Ullman Argov
admin_assigned
2024-12-02T09:33:34.295Z
66a0d0b2d4ab10840b5af2b0
/avatars/8fa12818329de9457f93081e8f345117.svg
Omer Ullman Argov
user
oargov-nv
674d3de96ab71cc666cce994
Ran Zilberstein
admin_assigned
2024-12-02T09:33:26.752Z
666027917c3f9c72113cc75c
/avatars/a276ebe8e2731b6a05e3c61c2ae0ddae.svg
Ran Zilberstein
user
RanZilberstein-Nvidia
674d3de96ab71cc666cce995
Ran El-Yaniv
admin_assigned
2024-12-02T09:33:16.354Z
66efd764155c986c6511b52d
/avatars/711f9c6954a8623a9e014c0c563ca708.svg
Ran El-Yaniv
user
relyaniv
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
Large language models (LLMs) have demonstrated remarkable capabilities, but
their adoption is limited by high computational costs during inference. While
increasing parameter counts enhances accuracy, it also widens the gap between
state-of-the-art capabilities and practical deployability. We present Puzzle, a
framework to accelerate LLM inference on specific hardware while preserving
their capabilities. Through an innovative application of neural architecture
search (NAS) at an unprecedented scale, Puzzle systematically optimizes models
with tens of billions of parameters under hardware constraints. Our approach
utilizes blockwise local knowledge distillation (BLD) for parallel architecture
exploration and employs mixed-integer programming for precise constraint
optimization.
  We demonstrate the real-world impact of our framework through
Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model
derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference
throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4%
of the original model's capabilities. Nemotron-51B currently stands as the most
accurate language model capable of inference on a single GPU with large batch
sizes. Remarkably, this transformation required just 45B training tokens,
compared to over 15T tokens used for the 70B model it was derived from. This
establishes a new paradigm where powerful models can be optimized for efficient
deployment with only negligible compromise of their capabilities, demonstrating
that inference performance, not parameter count alone, should guide model
selection. With the release of Nemotron-51B and the presentation of the Puzzle
framework, we provide practitioners immediate access to state-of-the-art
language modeling capabilities at significantly reduced computational costs.
Puzzle is a framework that optimizes large language model inference using neural architecture search, blockwise local knowledge distillation, and mixed-integer programming to achieve faster throughput and reduced computational costs while preserving model accuracy.
neural architecture search blockwise local knowledge distillation mixed-integer programming large language models LLM inference throughput single GPU hardware constraints model capabilities state-of-the-art language model computational costs parameter count inference performance