Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large
  Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
The rapid advancement of large language models (LLMs) has revolutionized
natural language processing (NLP). While these models excel at understanding
and generating human-like text, their widespread deployment can be
prohibitively expensive. SortedNet is a recent training technique for enabling
dynamic inference for deep neural networks. It leverages network modularity to
create sub-models with varying computational loads, sorting them based on
computation/accuracy characteristics in a nested manner. We extend SortedNet to
generative NLP tasks, making large language models dynamic without any
pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with
Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model
efficiency, eliminating the need for multiple models for various scenarios
during inference. We show that using this approach, we are able to unlock the
potential of intermediate layers of transformers in generating the target
output. Our sub-models remain integral components of the original model,
minimizing storage requirements and transition costs between different
computational/latency budgets. By applying this approach on LLaMa 2 13B for
tuning on the Stanford Alpaca dataset and comparing it to normal tuning and
early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver
models twice as fast as the original model while maintaining or exceeding
performance.
65090daa7e0d56c2715720d3
Parsa Kavehzadeh
admin_assigned
2023-09-20T13:15:58.939Z
630e8c0f8df86f1e5bf3a510
/avatars/b9e06d64339d0ebbfc0a8f5837ca13eb.svg
Parsa Kavehzadeh
user
parsareal
65090daa7e0d56c2715720d4
Mojtaba Valipour
claimed_verified
2023-09-19T11:29:54.141Z
63ecf846f765928ceeaae5bc
https://cdn-avatars.huggingface.co/v1/production/uploads/63ecf846f765928ceeaae5bc/PKDo1dERknoi-GFjmK2_L.png
Mojtaba Valipour
user
vpcom
65090daa7e0d56c2715720d5
Marzieh Tahaei
admin_assigned
2023-09-20T13:16:06.784Z
6306331214e7d8892aaa3929
/avatars/c5c1007487f18730c3e4df9c30c18337.svg
Marzieh Tahaei
user
marzieh7
65090daa7e0d56c2715720d6
Ali Ghodsi
admin_assigned
2023-09-20T13:16:14.918Z
63bc8dc4b8c61b8aa49cbc28
/avatars/c982663d88901584d65b7393af8d14a2.svg
Ali Ghodsi
user
alighodsi
65090daa7e0d56c2715720d7
Boxing Chen
65090daa7e0d56c2715720d8
Mehdi Rezagholizadeh
admin_assigned
2023-09-20T13:16:29.230Z
6348322f0744c555b6292073
https://cdn-avatars.huggingface.co/v1/production/uploads/1665675864273-6348322f0744c555b6292073.jpeg
Mehdi Rezagholizadeh
user
mrgzadeh
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large
  Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
The rapid advancement of large language models (LLMs) has revolutionized
natural language processing (NLP). While these models excel at understanding
and generating human-like text, their widespread deployment can be
prohibitively expensive. SortedNet is a recent training technique for enabling
dynamic inference for deep neural networks. It leverages network modularity to
create sub-models with varying computational loads, sorting them based on
computation/accuracy characteristics in a nested manner. We extend SortedNet to
generative NLP tasks, making large language models dynamic without any
pretraining and by only replacing standard Supervised Fine-Tuning (SFT) with
Sorted Fine-Tuning (SoFT) at the same costs. Our approach boosts model
efficiency, eliminating the need for multiple models for various scenarios
during inference. We show that using this approach, we are able to unlock the
potential of intermediate layers of transformers in generating the target
output. Our sub-models remain integral components of the original model,
minimizing storage requirements and transition costs between different
computational/latency budgets. By applying this approach on LLaMa 2 13B for
tuning on the Stanford Alpaca dataset and comparing it to normal tuning and
early exit via PandaLM benchmark, we show that Sorted Fine-Tuning can deliver
models twice as fast as the original model while maintaining or exceeding
performance.
Sorted Fine-Tuning extends SortedNet to generative NLP tasks, enhancing efficiency and speed of large language models without the need for multiple models or complex pretraining.
SortedNet dynamic inference network modularity sub-models computation/accuracy characteristics Sorted Fine-Tuning Supervised Fine-Tuning large language models generative NLP intermediate layers transformers storage requirements LLaMa 2 Stanford Alpaca dataset PandaLM benchmark