Distributed Inference and Fine-tuning of Large Language Models Over The
  Internet
Large language models (LLMs) are useful in many NLP tasks and become more
capable with size, with the best open-source models having over 50 billion
parameters. However, using these 50B+ models requires high-end hardware, making
them inaccessible to most researchers. In this work, we investigate methods for
cost-efficient inference and fine-tuning of LLMs, comparing local and
distributed strategies. We observe that a large enough model (50B+) can run
efficiently even on geodistributed devices in a consumer-grade network. This
could allow running LLM efficiently by pooling together idle compute resources
of multiple research groups and volunteers. We address two open problems: (1)
how to perform inference and fine-tuning reliably if any device can disconnect
abruptly and (2) how to partition LLMs between devices with uneven hardware,
joining and leaving at will. In order to do that, we develop special
fault-tolerant inference algorithms and load-balancing protocols that
automatically assign devices to maximize the total system throughput. We
showcase these algorithms in Petals - a decentralized system that runs Llama 2
(70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
interactive generation. We evaluate the performance of our system in simulated
conditions and a real-world setup spanning two continents.
657a6c1f857f942849b2ee22
Alexander Borzunov
admin_assigned
2023-12-14T13:34:29.506Z
609147936885f3a4314a6aa2
https://cdn-avatars.huggingface.co/v1/production/uploads/1620133776745-noauth.jpeg
Alexander Borzunov
user
borzunov
657a6c1f857f942849b2ee23
Max Ryabinin
admin_assigned
2023-12-14T13:34:36.948Z
607d59fb921db717010c7ccc
https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
Max Ryabinin
user
mryab
657a6c1f857f942849b2ee24
Artem Chumachenko
admin_assigned
2023-12-14T13:34:43.404Z
6329ce19cf7d40df9b3a8f0e
https://cdn-avatars.huggingface.co/v1/production/uploads/1663684111181-noauth.jpeg
Artem Chumachenko
user
artek0chumak
657a6c1f857f942849b2ee25
Dmitry Baranchuk
admin_assigned
2023-12-14T13:34:51.131Z
62b6cc49752323892323bc04
https://cdn-avatars.huggingface.co/v1/production/uploads/62b6cc49752323892323bc04/gGBld1KJIP9AIpd81L3PC.jpeg
Dmitry Baranchuk
user
dbaranchuk
657a6c1f857f942849b2ee26
Tim Dettmers
admin_assigned
2023-12-14T13:34:56.724Z
61a8f52d89ea472e600a0a10
https://cdn-avatars.huggingface.co/v1/production/uploads/1638462736111-noauth.png
Tim Dettmers
user
timdettmers
657a6c1f857f942849b2ee27
Younes Belkada
admin_assigned
2023-12-14T13:35:04.275Z
62441d1d9fdefb55a0b7d12c
https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
Younes B
user
ybelkada
657a6c1f857f942849b2ee28
Pavel Samygin
657a6c1f857f942849b2ee29
Colin Raffel
admin_assigned
2023-12-14T13:35:18.194Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
Distributed Inference and Fine-tuning of Large Language Models Over The
  Internet
Large language models (LLMs) are useful in many NLP tasks and become more
capable with size, with the best open-source models having over 50 billion
parameters. However, using these 50B+ models requires high-end hardware, making
them inaccessible to most researchers. In this work, we investigate methods for
cost-efficient inference and fine-tuning of LLMs, comparing local and
distributed strategies. We observe that a large enough model (50B+) can run
efficiently even on geodistributed devices in a consumer-grade network. This
could allow running LLM efficiently by pooling together idle compute resources
of multiple research groups and volunteers. We address two open problems: (1)
how to perform inference and fine-tuning reliably if any device can disconnect
abruptly and (2) how to partition LLMs between devices with uneven hardware,
joining and leaving at will. In order to do that, we develop special
fault-tolerant inference algorithms and load-balancing protocols that
automatically assign devices to maximize the total system throughput. We
showcase these algorithms in Petals - a decentralized system that runs Llama 2
(70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
interactive generation. We evaluate the performance of our system in simulated
conditions and a real-world setup spanning two continents.
A decentralized system pools idle compute resources to enable efficient inference and fine-tuning of large language models over a consumer-grade network, showcasing significant performance improvements.
large language models cost-efficient inference fine-tuning geodistributed devices fault-tolerant inference load-balancing protocols Petals Llama 2 BLOOM