TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
Large model inference is shifting from cloud to edge due to concerns about
the privacy of user interaction data. However, edge devices often struggle with
limited computing power, memory, and bandwidth, requiring collaboration across
multiple devices to run and speed up LLM inference. Pipeline parallelism, the
mainstream solution, is inefficient for single-user scenarios, while tensor
parallelism struggles with frequent communications. In this paper, we argue
that tensor parallelism can be more effective than pipeline on low-resource
devices, and present a compute- and memory-efficient tensor parallel inference
system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw
data local in the users' devices and introduces a sliding window memory
scheduler to dynamically manage layer weights during inference, with disk I/O
latency overlapped with the computation and communication. This allows larger
models to run smoothly on memory-limited devices. We analyze the communication
bottleneck and find that link latency, not bandwidth, emerges as the main
issue, so a star-based allreduce algorithm is implemented. Through extensive
experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80%
less time-to-first-token and token latency compared to Accelerate, and over 90%
compared to Transformers and Galaxy, while cutting the peak memory footprint of
Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.
66fcb61ce97377a6e456c1f9
Zonghang Li
claimed_verified
2024-10-02T09:13:26.241Z
647466b8b68461d5cf795e3c
https://cdn-avatars.huggingface.co/v1/production/uploads/647466b8b68461d5cf795e3c/zaK6sdCbdPfYu14vg2Ty6.png
LIKirin
user
LIKirin
66fcb61ce97377a6e456c1fa
Wenjiao Feng
admin_assigned
2024-10-02T08:40:51.106Z
6513c6c9c1fbded3b1977acc
/avatars/d169b3462f952c5639e1471ed8bf84c9.svg
Wenjiao Feng
user
NeuronNomad
66fcb61ce97377a6e456c1fb
Mohsen Guizani
66fcb61ce97377a6e456c1fc
Hongfang Yu
TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
Large model inference is shifting from cloud to edge due to concerns about
the privacy of user interaction data. However, edge devices often struggle with
limited computing power, memory, and bandwidth, requiring collaboration across
multiple devices to run and speed up LLM inference. Pipeline parallelism, the
mainstream solution, is inefficient for single-user scenarios, while tensor
parallelism struggles with frequent communications. In this paper, we argue
that tensor parallelism can be more effective than pipeline on low-resource
devices, and present a compute- and memory-efficient tensor parallel inference
system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw
data local in the users' devices and introduces a sliding window memory
scheduler to dynamically manage layer weights during inference, with disk I/O
latency overlapped with the computation and communication. This allows larger
models to run smoothly on memory-limited devices. We analyze the communication
bottleneck and find that link latency, not bandwidth, emerges as the main
issue, so a star-based allreduce algorithm is implemented. Through extensive
experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80%
less time-to-first-token and token latency compared to Accelerate, and over 90%
compared to Transformers and Galaxy, while cutting the peak memory footprint of
Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.
A tensor parallel inference system named TPI-LLM improves large language model (LLM) inference on edge devices by minimizing memory usage and reducing latency.
large model inference edge devices pipeline parallelism tensor parallelism sliding window memory scheduler disk I/O latency star-based allreduce algorithm time-to-first-token token latency peak memory footprint