PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday
  Home Clusters
Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers
for running frontier large language models (LLMs) on home devices. While
consumer hardware is getting stronger and model quantization is improving,
existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high
bandwidth, far beyond what a common home cluster can handle. This paper
introduces prima.cpp, a distributed inference system that runs 70B-scale models
on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and
cross-platform support. It uses mmap to manage model weights and introduces
piped-ring parallelism with prefetching to hide disk loading. By modeling
heterogeneity in computation, communication, disk, memory (and its management
behavior), and OS, it optimally assigns model layers to each device's CPU and
GPU, further reducing token latency. An elegant algorithm named Halda is
proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a
common four-node home cluster. It outperforms llama.cpp, exo, and dllama on
30B+ models while keeping memory pressure below 6%. This brings frontier
30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home
assistants, making advanced AI truly accessible to individuals. The code is
open source and available at https://github.com/Lizonghang/prima.cpp.
67fdbde764a418633ee9fa1b
Zonghang Li
claimed_verified
2025-04-15T07:54:53.716Z
647466b8b68461d5cf795e3c
https://cdn-avatars.huggingface.co/v1/production/uploads/647466b8b68461d5cf795e3c/zaK6sdCbdPfYu14vg2Ty6.png
LIKirin
user
LIKirin
67fdbde764a418633ee9fa1c
Tao Li
claimed_verified
2025-04-15T07:54:50.518Z
669e0c108b279f0a2704a5ba
https://cdn-avatars.huggingface.co/v1/production/uploads/669e0c108b279f0a2704a5ba/uPyioyK0RWSG7CT52xBao.png
TaoLi
user
LiPhilip
67fdbde764a418633ee9fa1d
Wenjiao Feng
admin_assigned
2025-04-15T08:15:55.453Z
6513c6c9c1fbded3b1977acc
/avatars/d169b3462f952c5639e1471ed8bf84c9.svg
Wenjiao Feng
user
NeuronNomad
67fdbde764a418633ee9fa1e
Mohsen Guizani
67fdbde764a418633ee9fa1f
Hongfang Yu
PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday
  Home Clusters
Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers
for running frontier large language models (LLMs) on home devices. While
consumer hardware is getting stronger and model quantization is improving,
existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high
bandwidth, far beyond what a common home cluster can handle. This paper
introduces prima.cpp, a distributed inference system that runs 70B-scale models
on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and
cross-platform support. It uses mmap to manage model weights and introduces
piped-ring parallelism with prefetching to hide disk loading. By modeling
heterogeneity in computation, communication, disk, memory (and its management
behavior), and OS, it optimally assigns model layers to each device's CPU and
GPU, further reducing token latency. An elegant algorithm named Halda is
proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a
common four-node home cluster. It outperforms llama.cpp, exo, and dllama on
30B+ models while keeping memory pressure below 6%. This brings frontier
30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home
assistants, making advanced AI truly accessible to individuals. The code is
open source and available at https://github.com/Lizonghang/prima.cpp.
Prima.cpp, a distributed inference system, enables running large language models on home devices using CPU/GPU and low memory, overcoming hardware limitations.
LLMs large language models mmap piped-ring parallelism prefetching token latency Halda home cluster memory pressure