LLM in a flash: Efficient Large Language Model Inference with Limited
  Memory
Large language models (LLMs) are central to modern natural language
processing, delivering exceptional performance in various tasks. However, their
intensive computational and memory requirements present challenges, especially
for devices with limited DRAM capacity. This paper tackles the challenge of
efficiently running LLMs that exceed the available DRAM capacity by storing the
model parameters on flash memory but bringing them on demand to DRAM. Our
method involves constructing an inference cost model that harmonizes with the
flash memory behavior, guiding us to optimize in two critical areas: reducing
the volume of data transferred from flash and reading data in larger, more
contiguous chunks. Within this flash memory-informed framework, we introduce
two principal techniques. First, "windowing'" strategically reduces data
transfer by reusing previously activated neurons, and second, "row-column
bundling", tailored to the sequential data access strengths of flash memory,
increases the size of data chunks read from flash memory. These methods
collectively enable running models up to twice the size of the available DRAM,
with a 4-5x and 20-25x increase in inference speed compared to naive loading
approaches in CPU and GPU, respectively. Our integration of sparsity awareness,
context-adaptive loading, and a hardware-oriented design paves the way for
effective inference of LLMs on devices with limited memory.
6582524a5f6d8343825e4af2
Keivan Alizadeh
extracted_confirmed
2024-04-23T19:22:30.221Z
66280a4c7c988f33b795f90f
/avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
Iman Mirzadeh
user
imirzadeh-apple
6582524a5f6d8343825e4af3
Iman Mirzadeh
6582524a5f6d8343825e4af4
Dmitry Belenko
admin_assigned
2023-12-20T08:59:28.800Z
63185e1a70b013d6609416ba
https://cdn-avatars.huggingface.co/v1/production/uploads/63185e1a70b013d6609416ba/LOB6bJ71wwgamgN3MCuFb.jpeg
Dmitry Belenko
user
depthwise
6582524a5f6d8343825e4af5
Karen Khatamifard
6582524a5f6d8343825e4af6
Minsik Cho
6582524a5f6d8343825e4af7
Carlo C Del Mundo
6582524a5f6d8343825e4af8
Mohammad Rastegari
6582524a5f6d8343825e4af9
Mehrdad Farajtabar
admin_assigned
2023-12-20T08:21:56.790Z
62d188a6d50433c35eb14020
/avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
Mehrdad Farajtabar
user
mfarajtabar
LLM in a flash: Efficient Large Language Model Inference with Limited
  Memory
Large language models (LLMs) are central to modern natural language
processing, delivering exceptional performance in various tasks. However, their
intensive computational and memory requirements present challenges, especially
for devices with limited DRAM capacity. This paper tackles the challenge of
efficiently running LLMs that exceed the available DRAM capacity by storing the
model parameters on flash memory but bringing them on demand to DRAM. Our
method involves constructing an inference cost model that harmonizes with the
flash memory behavior, guiding us to optimize in two critical areas: reducing
the volume of data transferred from flash and reading data in larger, more
contiguous chunks. Within this flash memory-informed framework, we introduce
two principal techniques. First, "windowing'" strategically reduces data
transfer by reusing previously activated neurons, and second, "row-column
bundling", tailored to the sequential data access strengths of flash memory,
increases the size of data chunks read from flash memory. These methods
collectively enable running models up to twice the size of the available DRAM,
with a 4-5x and 20-25x increase in inference speed compared to naive loading
approaches in CPU and GPU, respectively. Our integration of sparsity awareness,
context-adaptive loading, and a hardware-oriented design paves the way for
effective inference of LLMs on devices with limited memory.
Efficient inference for large language models on devices with limited DRAM by optimizing data transfer and access from flash memory.
large language models (LLMs) flash memory DRAM inference cost model data transfer row-column bundling sparsity awareness context-adaptive loading hardware-oriented design