HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference
Inho Jeong, Sunghyeon Woo, Sol Namkung, Dongsuk Jeon
Large‑language‑model (LLM) inference with long contexts often produces key–value (KV) caches whose footprint exceeds the capacity of GPU high‑bandwidth memory (HBM). Prior LLM inference frameworks such as vLLM mitigate this pressure by swapping KV cache pages to host DRAM. However, the high total cost of ownership (TCO) of large DRAM pools makes this solution economically unattractive. Although offloading to SSDs can be an alternative for reducing TCO, conventional frameworks such as FlexGen experience a substantial throughput drop since the data path that routes SSD traffic through CPU to GPU is severely bandwidth-constrained. To overcome these limitations. we introduce HiFC, a novel DRAM‑free swapping scheme that stores KV pages in the pseudo‑SLC region of a commodity NVMe SSD and transfers the KV cache to and from GPU directly through GPU-Direct-Storage (GDS). HiFC attains inference throughput compared with DRAM‑based swapping under diverse long‑context workloads while significantly lowering the memory‑related TCO of a single‑GPU system by 4.5$\times$ over three years.