TeleRAG: Efficient Retrieval-Augmented Generation Inference with
  Lookahead Retrieval
Retrieval-augmented generation (RAG) extends large language models (LLMs)
with external data sources to enhance factual correctness and domain coverage.
Modern RAG pipelines rely on large datastores, leading to system challenges in
latency-sensitive deployments, especially when limited GPU memory is available.
To address these challenges, we propose TeleRAG, an efficient inference system
that reduces RAG latency with minimal GPU memory requirements. The core
innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that
anticipates required data and transfers it from CPU to GPU in parallel with LLM
generation. By leveraging the modularity of RAG pipelines, the inverted file
index (IVF) search algorithm and similarities between queries, TeleRAG
optimally overlaps data movement and computation. Experimental results show
that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average
compared to state-of-the-art systems, enabling faster, more memory-efficient
deployments of advanced RAG applications.
67c5bc8babe08983d98a4248
Chien-Yu Lin
67c5bc8babe08983d98a4249
Keisuke Kamahori
claimed_verified
2025-03-03T16:07:17.078Z
6304ac1a412a1b9d381ca378
/avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
Keisuke Kamahori
user
kamahori
67c5bc8babe08983d98a424a
Yiyu Liu
67c5bc8babe08983d98a424b
Xiaoxiang Shi
67c5bc8babe08983d98a424c
Madhav Kashyap
claimed_verified
2025-03-07T09:10:09.964Z
66451f5f27c307a7620599a9
/avatars/80b54598e28e8dfc6fd228e3b56c7330.svg
Madhav Kashyap
user
madhavmk
67c5bc8babe08983d98a424d
Yile Gu
67c5bc8babe08983d98a424e
Rulin Shao
67c5bc8babe08983d98a424f
Zihao Ye
67c5bc8babe08983d98a4250
Kan Zhu
67c5bc8babe08983d98a4251
Stephanie Wang
67c5bc8babe08983d98a4252
Arvind Krishnamurthy
67c5bc8babe08983d98a4253
Rohan Kadekodi
claimed_verified
2025-06-26T09:33:49.650Z
67b63b314dde7e635d02e61d
/avatars/cf8909aa2642294405f40abfeb6caaec.svg
Rohan Kadekodi
user
rohankadekodi
67c5bc8babe08983d98a4254
Luis Ceze
67c5bc8babe08983d98a4255
Baris Kasikci
TeleRAG: Efficient Retrieval-Augmented Generation Inference with
  Lookahead Retrieval
Retrieval-augmented generation (RAG) extends large language models (LLMs)
with external data sources to enhance factual correctness and domain coverage.
Modern RAG pipelines rely on large datastores, leading to system challenges in
latency-sensitive deployments, especially when limited GPU memory is available.
To address these challenges, we propose TeleRAG, an efficient inference system
that reduces RAG latency with minimal GPU memory requirements. The core
innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that
anticipates required data and transfers it from CPU to GPU in parallel with LLM
generation. By leveraging the modularity of RAG pipelines, the inverted file
index (IVF) search algorithm and similarities between queries, TeleRAG
optimally overlaps data movement and computation. Experimental results show
that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average
compared to state-of-the-art systems, enabling faster, more memory-efficient
deployments of advanced RAG applications.
TeleRAG reduces RAG inference latency and GPU memory usage by using lookahead retrieval to prefetch data from CPU to GPU in parallel with LLM generation.
Retrieval-augmented generation large language models external data sources prefetching mechanism lookahead retrieval inverted file index data movement computation LLM generation