OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs
Despite the recent advancements in Large Language Models (LLMs), which have
significantly enhanced the generative capabilities for various NLP tasks, LLMs
still face limitations in directly handling retrieval tasks. However, many
practical applications demand the seamless integration of both retrieval and
generation. This paper introduces a novel and efficient One-pass Generation and
retrieval framework (OneGen), designed to improve LLMs' performance on tasks
that require both generation and retrieval. The proposed framework bridges the
traditionally separate training approaches for generation and retrieval by
incorporating retrieval tokens generated autoregressively. This enables a
single LLM to handle both tasks simultaneously in a unified forward pass. We
conduct experiments on two distinct types of composite tasks, RAG and Entity
Linking, to validate the pluggability, effectiveness, and efficiency of OneGen
in training and inference. Furthermore, our results show that integrating
generation and retrieval within the same context preserves the generative
capabilities of LLMs while improving retrieval performance. To the best of our
knowledge, OneGen is the first to enable LLMs to conduct vector retrieval
during the generation.
66dfdaf8c8818243494a6da4
Jintian Zhang
admin_assigned
2024-09-10T08:46:24.769Z
63f89fe7565506a9cadcd2cf
/avatars/7eb449a1109dcff051cb3ba680f0c082.svg
Jintian Zhang
user
MikeDean
66dfdaf8c8818243494a6da5
Cheng Peng
66dfdaf8c8818243494a6da6
Mengshu Sun
admin_assigned
2024-09-10T08:47:03.170Z
64f6d1ed46284aa28d9abf6c
/avatars/d6beaecfd00345e4a664862fff217427.svg
Sun mengshu
user
sunmengshu
66dfdaf8c8818243494a6da7
Xiang Chen
admin_assigned
2024-09-10T08:47:46.247Z
64c073417255ff868721fb54
/avatars/c2a00114598aa89f071b937f0af4077f.svg
Xiang CHEN
user
xiangchen-dvi
66dfdaf8c8818243494a6da8
Lei Liang
admin_assigned
2024-09-10T08:47:57.401Z
64588a9caa426bae5e080567
/avatars/76c11bfbe64058014d4b2b89d18ba2e4.svg
Lei Liang
user
square0083
66dfdaf8c8818243494a6da9
Zhiqiang Zhang
66dfdaf8c8818243494a6daa
Jun Zhou
66dfdaf8c8818243494a6dab
Huajun Chen
admin_assigned
2024-09-10T08:49:05.458Z
64931296137833d7ec7689cd
https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
Huajun Chen
user
huajunsir
66dfdaf8c8818243494a6dac
Ningyu Zhang
claimed_verified
2024-09-10T07:10:27.839Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs
Despite the recent advancements in Large Language Models (LLMs), which have
significantly enhanced the generative capabilities for various NLP tasks, LLMs
still face limitations in directly handling retrieval tasks. However, many
practical applications demand the seamless integration of both retrieval and
generation. This paper introduces a novel and efficient One-pass Generation and
retrieval framework (OneGen), designed to improve LLMs' performance on tasks
that require both generation and retrieval. The proposed framework bridges the
traditionally separate training approaches for generation and retrieval by
incorporating retrieval tokens generated autoregressively. This enables a
single LLM to handle both tasks simultaneously in a unified forward pass. We
conduct experiments on two distinct types of composite tasks, RAG and Entity
Linking, to validate the pluggability, effectiveness, and efficiency of OneGen
in training and inference. Furthermore, our results show that integrating
generation and retrieval within the same context preserves the generative
capabilities of LLMs while improving retrieval performance. To the best of our
knowledge, OneGen is the first to enable LLMs to conduct vector retrieval
during the generation.
OneGen is a framework that integrates retrieval and generation in a single pass within Large Language Models, improving retrieval performance without compromising generative capabilities.
Large Language Models One-pass Generation retrieval tokens autoregressive generation unified forward pass RAG Entity Linking vector retrieval