Long-range Language Modeling with Self-retrieval
Retrieval-augmented language models (LMs) have received much attention
recently. However, typically the retriever is not trained jointly as a native
component of the LM, but added to an already-pretrained LM, which limits the
ability of the LM and the retriever to adapt to one another. In this work, we
propose the Retrieval-Pretrained Transformer (RPT), an architecture and
training procedure for jointly training a retrieval-augmented LM from scratch
for the task of modeling long texts. Given a recently generated text chunk in a
long document, the LM computes query representations, which are then used to
retrieve earlier chunks in the document, located potentially tens of thousands
of tokens before. Information from retrieved chunks is fused into the LM
representations to predict the next target chunk. We train the retriever
component with a semantic objective, where the goal is to retrieve chunks that
increase the probability of the next chunk, according to a reference LM. We
evaluate RPT on four long-range language modeling tasks, spanning books, code,
and mathematical writing, and demonstrate that RPT improves retrieval quality
and subsequently perplexity across the board compared to strong baselines.
64991fad42d585137eba2df4
Ohad Rubin
admin_assigned
2023-06-26T08:47:00.919Z
61d0bec503bc10eb8e1c297d
/avatars/7ae0cbe5abbfc056de47640f1592595e.svg
Ohad Rubin
user
iohadrubin
64991fad42d585137eba2df5
Jonathan Berant
admin_assigned
2023-06-26T08:48:34.915Z
60811c9a4ad99100d63ce36c
https://cdn-avatars.huggingface.co/v1/production/uploads/1619119757620-60811c9a4ad99100d63ce36c.png
Jonathan Berant
user
joberant
Long-range Language Modeling with Self-retrieval
Retrieval-augmented language models (LMs) have received much attention
recently. However, typically the retriever is not trained jointly as a native
component of the LM, but added to an already-pretrained LM, which limits the
ability of the LM and the retriever to adapt to one another. In this work, we
propose the Retrieval-Pretrained Transformer (RPT), an architecture and
training procedure for jointly training a retrieval-augmented LM from scratch
for the task of modeling long texts. Given a recently generated text chunk in a
long document, the LM computes query representations, which are then used to
retrieve earlier chunks in the document, located potentially tens of thousands
of tokens before. Information from retrieved chunks is fused into the LM
representations to predict the next target chunk. We train the retriever
component with a semantic objective, where the goal is to retrieve chunks that
increase the probability of the next chunk, according to a reference LM. We
evaluate RPT on four long-range language modeling tasks, spanning books, code,
and mathematical writing, and demonstrate that RPT improves retrieval quality
and subsequently perplexity across the board compared to strong baselines.
The Retrieval-Pretrained Transformer jointly trains a retrieval-augmented language model to model long texts by fusing retrieved information for improved prediction.
retrieval-augmented language models Retrieval-Pretrained Transformer query representations semantic objective long-range language modeling perplexity