Can Few-shot Work in Long-Context? Recycling the Context to Generate
  Demonstrations
Despite recent advancements in Large Language Models (LLMs), their
performance on tasks involving long contexts remains sub-optimal. In-Context
Learning (ICL) with few-shot examples may be an appealing solution to enhance
LLM performance in this scenario; However, naively adding ICL examples with
long context introduces challenges, including substantial token overhead added
for each few-shot example and context mismatch between the demonstrations and
the target query. In this work, we propose to automatically generate few-shot
examples for long context QA tasks by recycling contexts. Specifically, given a
long input context (1-3k tokens) and a query, we generate additional
query-output pairs from the given context as few-shot examples, while
introducing the context only once. This ensures that the demonstrations are
leveraging the same context as the target query while only adding a small
number of tokens to the prompt. We further enhance each demonstration by
instructing the model to explicitly identify the relevant paragraphs before the
answer, which improves performance while providing fine-grained attribution to
the answer source. We apply our method on multiple LLMs and obtain substantial
improvements (+23\% on average across models) on various QA datasets with long
context, especially when the answer lies within the middle of the context.
Surprisingly, despite introducing only single-hop ICL examples, LLMs also
successfully generalize to multi-hop long-context QA using our approach.
667afc55835224aefa0464db
Arie Cattan
667afc55835224aefa0464dc
Alon Jacovi
claimed_verified
2024-08-08T07:42:20.464Z
6596c90dfe0f9826a63654d0
/avatars/e616fbdff1a345ad22081f5cd019329a.svg
Alon Jacovi
user
alonjacovi
667afc55835224aefa0464dd
Alex Fabrikant
667afc55835224aefa0464de
Jonathan Herzig
667afc55835224aefa0464df
Roee Aharoni
667afc55835224aefa0464e0
Hannah Rashkin
667afc55835224aefa0464e1
Dror Marcus
667afc55835224aefa0464e2
Avinatan Hassidim
667afc55835224aefa0464e3
Yossi Matias
667afc55835224aefa0464e4
Idan Szpektor
667afc55835224aefa0464e5
Avi Caciularu
Can Few-shot Work in Long-Context? Recycling the Context to Generate
  Demonstrations
Despite recent advancements in Large Language Models (LLMs), their
performance on tasks involving long contexts remains sub-optimal. In-Context
Learning (ICL) with few-shot examples may be an appealing solution to enhance
LLM performance in this scenario; However, naively adding ICL examples with
long context introduces challenges, including substantial token overhead added
for each few-shot example and context mismatch between the demonstrations and
the target query. In this work, we propose to automatically generate few-shot
examples for long context QA tasks by recycling contexts. Specifically, given a
long input context (1-3k tokens) and a query, we generate additional
query-output pairs from the given context as few-shot examples, while
introducing the context only once. This ensures that the demonstrations are
leveraging the same context as the target query while only adding a small
number of tokens to the prompt. We further enhance each demonstration by
instructing the model to explicitly identify the relevant paragraphs before the
answer, which improves performance while providing fine-grained attribution to
the answer source. We apply our method on multiple LLMs and obtain substantial
improvements (+23\% on average across models) on various QA datasets with long
context, especially when the answer lies within the middle of the context.
Surprisingly, despite introducing only single-hop ICL examples, LLMs also
successfully generalize to multi-hop long-context QA using our approach.
A method for generating few-shot examples for long context QA tasks improves LLM performance by recycling contexts and instructing models to identify relevant paragraphs.
In-Context Learning (ICL) Large Language Models (LLMs) query-output pairs fine-grained attribution single-hop ICL multi-hop long-context QA