Make Your LLM Fully Utilize the Context
While many contemporary large language models (LLMs) can process lengthy
input, they still struggle to fully utilize information within the long
context, known as the lost-in-the-middle challenge. We hypothesize that it
stems from insufficient explicit supervision during the long-context training,
which fails to emphasize that any position in a long context can hold crucial
information. Based on this intuition, our study presents information-intensive
(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer
dataset, where the answer requires (1) fine-grained information awareness on a
short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
and (2) the integration and reasoning of information from two or more short
segments. Through applying this information-intensive training on Mistral-7B,
we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
FILM-7B for utilizing long contexts, we design three probing tasks that
encompass various context styles (document, code, and structured-data context)
and information retrieval patterns (forward, backward, and bi-directional
retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
information from different positions in its 32K context window. Beyond these
probing tasks, FILM-7B significantly improves the performance on real-world
long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
662b1d65329e3263432600f7
Shengnan An
claimed_verified
2024-04-26T09:04:00.237Z
64db5f5dd68a6ddcc7bd89e9
/avatars/69375ec915927b855813df8a6d486837.svg
Shengnan An
user
ShengnanAn
662b1d65329e3263432600f8
Zexiong Ma
claimed_verified
2024-09-30T08:09:57.458Z
654da66fb36f85a025bc24b6
/avatars/e5542856ab4bf1845e8f546b5f17cd99.svg
Zexiong Ma
user
mizersy
662b1d65329e3263432600f9
Zeqi Lin
admin_assigned
2024-04-26T09:08:12.357Z
642a862525442667e9ba5e78
/avatars/f15bba13cb4096f6b3f3508ac2c56e5d.svg
Zeqi Lin
user
linzeqipku
662b1d65329e3263432600fa
Nanning Zheng
662b1d65329e3263432600fb
Jian-Guang Lou
admin_assigned
2024-04-26T09:08:28.504Z
6554338ba4de813482e91ad5
/avatars/bee879fa4a818fd6c070ee001769ce05.svg
Jian-Guang Lou
user
substill
Make Your LLM Fully Utilize the Context
While many contemporary large language models (LLMs) can process lengthy
input, they still struggle to fully utilize information within the long
context, known as the lost-in-the-middle challenge. We hypothesize that it
stems from insufficient explicit supervision during the long-context training,
which fails to emphasize that any position in a long context can hold crucial
information. Based on this intuition, our study presents information-intensive
(IN2) training, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer
dataset, where the answer requires (1) fine-grained information awareness on a
short segment (~128 tokens) within a synthesized long context (4K-32K tokens),
and (2) the integration and reasoning of information from two or more short
segments. Through applying this information-intensive training on Mistral-7B,
we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the ability of
FILM-7B for utilizing long contexts, we design three probing tasks that
encompass various context styles (document, code, and structured-data context)
and information retrieval patterns (forward, backward, and bi-directional
retrieval). The probing results demonstrate that FILM-7B can robustly retrieve
information from different positions in its 32K context window. Beyond these
probing tasks, FILM-7B significantly improves the performance on real-world
long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while
maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2
accuracy on MMLU). Github Link: https://github.com/microsoft/FILM.
FILM-7B enhances long-context processing by using information-intensive training with synthesized datasets, improving retrieval and performance on real-world tasks without compromising short-context performance.
information-intensive training long-context training synthesized long-context question-answer dataset fine-grained information awareness FILM-7B probing tasks document context code context structured-data context forward retrieval backward retrieval bi-directional retrieval NarrativeQA MMLU