Writing in the Margins: Better Inference Pattern for Long Context
  Retrieval
In this paper, we introduce Writing in the Margins (WiM), a new inference
pattern for Large Language Models designed to optimize the handling of long
input sequences in retrieval-oriented tasks. This approach leverages the
chunked prefill of the key-value cache to perform segment-wise inference, which
enables efficient processing of extensive contexts along with the generation
and classification of intermediate information ("margins") that guide the model
towards specific tasks. This method increases computational overhead marginally
while significantly enhancing the performance of off-the-shelf models without
the need for fine-tuning. Specifically, we observe that WiM provides an average
enhancement of 7.5% in accuracy for reasoning skills (HotpotQA, MultiHop-RAG)
and more than a 30.0% increase in the F1-score for aggregation tasks (CWE).
Additionally, we show how the proposed pattern fits into an interactive
retrieval design that provides end-users with ongoing updates about the
progress of context processing, and pinpoints the integration of relevant
information into the final response. We release our implementation of WiM using
Hugging Face Transformers library at
https://github.com/writer/writing-in-the-margins.
66ced55c5e180b9b9ca195ee
Melisa Russak
claimed_verified
2024-08-28T09:33:44.129Z
60e61b3969bd0df25c9375da
https://cdn-avatars.huggingface.co/v1/production/uploads/1625692968400-noauth.jpeg
Melisa Russak
user
melisa
66ced55c5e180b9b9ca195ef
Umar Jamil
admin_assigned
2024-08-28T09:36:57.524Z
63805c4a9f1f158b014d63d8
/avatars/3221a1ed4d3f809993ba5c79c341f88d.svg
Umar Jamil
user
hkproj
66ced55c5e180b9b9ca195f0
Christopher Bryant
66ced55c5e180b9b9ca195f1
Kiran Kamble
claimed_verified
2024-08-28T08:26:37.049Z
621d6f532165dc431641e438
/avatars/56ccef10a8426d7160ef3586a771bd63.svg
Kiran Kamble
user
kiranr
66ced55c5e180b9b9ca195f2
Axel Magnuson
admin_assigned
2024-08-28T09:37:33.499Z
65f0d67922af97ebe97d9cd8
/avatars/2075546f2dfdd060e07b82c945180e97.svg
Axel Magnuson
user
axel-writer
66ced55c5e180b9b9ca195f3
Mateusz Russak
admin_assigned
2024-08-28T09:42:18.925Z
6357a6f4a1f8ad1c31bdcc81
/avatars/5996bbfdef627bf43f5d74866f2ef3bd.svg
Mateusz Russak
user
mateuszrussak
66ced55c5e180b9b9ca195f4
Waseem AlShikh
claimed_verified
2024-08-28T09:33:42.700Z
60cd486d723acf5eb46fe8d3
https://cdn-avatars.huggingface.co/v1/production/uploads/60cd486d723acf5eb46fe8d3/Z1bD1kjvZ0QAOjZna41Xr.jpeg
Waseem AlShikh
user
wassemgtk
Writing in the Margins: Better Inference Pattern for Long Context
  Retrieval
In this paper, we introduce Writing in the Margins (WiM), a new inference
pattern for Large Language Models designed to optimize the handling of long
input sequences in retrieval-oriented tasks. This approach leverages the
chunked prefill of the key-value cache to perform segment-wise inference, which
enables efficient processing of extensive contexts along with the generation
and classification of intermediate information ("margins") that guide the model
towards specific tasks. This method increases computational overhead marginally
while significantly enhancing the performance of off-the-shelf models without
the need for fine-tuning. Specifically, we observe that WiM provides an average
enhancement of 7.5% in accuracy for reasoning skills (HotpotQA, MultiHop-RAG)
and more than a 30.0% increase in the F1-score for aggregation tasks (CWE).
Additionally, we show how the proposed pattern fits into an interactive
retrieval design that provides end-users with ongoing updates about the
progress of context processing, and pinpoints the integration of relevant
information into the final response. We release our implementation of WiM using
Hugging Face Transformers library at
https://github.com/writer/writing-in-the-margins.
Writing in the Margins (WiM) enhances Large Language Models' performance on long input sequences and retrieval tasks by using chunked prefill and marginal information, improving accuracy and F1-score without fine-tuning.
Large Language Models chunked prefill key-value cache segment-wise inference reasoning skills HotpotQA MultiHop-RAG aggregation tasks CWE interactive retrieval design F1-score