PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
Long-context reasoning requires accurately identifying relevant information
in extensive, noisy input contexts. Previous research shows that using
test-time learning to encode context directly into model parameters can
effectively enable reasoning over noisy information. However, meta-learning
methods for enabling test-time learning are prohibitively memory-intensive,
preventing their application to long context settings. In this work, we propose
PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for
learning to encode long input contexts using gradient updates to a lightweight
model adapter at test time. Specifically, PERK employs two nested optimization
loops in a meta-training phase. The inner loop rapidly encodes contexts into a
low-rank adapter (LoRA) that serves as a parameter-efficient memory module for
the base model. Concurrently, the outer loop learns to use the updated adapter
to accurately recall and reason over relevant information from the encoded long
context. Our evaluations on several long-context reasoning tasks show that PERK
significantly outperforms the standard prompt-based long-context baseline,
achieving average absolute performance gains of up to 90% for smaller models
(GPT-2) and up to 27% for our largest evaluated model, Qwen-2.5-0.5B. In
general, PERK is more robust to reasoning complexity, length extrapolation, and
the locations of relevant information in contexts. Finally, we show that while
PERK is memory-intensive during training, it scales more efficiently at
inference time than prompt-based long-context inference.
686fe7d21de169176ba58b6d
Zeming Chen
claimed_verified
2025-07-11T08:02:49.434Z
654b92086a49f6f6e0edbcd9
https://cdn-avatars.huggingface.co/v1/production/uploads/654b92086a49f6f6e0edbcd9/1bWswk7EcL04mWwgx1TpA.jpeg
Zeming Chen - Eric
user
zechen-nlp
686fe7d21de169176ba58b6e
Angelika Romanou
686fe7d21de169176ba58b6f
Gail Weiss
686fe7d21de169176ba58b70
Antoine Bosselut
PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
Long-context reasoning requires accurately identifying relevant information
in extensive, noisy input contexts. Previous research shows that using
test-time learning to encode context directly into model parameters can
effectively enable reasoning over noisy information. However, meta-learning
methods for enabling test-time learning are prohibitively memory-intensive,
preventing their application to long context settings. In this work, we propose
PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for
learning to encode long input contexts using gradient updates to a lightweight
model adapter at test time. Specifically, PERK employs two nested optimization
loops in a meta-training phase. The inner loop rapidly encodes contexts into a
low-rank adapter (LoRA) that serves as a parameter-efficient memory module for
the base model. Concurrently, the outer loop learns to use the updated adapter
to accurately recall and reason over relevant information from the encoded long
context. Our evaluations on several long-context reasoning tasks show that PERK
significantly outperforms the standard prompt-based long-context baseline,
achieving average absolute performance gains of up to 90% for smaller models
(GPT-2) and up to 27% for our largest evaluated model, Qwen-2.5-0.5B. In
general, PERK is more robust to reasoning complexity, length extrapolation, and
the locations of relevant information in contexts. Finally, we show that while
PERK is memory-intensive during training, it scales more efficiently at
inference time than prompt-based long-context inference.
PERK, a parameter-efficient method using gradient updates to a lightweight adapter, enhances long-context reasoning by encoding contexts into a low-rank adapter and outperforms prompt-based baselines.
test-time learning meta-learning parameter-efficient PERK nested optimization loops low-rank adapter LoRA long-context reasoning prompt-based GPT-2 Qwen-2.5-0.5B reasoning complexity length extrapolation