Haystack Engineering: Context Engineering for Heterogeneous and Agentic
  Long-Context Evaluation
Modern long-context large language models (LLMs) perform well on synthetic
"needle-in-a-haystack" (NIAH) benchmarks, but such tests overlook how noisy
contexts arise from biased retrieval and agentic workflows. We argue that
haystack engineering is necessary to construct noisy long contexts that
faithfully capture key real-world factors -- distraction from heterogeneous
biased retrievers and cascading errors in agentic workflows -- to test models'
long-context robustness. We instantiate it through HaystackCraft, a new NIAH
benchmark built on the full English Wikipedia hyperlink network with multi-hop
questions. HaystackCraft evaluates how heterogeneous retrieval strategies
(e.g., sparse, dense, hybrid, and graph-based) affect distractor composition,
haystack ordering, and downstream LLM performance. HaystackCraft further
extends NIAH to dynamic, LLM-dependent settings that simulate agentic
operations, where models refine queries, reflect on their past reasonings, and
decide when to stop. Experiments with 15 long-context models show that (1)
while stronger dense retrievers can introduce more challenging distractors,
graph-based reranking simultaneously improves retrieval effectiveness and
mitigates more harmful distractors; (2) in agentic tests, even advanced models
like Gemini 2.5 Pro and GPT-5 suffer cascading failures from self-generated
distractors or struggle to perform early stops. These results highlight
persistent challenges in agentic long-context reasoning and establish
HaystackCraft as a valuable testbed for future progress.
68f1154bdf97aa3a74268605
Mufei Li
claimed_verified
2025-10-17T04:12:27.445Z
667113dbef4c0c97967ceff3
/avatars/366c8ef61e4cecbbd5c5e9fae01f1855.svg
Mufei Li
user
ml1996
68f1154bdf97aa3a74268606
Dongqi Fu
68f1154bdf97aa3a74268607
Limei Wang
68f1154bdf97aa3a74268608
Si Zhang
68f1154bdf97aa3a74268609
Hanqing Zeng
68f1154bdf97aa3a7426860a
Kaan Sancak
68f1154bdf97aa3a7426860b
Ruizhong Qiu
68f1154bdf97aa3a7426860c
Haoyu Wang
68f1154bdf97aa3a7426860d
Xiaoxin He
68f1154bdf97aa3a7426860e
Xavier Bresson
68f1154bdf97aa3a7426860f
Yinglong Xia
68f1154bdf97aa3a74268610
Chonglin Sun
68f1154bdf97aa3a74268611
Pan Li
Haystack Engineering: Context Engineering for Heterogeneous and Agentic
  Long-Context Evaluation
Modern long-context large language models (LLMs) perform well on synthetic
"needle-in-a-haystack" (NIAH) benchmarks, but such tests overlook how noisy
contexts arise from biased retrieval and agentic workflows. We argue that
haystack engineering is necessary to construct noisy long contexts that
faithfully capture key real-world factors -- distraction from heterogeneous
biased retrievers and cascading errors in agentic workflows -- to test models'
long-context robustness. We instantiate it through HaystackCraft, a new NIAH
benchmark built on the full English Wikipedia hyperlink network with multi-hop
questions. HaystackCraft evaluates how heterogeneous retrieval strategies
(e.g., sparse, dense, hybrid, and graph-based) affect distractor composition,
haystack ordering, and downstream LLM performance. HaystackCraft further
extends NIAH to dynamic, LLM-dependent settings that simulate agentic
operations, where models refine queries, reflect on their past reasonings, and
decide when to stop. Experiments with 15 long-context models show that (1)
while stronger dense retrievers can introduce more challenging distractors,
graph-based reranking simultaneously improves retrieval effectiveness and
mitigates more harmful distractors; (2) in agentic tests, even advanced models
like Gemini 2.5 Pro and GPT-5 suffer cascading failures from self-generated
distractors or struggle to perform early stops. These results highlight
persistent challenges in agentic long-context reasoning and establish
HaystackCraft as a valuable testbed for future progress.
HaystackCraft, a new benchmark using Wikipedia, evaluates long-context LLM robustness by simulating noisy retrieval and agentic workflows, revealing challenges in handling distractors and cascading errors.
long-context large language models needle-in-a-haystack benchmarks haystack engineering heterogeneous biased retrievers cascading errors agentic workflows HaystackCraft Wikipedia hyperlink network multi-hop questions sparse retrievers dense retrievers hybrid retrievers graph-based retrievers reranking retrieval effectiveness agentic operations self-generated distractors early stops Gemini 2.5 Pro GPT-5