IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in
  Expert-Domain Information Retrieval
We introduce IFIR, the first comprehensive benchmark designed to evaluate
instruction-following information retrieval (IR) in expert domains. IFIR
includes 2,426 high-quality examples and covers eight subsets across four
specialized domains: finance, law, healthcare, and science literature. Each
subset addresses one or more domain-specific retrieval tasks, replicating
real-world scenarios where customized instructions are critical. IFIR enables a
detailed analysis of instruction-following retrieval capabilities by
incorporating instructions at different levels of complexity. We also propose a
novel LLM-based evaluation method to provide a more precise and reliable
assessment of model performance in following instructions. Through extensive
experiments on 15 frontier retrieval models, including those based on LLMs, our
results reveal that current models face significant challenges in effectively
following complex, domain-specific instructions. We further provide in-depth
analyses to highlight these limitations, offering valuable insights to guide
future advancements in retriever development.
67ca5d2783ac16a063a56241
Tingyu Song
admin_assigned
2025-03-07T09:53:24.813Z
64dc29d9b5d625e0e9a6ecb9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
Tingyu Song
user
songtingyu
67ca5d2783ac16a063a56242
Guo Gan
claimed_verified
2025-03-07T09:09:52.646Z
65dfeee3d16fb170031df293
https://cdn-avatars.huggingface.co/v1/production/uploads/65dfeee3d16fb170031df293/DZefH96Ol6joeAaGVBe3j.jpeg
gan
user
guo9
67ca5d2783ac16a063a56243
Mingsheng Shang
67ca5d2783ac16a063a56244
Yilun Zhao
admin_assigned
2025-03-07T09:52:59.499Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in
  Expert-Domain Information Retrieval
We introduce IFIR, the first comprehensive benchmark designed to evaluate
instruction-following information retrieval (IR) in expert domains. IFIR
includes 2,426 high-quality examples and covers eight subsets across four
specialized domains: finance, law, healthcare, and science literature. Each
subset addresses one or more domain-specific retrieval tasks, replicating
real-world scenarios where customized instructions are critical. IFIR enables a
detailed analysis of instruction-following retrieval capabilities by
incorporating instructions at different levels of complexity. We also propose a
novel LLM-based evaluation method to provide a more precise and reliable
assessment of model performance in following instructions. Through extensive
experiments on 15 frontier retrieval models, including those based on LLMs, our
results reveal that current models face significant challenges in effectively
following complex, domain-specific instructions. We further provide in-depth
analyses to highlight these limitations, offering valuable insights to guide
future advancements in retriever development.
A benchmark and evaluation method for instruction-following information retrieval in expert domains reveal challenges for current models in handling complex, domain-specific instructions.
instruction-following information retrieval benchmark LLM-based evaluation method retriever development