BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
  Retrieval
Existing retrieval benchmarks primarily consist of information-seeking
queries (e.g., aggregated questions from search engines) where keyword or
semantic-based retrieval is usually sufficient. However, many complex
real-world queries require in-depth reasoning to identify relevant documents
that go beyond surface form matching. For example, finding documentation for a
coding question requires understanding the logic and syntax of the functions
involved. To better benchmark retrieval on such challenging queries, we
introduce BRIGHT, the first text retrieval benchmark that requires intensive
reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
real-world queries collected from diverse domains (such as economics,
psychology, robotics, software engineering, earth sciences, etc.), sourced from
naturally occurring or carefully curated human data. Extensive evaluation
reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
that augmenting queries with Chain-of-Thought reasoning generated by large
language models (LLMs) improves performance by up to 12.2 points. Moreover,
BRIGHT is robust against data leakage during pretraining of the benchmarked
models as we validate by showing similar performance even when documents from
the benchmark are included in the training data. We believe that BRIGHT paves
the way for future research on retrieval systems in more realistic and
challenging settings. Our code and data are available at
https://brightbenchmark.github.io.
669a08cdf21b09fdce917195
Hongjin Su
claimed_verified
2024-07-19T11:22:55.865Z
632d53951538d4798a73c849
/avatars/b7d0a895e669bcd1303c4716b5401c36.svg
Hongjin SU
user
multi-train
669a08cdf21b09fdce917196
Howard Yen
669a08cdf21b09fdce917197
Mengzhou Xia
669a08cdf21b09fdce917198
Weijia Shi
669a08cdf21b09fdce917199
Niklas Muennighoff
669a08cdf21b09fdce91719a
Han-yu Wang
669a08cdf21b09fdce91719b
Haisu Liu
669a08cdf21b09fdce91719c
Quan Shi
claimed_verified
2025-04-18T09:42:12.653Z
654fe67e45c0dccd570cf4bb
/avatars/5bf84b02b452acf9c13e9254fc820a5b.svg
Ben Shi
user
benshi34
669a08cdf21b09fdce91719d
Zachary S. Siegel
669a08cdf21b09fdce91719e
Michael Tang
669a08cdf21b09fdce91719f
Ruoxi Sun
669a08cdf21b09fdce9171a0
Jinsung Yoon
669a08cdf21b09fdce9171a1
Sercan O. Arik
669a08cdf21b09fdce9171a2
Danqi Chen
669a08cdf21b09fdce9171a3
Tao Yu
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
  Retrieval
Existing retrieval benchmarks primarily consist of information-seeking
queries (e.g., aggregated questions from search engines) where keyword or
semantic-based retrieval is usually sufficient. However, many complex
real-world queries require in-depth reasoning to identify relevant documents
that go beyond surface form matching. For example, finding documentation for a
coding question requires understanding the logic and syntax of the functions
involved. To better benchmark retrieval on such challenging queries, we
introduce BRIGHT, the first text retrieval benchmark that requires intensive
reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
real-world queries collected from diverse domains (such as economics,
psychology, robotics, software engineering, earth sciences, etc.), sourced from
naturally occurring or carefully curated human data. Extensive evaluation
reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
that augmenting queries with Chain-of-Thought reasoning generated by large
language models (LLMs) improves performance by up to 12.2 points. Moreover,
BRIGHT is robust against data leakage during pretraining of the benchmarked
models as we validate by showing similar performance even when documents from
the benchmark are included in the training data. We believe that BRIGHT paves
the way for future research on retrieval systems in more realistic and
challenging settings. Our code and data are available at
https://brightbenchmark.github.io.
BRIGHT is a new text retrieval benchmark requiring deep reasoning, showing that existing models struggle and performs robustly even with document leakage during training.
retrieval benchmark information-seeking queries semantic-based retrieval deep reasoning BRIGHT real-world queries diverse domains nDCG@10 state-of-the-art retrieval models Chain-of-Thought reasoning large language models LLMs data leakage