Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard
  Negatives for Robust Information Retrieval
Training robust retrieval and reranker models typically relies on large-scale
retrieval datasets; for example, the BGE collection contains 1.6 million
query-passage pairs sourced from various data sources. However, we find that
certain datasets can negatively impact model effectiveness -- pruning 8 out of
15 datasets from the BGE collection reduces the training set size by
2.35times and increases nDCG@10 on BEIR by 1.0 point. This motivates a
deeper examination of training data quality, with a particular focus on "false
negatives", where relevant passages are incorrectly labeled as irrelevant. We
propose a simple, cost-effective approach using cascading LLM prompts to
identify and relabel hard negatives. Experimental results show that relabeling
false negatives with true positives improves both E5 (base) and Qwen2.5-7B
retrieval models by 0.7-1.4 nDCG@10 on BEIR and by 1.7-1.8 nDCG@10 on zero-shot
AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on
the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of the
cascading design is further supported by human annotation results, where we
find judgment by GPT-4o shows much higher agreement with humans than
GPT-4o-mini.
683049bd842b92383b946392
Nandan Thakur
claimed_verified
2025-05-26T08:14:21.488Z
60196690dd31fde3c1062960
https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
Nandan Thakur
user
nthakur
683049bd842b92383b946393
Crystina Zhang
claimed_verified
2025-05-26T08:14:18.843Z
611fb1cbfa8355ed0309de81
https://cdn-avatars.huggingface.co/v1/production/uploads/1665676377085-611fb1cbfa8355ed0309de81.jpeg
Xinyu ZHANG
user
crystina-z
683049bd842b92383b946394
Xueguang Ma
683049bd842b92383b946395
Jimmy Lin
Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard
  Negatives for Robust Information Retrieval
Training robust retrieval and reranker models typically relies on large-scale
retrieval datasets; for example, the BGE collection contains 1.6 million
query-passage pairs sourced from various data sources. However, we find that
certain datasets can negatively impact model effectiveness -- pruning 8 out of
15 datasets from the BGE collection reduces the training set size by
2.35times and increases nDCG@10 on BEIR by 1.0 point. This motivates a
deeper examination of training data quality, with a particular focus on "false
negatives", where relevant passages are incorrectly labeled as irrelevant. We
propose a simple, cost-effective approach using cascading LLM prompts to
identify and relabel hard negatives. Experimental results show that relabeling
false negatives with true positives improves both E5 (base) and Qwen2.5-7B
retrieval models by 0.7-1.4 nDCG@10 on BEIR and by 1.7-1.8 nDCG@10 on zero-shot
AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on
the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of the
cascading design is further supported by human annotation results, where we
find judgment by GPT-4o shows much higher agreement with humans than
GPT-4o-mini.
Using cascading LLM prompts to identify and relabel false negatives in datasets improves retrieval and reranking models' performance.
retrieval models reranker models BGE collection nDCG@10 BEIR false negatives E5 (base) Qwen2.5-7B zero-shot AIR-Bench GPT-4o GPT-4o-mini