Scalable In-context Ranking with Generative Models
In-context Ranking (ICR) is an emerging paradigm for Information Retrieval
(IR), which leverages contextual understanding of LLMs by directly
incorporating the task description, candidate documents, and the query into the
model's input prompt and tasking the LLM to identify relevant document(s).
While it is effective, efficiency is a significant challenge in this paradigm,
especially as the candidate list grows due to quadratic/super-linear scaling of
attention operation with context length. To this end, this paper first
identifies inherent and exploitable structures in the attention of LLMs
finetuned for ICR: (1) inter-document block sparsity: attention is dense within
each document block but sparse across different documents in the context; and
(2) query-document block relevance: the attention scores from certain query
tokens to a document block in middle layers strongly correlate with that
document's actual relevance. Motivated by these observations, we introduce
BlockRank (Blockwise In-context Ranking), a novel method that adapts the
attention operation in an LLM by (a) architecturally enforcing the observed
inter-document block sparsity, reducing attention complexity from quadratic to
linear without loss in performance, and (b) optimizing query-document block
relevance for true relevant documents during fine-tuning using an auxiliary
contrastive training objective, improving retrieval in attention. Experiments
on BEIR, MSMarco and NQ with Mistral-7B demonstrate that FLARE Mistral matches
or outperforms existing SOTA listwise rankers and controlled fine-tuned
baseline while being significantly more efficient at inference (4.7x for 100
MSMarco documents in context) and scaling gracefully to long-context
shortlists, around 500 documents in-context (approximately 100K context length)
within a second, presenting a scalable and effective solution for ICR.
68e6ac940d37f65d7721a61f
Nilesh Gupta
claimed_verified
2025-10-09T06:51:02.541Z
6527151d5606f146974d60d8
/avatars/00ac8ab005ceadae866dea5471f6aab9.svg
Nilesh Gupta
user
quicktensor
68e6ac940d37f65d7721a620
Chong You
68e6ac940d37f65d7721a621
Srinadh Bhojanapalli
68e6ac940d37f65d7721a622
Sanjiv Kumar
68e6ac940d37f65d7721a623
Inderjit Dhillon
68e6ac940d37f65d7721a624
Felix Yu
Scalable In-context Ranking with Generative Models
In-context Ranking (ICR) is an emerging paradigm for Information Retrieval
(IR), which leverages contextual understanding of LLMs by directly
incorporating the task description, candidate documents, and the query into the
model's input prompt and tasking the LLM to identify relevant document(s).
While it is effective, efficiency is a significant challenge in this paradigm,
especially as the candidate list grows due to quadratic/super-linear scaling of
attention operation with context length. To this end, this paper first
identifies inherent and exploitable structures in the attention of LLMs
finetuned for ICR: (1) inter-document block sparsity: attention is dense within
each document block but sparse across different documents in the context; and
(2) query-document block relevance: the attention scores from certain query
tokens to a document block in middle layers strongly correlate with that
document's actual relevance. Motivated by these observations, we introduce
BlockRank (Blockwise In-context Ranking), a novel method that adapts the
attention operation in an LLM by (a) architecturally enforcing the observed
inter-document block sparsity, reducing attention complexity from quadratic to
linear without loss in performance, and (b) optimizing query-document block
relevance for true relevant documents during fine-tuning using an auxiliary
contrastive training objective, improving retrieval in attention. Experiments
on BEIR, MSMarco and NQ with Mistral-7B demonstrate that FLARE Mistral matches
or outperforms existing SOTA listwise rankers and controlled fine-tuned
baseline while being significantly more efficient at inference (4.7x for 100
MSMarco documents in context) and scaling gracefully to long-context
shortlists, around 500 documents in-context (approximately 100K context length)
within a second, presenting a scalable and effective solution for ICR.
BlockRank optimizes in-context ranking by enforcing inter-document block sparsity and enhancing query-document relevance, improving efficiency and scalability in large-scale information retrieval.
in-context ranking ICR LLMs attention operation inter-document block sparsity query-document block relevance BlockRank BEIR MSMarco NQ Mistral-7B FLARE Mistral listwise rankers contrastive training objective