Context is Gold to find the Gold Passage: Evaluating and Training
  Contextual Document Embeddings
A limitation of modern document retrieval embedding methods is that they
typically encode passages (chunks) from the same documents independently, often
overlooking crucial contextual information from the rest of the document that
could greatly improve individual chunk representations.
  In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a
benchmark designed to evaluate retrieval models on their ability to leverage
document-wide context. Our results show that state-of-the-art embedding models
struggle in retrieval scenarios where context is required. To address this
limitation, we propose InSeNT (In-sequence Negative Training), a novel
contrastive post-training approach which combined with late chunking pooling
enhances contextual representation learning while preserving computational
efficiency. Our method significantly improves retrieval quality on ConTEB
without sacrificing base model performance. We further find chunks embedded
with our method are more robust to suboptimal chunking strategies and larger
retrieval corpus sizes. We open-source all artifacts at
https://github.com/illuin-tech/contextual-embeddings.
683d53ace2a7d8d977a00dab
Max Conti
683d53ace2a7d8d977a00dac
Manuel Faysse
extracted_confirmed
2025-06-02T15:07:26.977Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
683d53ace2a7d8d977a00dad
Gautier Viaud
683d53ace2a7d8d977a00dae
Antoine Bosselut
683d53ace2a7d8d977a00daf
Céline Hudelot
683d53ace2a7d8d977a00db0
Pierre Colombo
Context is Gold to find the Gold Passage: Evaluating and Training
  Contextual Document Embeddings
A limitation of modern document retrieval embedding methods is that they
typically encode passages (chunks) from the same documents independently, often
overlooking crucial contextual information from the rest of the document that
could greatly improve individual chunk representations.
  In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a
benchmark designed to evaluate retrieval models on their ability to leverage
document-wide context. Our results show that state-of-the-art embedding models
struggle in retrieval scenarios where context is required. To address this
limitation, we propose InSeNT (In-sequence Negative Training), a novel
contrastive post-training approach which combined with late chunking pooling
enhances contextual representation learning while preserving computational
efficiency. Our method significantly improves retrieval quality on ConTEB
without sacrificing base model performance. We further find chunks embedded
with our method are more robust to suboptimal chunking strategies and larger
retrieval corpus sizes. We open-source all artifacts at
https://github.com/illuin-tech/contextual-embeddings.
A context-aware benchmark and contrastive training method improve document retrieval quality by leveraging full-document context and maintaining computational efficiency.
ConTEB Context-aware Text Embedding Benchmark InSeNT In-sequence Negative Training contrastive post-training contextual representation learning chunking strategies retrieval corpus size