ColPali: Efficient Document Retrieval with Vision Language Models
Documents are visually rich structures that convey information through text,
as well as tables, figures, page layouts, or fonts. While modern document
retrieval systems exhibit strong performance on query-to-text matching, they
struggle to exploit visual cues efficiently, hindering their performance on
practical document retrieval applications such as Retrieval Augmented
Generation. To benchmark current systems on visually rich document retrieval,
we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of
various page-level retrieving tasks spanning multiple domains, languages, and
settings. The inherent shortcomings of modern systems motivate the introduction
of a new retrieval model architecture, ColPali, which leverages the document
understanding capabilities of recent Vision Language Models to produce
high-quality contextualized embeddings solely from images of document pages.
Combined with a late interaction matching mechanism, ColPali largely
outperforms modern document retrieval pipelines while being drastically faster
and end-to-end trainable.
668397082235ae231ac5eeba
Manuel Faysse
extracted_confirmed
2024-07-02T06:09:46.709Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
668397082235ae231ac5eebb
Hugues Sibille
claimed_verified
2024-07-02T11:59:52.839Z
66211794ae2f58da4f00d317
https://cdn-avatars.huggingface.co/v1/production/uploads/66211794ae2f58da4f00d317/kL3nsgW9ri56J0PaaSMkD.jpeg
Hugues Sibille
user
HugSib
668397082235ae231ac5eebc
Tony Wu
claimed_verified
2024-07-02T08:01:10.759Z
6264f9655f6f2e14d6ac981c
https://cdn-avatars.huggingface.co/v1/production/uploads/1650784534234-noauth.png
Tony Wu
user
tonywu71
668397082235ae231ac5eebd
Gautier Viaud
668397082235ae231ac5eebe
Céline Hudelot
668397082235ae231ac5eebf
Pierre Colombo
ColPali: Efficient Document Retrieval with Vision Language Models
Documents are visually rich structures that convey information through text,
as well as tables, figures, page layouts, or fonts. While modern document
retrieval systems exhibit strong performance on query-to-text matching, they
struggle to exploit visual cues efficiently, hindering their performance on
practical document retrieval applications such as Retrieval Augmented
Generation. To benchmark current systems on visually rich document retrieval,
we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of
various page-level retrieving tasks spanning multiple domains, languages, and
settings. The inherent shortcomings of modern systems motivate the introduction
of a new retrieval model architecture, ColPali, which leverages the document
understanding capabilities of recent Vision Language Models to produce
high-quality contextualized embeddings solely from images of document pages.
Combined with a late interaction matching mechanism, ColPali largely
outperforms modern document retrieval pipelines while being drastically faster
and end-to-end trainable.
The ColPali retrieval model uses Vision Language Models to create embeddings from images of document pages, improving performance and speed in visually rich document retrieval tasks.
Visual Document Retrieval Benchmark ViDoRe ColPali Vision Language Models contextualized embeddings late interaction matching mechanism