OmniDocBench: Benchmarking Diverse PDF Document Parsing with
  Comprehensive Annotations
Document content extraction is crucial in computer vision, especially for
meeting the high-quality data needs of large language models (LLMs) and
retrieval-augmented generation (RAG) technologies. However, current document
parsing methods suffer from significant limitations in terms of diversity and
comprehensive evaluation. To address these challenges, we introduce
OmniDocBench, a novel multi-source benchmark designed to advance automated
document content extraction. OmniDocBench includes a meticulously curated and
annotated high-quality evaluation dataset comprising nine diverse document
types, such as academic papers, textbooks, slides, among others. Our benchmark
provides a flexible and comprehensive evaluation framework with 19 layout
category labels and 14 attribute labels, enabling multi-level assessments
across entire datasets, individual modules, or specific data types. Using
OmniDocBench, we perform an exhaustive comparative analysis of existing modular
pipelines and multimodal end-to-end methods, highlighting their limitations in
handling document diversity and ensuring fair evaluation. OmniDocBench
establishes a robust, diverse, and fair evaluation standard for the document
content extraction field, offering crucial insights for future advancements and
fostering the development of document parsing technologies. The codes and
dataset is available in https://github.com/opendatalab/OmniDocBench.
6758f2c55cd36ca4c1a4539b
Linke Ouyang
claimed_verified
2024-12-11T09:20:06.000Z
66753c556f2ac48ee625d7d1
/avatars/8f7c252675fd8a096794d12971903722.svg
Linke Ouyang
user
ouyanglinke
6758f2c55cd36ca4c1a4539c
Yuan Qu
6758f2c55cd36ca4c1a4539d
Hongbin Zhou
6758f2c55cd36ca4c1a4539e
Jiawei Zhu
6758f2c55cd36ca4c1a4539f
Rui Zhang
6758f2c55cd36ca4c1a453a0
Qunshu Lin
6758f2c55cd36ca4c1a453a1
Bin Wang
claimed_verified
2024-12-11T09:20:11.073Z
63ae9ff5557befe297a76f90
https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
Bin Wang
user
wanderkid
6758f2c55cd36ca4c1a453a2
Zhiyuan Zhao
6758f2c55cd36ca4c1a453a3
Man Jiang
6758f2c55cd36ca4c1a453a4
Xiaomeng Zhao
claimed_verified
2024-12-12T13:41:08.395Z
668f77a3b3991ac0c308a441
https://cdn-avatars.huggingface.co/v1/production/uploads/668f77a3b3991ac0c308a441/w5-9UAxi32HnrTjW9_fGC.jpeg
xiaomeng zhao
user
myhloli
6758f2c55cd36ca4c1a453a5
Jin Shi
6758f2c55cd36ca4c1a453a6
Fan Wu
6758f2c55cd36ca4c1a453a7
Pei Chu
claimed_verified
2024-12-17T08:04:27.537Z
64c9beb2904317f42de06dd8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c9beb2904317f42de06dd8/he3rxfyzfwEd1vLuK6_o2.jpeg
Pei Chu
user
chupei
6758f2c55cd36ca4c1a453a8
Minghao Liu
claimed_verified
2024-12-11T09:20:08.296Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
user
Liam-Liu
6758f2c55cd36ca4c1a453a9
Zhenxiang Li
6758f2c55cd36ca4c1a453aa
Chao Xu
6758f2c55cd36ca4c1a453ab
Bo Zhang
claimed_verified
2024-12-18T09:53:47.544Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
6758f2c55cd36ca4c1a453ac
Botian Shi
admin_assigned
2024-12-11T10:17:19.807Z
643df87f7cd64d872cb9fabd
/avatars/c53bfabcee08de448dde973915e8b31d.svg
Botian Shi
user
friskit
6758f2c55cd36ca4c1a453ad
Zhongying Tu
6758f2c55cd36ca4c1a453ae
Conghui He
admin_assigned
2024-12-11T10:17:05.536Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
OmniDocBench: Benchmarking Diverse PDF Document Parsing with
  Comprehensive Annotations
Document content extraction is crucial in computer vision, especially for
meeting the high-quality data needs of large language models (LLMs) and
retrieval-augmented generation (RAG) technologies. However, current document
parsing methods suffer from significant limitations in terms of diversity and
comprehensive evaluation. To address these challenges, we introduce
OmniDocBench, a novel multi-source benchmark designed to advance automated
document content extraction. OmniDocBench includes a meticulously curated and
annotated high-quality evaluation dataset comprising nine diverse document
types, such as academic papers, textbooks, slides, among others. Our benchmark
provides a flexible and comprehensive evaluation framework with 19 layout
category labels and 14 attribute labels, enabling multi-level assessments
across entire datasets, individual modules, or specific data types. Using
OmniDocBench, we perform an exhaustive comparative analysis of existing modular
pipelines and multimodal end-to-end methods, highlighting their limitations in
handling document diversity and ensuring fair evaluation. OmniDocBench
establishes a robust, diverse, and fair evaluation standard for the document
content extraction field, offering crucial insights for future advancements and
fostering the development of document parsing technologies. The codes and
dataset is available in https://github.com/opendatalab/OmniDocBench.