BigDocs: An Open and Permissively-Licensed Dataset for Training
  Multimodal Models on Document and Code Tasks
Multimodal AI has the potential to significantly enhance
document-understanding tasks, such as processing receipts, understanding
workflows, extracting data from documents, and summarizing reports. Code
generation tasks that require long-structured outputs can also be enhanced by
multimodality. Despite this, their use in commercial applications is often
limited due to limited access to training data and restrictive licensing, which
hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
high-quality, open-access dataset comprising 7.5 million multimodal documents
across 30 tasks. We use an efficient data curation process to ensure our data
is high-quality and license-permissive. Our process emphasizes accountability,
responsibility, and transparency through filtering rules, traceable metadata,
and careful content analysis. Additionally, we introduce BigDocs-Bench, a
benchmark suite with 10 novel tasks where we create datasets that reflect
real-world use cases involving reasoning over Graphical User Interfaces (GUI)
and code generation from images. Our experiments show that training with
BigDocs-Bench improves average performance up to 25.8% over closed-source
GPT-4o in document reasoning and structured output tasks such as
Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
preference for outputs from models trained on BigDocs over GPT-4o. This
suggests that BigDocs can help both academics and the open-source community
utilize and improve AI tools to enhance multimodal capabilities and document
reasoning. The project is hosted at https://bigdocs.github.io .
675714ab0ab5c19744cedc4c
Juan Rodriguez
claimed_verified
2024-12-19T18:05:01.009Z
63a614d264f470027818b066
https://cdn-avatars.huggingface.co/v1/production/uploads/63a614d264f470027818b066/Q5Eih2VqD4NaHB_MkNp32.jpeg
Juan A. Rodriguez
user
joanrodai
675714ab0ab5c19744cedc4d
Xiangru Jian
admin_assigned
2024-12-18T13:49:25.747Z
66155491f1214e73d69074b5
/avatars/00572ab695a4188422e8ee38fc87680b.svg
Xiangru Jian
user
EdwardXJ
675714ab0ab5c19744cedc4e
Siba Smarak Panigrahi
claimed_verified
2025-03-16T21:22:47.380Z
67574a29dd1f47442293becd
/avatars/5e6a28899e5ea631d8f0677a9dd1a51a.svg
Siba Smarak Panigrahi
user
sibasmarakp
675714ab0ab5c19744cedc4f
Tianyu Zhang
675714ab0ab5c19744cedc50
Aarash Feizi
admin_assigned
2024-12-18T13:51:34.277Z
62e98e784fa4bc6de6c0f65b
/avatars/4354e022061d78ec26b14c0b0cd4e433.svg
Aarash Feizi
user
aarashfeizi
675714ab0ab5c19744cedc51
Abhay Puri
admin_assigned
2024-12-18T13:51:41.042Z
65830af2a1707aa10effcc32
/avatars/0626454399b711fca7fb2b66fcecaca8.svg
Abhay Puri
user
abhaypuri
675714ab0ab5c19744cedc52
Akshay Kalkunte
admin_assigned
2024-12-18T13:51:47.592Z
647792b1e5ddd7d2a1714550
/avatars/067cad5fef442146217fd11316985486.svg
Akshay Kalkunte
user
akshaykalkunte
675714ab0ab5c19744cedc53
François Savard
admin_assigned
2024-12-18T13:51:53.902Z
66c600c28991d6123dd89abd
/avatars/4236d4637a0f23e90aa5c79fa8ced466.svg
François Savard
user
fsavard-eai
675714ab0ab5c19744cedc54
Ahmed Masry
claimed_verified
2024-12-10T10:25:39.506Z
63efd75a5c2ceb16fc6e98fc
https://cdn-avatars.huggingface.co/v1/production/uploads/63efd75a5c2ceb16fc6e98fc/qoA4LKuLTEr7hx90i90UK.jpeg
Ahmed Masry
user
ahmed-masry
675714ab0ab5c19744cedc55
Shravan Nayak
admin_assigned
2024-12-18T13:51:59.460Z
643ee7606d5be535d28034f6
https://cdn-avatars.huggingface.co/v1/production/uploads/643ee7606d5be535d28034f6/1F241mP_buW74byC2GAWH.jpeg
Shravan Nayak
user
BAJUKA
675714ab0ab5c19744cedc56
Rabiul Awal
admin_assigned
2024-12-18T13:52:13.031Z
60edf5e03203a5daf7d3912e
https://cdn-avatars.huggingface.co/v1/production/uploads/60edf5e03203a5daf7d3912e/fq1AzGutKI_qojhevc03P.jpeg
Rabiul Awal
user
rabiulawal
675714ab0ab5c19744cedc57
Mahsa Massoud
admin_assigned
2024-12-18T13:52:43.526Z
655502305f7b57f585042408
/avatars/8ce57b8193ee2e6fae42d66aeaafef12.svg
Mahsa
user
mahsamassoud
675714ab0ab5c19744cedc58
Amirhossein Abaskohi
admin_assigned
2024-12-18T13:52:49.625Z
60e32baedc56466240084155
https://cdn-avatars.huggingface.co/v1/production/uploads/60e32baedc56466240084155/Q-cJh3Q3-vvMbe749Gt5B.jpeg
Amirhossein Abaskohi
user
AmirhosseinAbaskohi
675714ab0ab5c19744cedc59
Zichao Li
admin_assigned
2024-12-18T13:53:02.483Z
6419632bb4adb0e101aefd21
/avatars/82ea8d77fd1d41c8a6d3a14547d076f3.svg
lizichao
user
lazyc
675714ab0ab5c19744cedc5a
Suyuchen Wang
admin_assigned
2024-12-18T13:53:08.933Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/MWNanci3x5g780xh-704U.png
Suyuchen Wang
user
sheryc
675714ab0ab5c19744cedc5b
Pierre-André Noël
admin_assigned
2024-12-18T13:53:15.480Z
646cc16b94eb019a96e1fb2e
/avatars/31f168a6c1ec45eb0c784d9119c1b9bf.svg
Pierre-Andre Noel
user
PierreAndreNoel
675714ab0ab5c19744cedc5c
Mats Leon Richter
675714ab0ab5c19744cedc5d
Saverio Vadacchino
675714ab0ab5c19744cedc5e
Shubbam Agarwal
675714ab0ab5c19744cedc5f
Sanket Biswas
675714ab0ab5c19744cedc60
Sara Shanian
675714ab0ab5c19744cedc61
Ying Zhang
675714ab0ab5c19744cedc62
Noah Bolger
675714ab0ab5c19744cedc63
Kurt MacDonald
admin_assigned
2024-12-18T13:53:54.337Z
64dbefc6176ca2c9a2d799ac
/avatars/d8f6d730091dfa0857275e48970770c6.svg
Kurt MacDonald
user
K-NOW
675714ab0ab5c19744cedc64
Simon Fauvel
675714ab0ab5c19744cedc65
Sathwik Tejaswi
claimed_verified
2025-02-07T12:56:37.773Z
63d3095c2727d7888cbb54e2
/avatars/51fd37f4216eec309cd439e56626d6ad.svg
Sathwik Tejaswi Madhusudhan
user
stm4
675714ab0ab5c19744cedc66
Srinivas Sunkara
675714ab0ab5c19744cedc67
Joao Monteiro
675714ab0ab5c19744cedc68
Krishnamurthy DJ Dvijotham
675714ab0ab5c19744cedc69
Torsten Scholak
admin_assigned
2024-12-18T13:54:13.721Z
60ecaa5efee13fee7ada7af4
https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
Torsten Scholak
user
tscholak
675714ab0ab5c19744cedc6a
Nicolas Chapados
admin_assigned
2024-12-18T13:55:42.051Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
675714ab0ab5c19744cedc6b
Sepideh Kharagani
675714ab0ab5c19744cedc6c
Sean Hughes
675714ab0ab5c19744cedc6d
M. Özsu
675714ab0ab5c19744cedc6e
Siva Reddy
675714ab0ab5c19744cedc6f
Marco Pedersoli
admin_assigned
2024-12-18T13:54:40.883Z
64b829042fccad9f5ff20cc7
/avatars/92cca111ddd0db5d998615c2257a0894.svg
Marco Pedersoli
user
Marcopede
675714ab0ab5c19744cedc70
Yoshua Bengio
675714ab0ab5c19744cedc71
Christopher Pal
675714ab0ab5c19744cedc72
Issam Laradji
admin_assigned
2024-12-18T13:55:29.825Z
64062855692855e65ae31688
/avatars/e35d22a037b8b35422d3ee982f133076.svg
Issam Laradji
user
issamlaradji
675714ab0ab5c19744cedc73
Spandanna Gella
claimed_verified
2025-06-11T16:03:08.489Z
65ca8745d64d82c92fa7c71f
https://cdn-avatars.huggingface.co/v1/production/uploads/65ca8745d64d82c92fa7c71f/xayDYjs_LoAsa8BEgDlhb.jpeg
G
user
spandanagella
675714ab0ab5c19744cedc74
Perouz Taslakian
675714ab0ab5c19744cedc75
David Vazquez
claimed_verified
2025-05-08T09:00:20.819Z
62f6691f329d4d014d1b4087
https://cdn-avatars.huggingface.co/v1/production/uploads/1660315866736-noauth.jpeg
David Vazquez
user
dvazquez
675714ab0ab5c19744cedc76
Sai Rajeswar
BigDocs: An Open and Permissively-Licensed Dataset for Training
  Multimodal Models on Document and Code Tasks
Multimodal AI has the potential to significantly enhance
document-understanding tasks, such as processing receipts, understanding
workflows, extracting data from documents, and summarizing reports. Code
generation tasks that require long-structured outputs can also be enhanced by
multimodality. Despite this, their use in commercial applications is often
limited due to limited access to training data and restrictive licensing, which
hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
high-quality, open-access dataset comprising 7.5 million multimodal documents
across 30 tasks. We use an efficient data curation process to ensure our data
is high-quality and license-permissive. Our process emphasizes accountability,
responsibility, and transparency through filtering rules, traceable metadata,
and careful content analysis. Additionally, we introduce BigDocs-Bench, a
benchmark suite with 10 novel tasks where we create datasets that reflect
real-world use cases involving reasoning over Graphical User Interfaces (GUI)
and code generation from images. Our experiments show that training with
BigDocs-Bench improves average performance up to 25.8% over closed-source
GPT-4o in document reasoning and structured output tasks such as
Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
preference for outputs from models trained on BigDocs over GPT-4o. This
suggests that BigDocs can help both academics and the open-source community
utilize and improve AI tools to enhance multimodal capabilities and document
reasoning. The project is hosted at https://bigdocs.github.io .
BigDocs-7.5M and BigDocs-Bench enhance multimodal AI tasks with high-quality, open-access data and benchmarking, improving performance and transparency compared to closed-source models.
multimodal AI document-understanding code generation BigDocs-7.5M BigDocs-Bench graphical user interfaces (GUI) screenshot2html image2latex human evaluations