OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images
  Interleaved with Text
Image-text interleaved data, consisting of multiple images and texts arranged
in a natural document format, aligns with the presentation paradigm of internet
data and closely resembles human reading habits. Recent studies have shown that
such data aids multimodal in-context learning and maintains the capabilities of
large language models during multimodal fine-tuning. However, the limited scale
and diversity of current image-text interleaved data restrict the development
of multimodal large language models. In this paper, we introduce OmniCorpus, a
10 billion-scale image-text interleaved dataset. Using an efficient data
engine, we filter and extract large-scale high-quality documents, which contain
8.6 billion images and 1,696 billion text tokens. Compared to counterparts
(e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
maintaining good data quality; 2) features more diverse sources, including both
English and non-English websites as well as video-centric websites; 3) is more
flexible, easily degradable from an image-text interleaved format to pure text
corpus and image-text pairs. Through comprehensive analysis and experiments, we
validate the quality, usability, and effectiveness of the proposed dataset. We
hope this could provide a solid data foundation for future multimodal model
research. Code and data are released at
https://github.com/OpenGVLab/OmniCorpus.
666fd32c4aae2409910e249b
Qingyun Li
claimed_verified
2024-10-21T07:54:12.447Z
637f100c1dbae09191088e4c
https://cdn-avatars.huggingface.co/v1/production/uploads/1669271529782-noauth.jpeg
Qingyun Li
user
Qingyun
666fd32c4aae2409910e249c
Zhe Chen
claimed_verified
2024-12-09T14:49:27.780Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
666fd32c4aae2409910e249d
Weiyun Wang
claimed_verified
2024-06-18T21:27:39.234Z
619507e7b74b6c591f794340
https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
Weiyun Wang
user
Weiyun1025
666fd32c4aae2409910e249e
Wenhai Wang
666fd32c4aae2409910e249f
Shenglong Ye
666fd32c4aae2409910e24a0
Zhenjiang Jin
666fd32c4aae2409910e24a1
Guanzhou Chen
claimed_verified
2025-06-09T10:15:04.325Z
6624ba6d79d897d7ddee24b5
https://cdn-avatars.huggingface.co/v1/production/uploads/6624ba6d79d897d7ddee24b5/eHkAquXvHBlCNNgaRDYgG.jpeg
Guanzhou Chen
user
Rayment
666fd32c4aae2409910e24a2
Yinan He
claimed_verified
2025-03-21T11:50:48.391Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
666fd32c4aae2409910e24a3
Zhangwei Gao
666fd32c4aae2409910e24a4
Erfei Cui
666fd32c4aae2409910e24a5
Jiashuo Yu
claimed_verified
2024-07-29T07:06:22.687Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
666fd32c4aae2409910e24a6
Hao Tian
666fd32c4aae2409910e24a7
Jiasheng Zhou
666fd32c4aae2409910e24a8
Chao Xu
666fd32c4aae2409910e24a9
Bin Wang
666fd32c4aae2409910e24aa
Xingjian Wei
666fd32c4aae2409910e24ab
Wei Li
666fd32c4aae2409910e24ac
Wenjian Zhang
666fd32c4aae2409910e24ad
Bo Zhang
claimed_verified
2024-12-18T09:54:20.763Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
666fd32c4aae2409910e24ae
Pinlong Cai
666fd32c4aae2409910e24af
Licheng Wen
666fd32c4aae2409910e24b0
Xiangchao Yan
666fd32c4aae2409910e24b1
Zhenxiang Li
666fd32c4aae2409910e24b2
Pei Chu
666fd32c4aae2409910e24b3
Yi Wang
666fd32c4aae2409910e24b4
Min Dou
666fd32c4aae2409910e24b5
Changyao Tian
666fd32c4aae2409910e24b6
Xizhou Zhu
666fd32c4aae2409910e24b7
Lewei Lu
666fd32c4aae2409910e24b8
Yushi Chen
666fd32c4aae2409910e24b9
Junjun He
666fd32c4aae2409910e24ba
Zhongying Tu
666fd32c4aae2409910e24bb
Tong Lu
666fd32c4aae2409910e24bc
Yali Wang
666fd32c4aae2409910e24bd
Limin Wang
666fd32c4aae2409910e24be
Dahua Lin
666fd32c4aae2409910e24bf
Yu Qiao
666fd32c4aae2409910e24c0
Botian Shi
666fd32c4aae2409910e24c1
Conghui He
666fd32c4aae2409910e24c2
Jifeng Dai
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images
  Interleaved with Text
Image-text interleaved data, consisting of multiple images and texts arranged
in a natural document format, aligns with the presentation paradigm of internet
data and closely resembles human reading habits. Recent studies have shown that
such data aids multimodal in-context learning and maintains the capabilities of
large language models during multimodal fine-tuning. However, the limited scale
and diversity of current image-text interleaved data restrict the development
of multimodal large language models. In this paper, we introduce OmniCorpus, a
10 billion-scale image-text interleaved dataset. Using an efficient data
engine, we filter and extract large-scale high-quality documents, which contain
8.6 billion images and 1,696 billion text tokens. Compared to counterparts
(e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
maintaining good data quality; 2) features more diverse sources, including both
English and non-English websites as well as video-centric websites; 3) is more
flexible, easily degradable from an image-text interleaved format to pure text
corpus and image-text pairs. Through comprehensive analysis and experiments, we
validate the quality, usability, and effectiveness of the proposed dataset. We
hope this could provide a solid data foundation for future multimodal model
research. Code and data are released at
https://github.com/OpenGVLab/OmniCorpus.
OmniCorpus, a large-scale image-text interleaved dataset, enhances multimodal model research by increasing volume, diversity, and flexibility over existing datasets.
multimodal in-context learning multimodal fine-tuning large language models image-text interleaved dataset data engine data quality image-text format image-text pairs