MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
  Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of
images and text are crucial for training frontier large multimodal models
(LMMs). Despite the rapid progression of open-source LMMs, there remains a
pronounced scarcity of large-scale, diverse open-source multimodal interleaved
datasets. In response, we introduce MINT-1T, the most extensive and diverse
open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
trillion text tokens and three billion images, a 10x scale-up from existing
open-source datasets. Additionally, we include previously untapped sources such
as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
substantial engineering effort, sharing the data curation process and releasing
the dataset greatly benefits the community. Our experiments show that LMMs
trained on MINT-1T rival the performance of models trained on the previous
leading dataset, OBELICS. Our data and code will be released at
https://github.com/mlfoundations/MINT-1T.
667143979ab144eb3aad5beb
Anas Awadalla
claimed_verified
2024-06-18T21:26:54.637Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
667143979ab144eb3aad5bec
Le Xue
667143979ab144eb3aad5bed
Oscar Lo
667143979ab144eb3aad5bee
Manli Shu
claimed_verified
2024-06-19T20:21:11.917Z
63bdcb129a15a3e94198e7d3
https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
Manli Shu
user
Manli
667143979ab144eb3aad5bef
Hannah Lee
667143979ab144eb3aad5bf0
Etash Kumar Guha
667143979ab144eb3aad5bf1
Matt Jordan
667143979ab144eb3aad5bf2
Sheng Shen
667143979ab144eb3aad5bf3
Mohamed Awadalla
667143979ab144eb3aad5bf4
Silvio Savarese
667143979ab144eb3aad5bf5
Caiming Xiong
667143979ab144eb3aad5bf6
Ran Xu
claimed_verified
2024-08-19T22:14:07.356Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
667143979ab144eb3aad5bf7
Yejin Choi
667143979ab144eb3aad5bf8
Ludwig Schmidt
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal
  Dataset with One Trillion Tokens
Multimodal interleaved datasets featuring free-form interleaved sequences of
images and text are crucial for training frontier large multimodal models
(LMMs). Despite the rapid progression of open-source LMMs, there remains a
pronounced scarcity of large-scale, diverse open-source multimodal interleaved
datasets. In response, we introduce MINT-1T, the most extensive and diverse
open-source Multimodal INTerleaved dataset to date. MINT-1T comprises one
trillion text tokens and three billion images, a 10x scale-up from existing
open-source datasets. Additionally, we include previously untapped sources such
as PDFs and ArXiv papers. As scaling multimodal interleaved datasets requires
substantial engineering effort, sharing the data curation process and releasing
the dataset greatly benefits the community. Our experiments show that LMMs
trained on MINT-1T rival the performance of models trained on the previous
leading dataset, OBELICS. Our data and code will be released at
https://github.com/mlfoundations/MINT-1T.
MINT-1T, the largest open-source multimodal interleaved dataset featuring a trillion text tokens and three billion images, enhances the performance of large multimodal models.
multimodal interleaved datasets large multimodal models LMMs MINT-1T PDFs ArXiv papers data curation