InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced
  Mathematical Reasoning
Pre-training on large-scale, high-quality datasets is crucial for enhancing
the reasoning capabilities of Large Language Models (LLMs), especially in
specialized domains such as mathematics. Despite the recognized importance, the
Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source
pre-training dataset specifically designed for mathematical reasoning. To
address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of
interleaved image-text documents. It comprises 24 million web pages, 85 million
associated image URLs, and 40 billion text tokens, all meticulously extracted
and filtered from CommonCrawl. We provide a detailed overview of our data
collection and processing pipeline. To demonstrate the robustness of
InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal
settings. Our evaluations on text-only benchmarks show that, despite utilizing
only 40 billion tokens, our dataset significantly enhances the performance of
our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses
120 billion tokens for the same model size. Nevertheless, with the introduction
of our multi-modal math pre-training dataset, our models set a new
state-of-the-art among open-source models on multi-modal math benchmarks such
as MathVerse and We-Math. We release our data at
https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
66ecd8f89cbe3098580d1a33
Xiaotian Han
claimed_verified
2024-09-20T07:46:01.294Z
650dde4ce14eeb01d42b37a1
https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
Xiaotian Han
user
xiaotianhan
66ecd8f89cbe3098580d1a34
Yiren Jian
admin_assigned
2024-09-20T07:51:24.540Z
64d2b7a76975d68147652a88
/avatars/13be8f052c3f8067be70c1b64b0774f8.svg
Yiren Jian
user
yjian-bytedance
66ecd8f89cbe3098580d1a35
Xuefeng Hu
admin_assigned
2024-09-20T07:52:20.890Z
65cb212725a3b8e44dc0a09a
/avatars/66c8457a8a8a85e676654479f6860a4b.svg
Xuefeng Hu
user
bytehxf
66ecd8f89cbe3098580d1a36
Haogeng Liu
claimed_verified
2024-09-20T07:45:59.371Z
64f58c8bab97f9a83eca4963
/avatars/b525fa19dcb6c051fe3ae260d85926e5.svg
haogengliu
user
lllliuhhhhggg
66ecd8f89cbe3098580d1a37
Yiqi Wang
claimed_verified
2024-09-20T07:45:57.179Z
6566d9088556065b5268c666
/avatars/85e275c3a1e70c21045f13b0cefc93fb.svg
Yiqi Wang
user
Yi-Qi638
66ecd8f89cbe3098580d1a38
Qihang Fan
66ecd8f89cbe3098580d1a39
Yuang Ai
claimed_verified
2024-10-30T09:55:54.668Z
65709464b3501cbcb8f4007e
/avatars/4fb7524cb21e38138b4c67ff5501b403.svg
Yuang Ai
user
shallowdream204
66ecd8f89cbe3098580d1a3a
Huaibo Huang
admin_assigned
2024-09-20T07:52:50.524Z
642259feeaad1bcb28b44f55
/avatars/685791fc7e66d685e9446228c9e278b1.svg
Huaibo Huang
user
Bob072
66ecd8f89cbe3098580d1a3b
Ran He
66ecd8f89cbe3098580d1a3c
Zhenheng Yang
admin_assigned
2024-09-20T07:53:14.014Z
6421183b69a2c2933882d652
/avatars/66813a8fa22915087cccd4dbfb945ca7.svg
Zhenheng Yang
user
zhenheny
66ecd8f89cbe3098580d1a3d
Quanzeng You
admin_assigned
2024-09-20T07:53:21.169Z
642bb8c87f152f6e72b8d3cd
https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
Quanzeng You
user
Ye27
InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced
  Mathematical Reasoning
Pre-training on large-scale, high-quality datasets is crucial for enhancing
the reasoning capabilities of Large Language Models (LLMs), especially in
specialized domains such as mathematics. Despite the recognized importance, the
Multimodal LLMs (MLLMs) field currently lacks a comprehensive open-source
pre-training dataset specifically designed for mathematical reasoning. To
address this gap, we introduce InfiMM-WebMath-40B, a high-quality dataset of
interleaved image-text documents. It comprises 24 million web pages, 85 million
associated image URLs, and 40 billion text tokens, all meticulously extracted
and filtered from CommonCrawl. We provide a detailed overview of our data
collection and processing pipeline. To demonstrate the robustness of
InfiMM-WebMath-40B, we conducted evaluations in both text-only and multimodal
settings. Our evaluations on text-only benchmarks show that, despite utilizing
only 40 billion tokens, our dataset significantly enhances the performance of
our 1.3B model, delivering results comparable to DeepSeekMath-1.3B, which uses
120 billion tokens for the same model size. Nevertheless, with the introduction
of our multi-modal math pre-training dataset, our models set a new
state-of-the-art among open-source models on multi-modal math benchmarks such
as MathVerse and We-Math. We release our data at
https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.
InfiMM-WebMath-40B, a large multimodal dataset, significantly improves mathematical reasoning capabilities in LLMs, achieving state-of-the-art performance on multimodal benchmarks.
Large Language Models Multimodal LLMs InfiMM-WebMath-40B image-text documents CommonCrawl text-only benchmarks DeepSeekMath-1.3B multi-modal math pre-training MathVerse We-Math