PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
  Documents
Recent advancements in Large Multimodal Models (LMMs) have leveraged
extensive multimodal datasets to enhance capabilities in complex
knowledge-driven tasks. However, persistent challenges in perceptual and
reasoning errors limit their efficacy, particularly in interpreting intricate
visual data and deducing multimodal relationships. Addressing these issues, we
introduce a novel dataset format, PIN (Paired and INterleaved multimodal
documents), designed to significantly improve both the depth and breadth of
multimodal training. The PIN format is built on three foundational principles:
knowledge intensity, scalability, and support for diverse training modalities.
This innovative format combines markdown files and comprehensive images to
enrich training data with a dense knowledge structure and versatile training
strategies. We present PIN-14M, an open-source dataset comprising 14 million
samples derived from a diverse range of Chinese and English sources, tailored
to include complex web and scientific content. This dataset is constructed
meticulously to ensure data quality and ethical integrity, aiming to facilitate
advanced training strategies and improve model robustness against common
multimodal training pitfalls. Our initial results, forming the basis of this
technical report, suggest significant potential for the PIN format in refining
LMM performance, with plans for future expansions and detailed evaluations of
its impact on model capabilities.
6674e8e6dbe75ba786077cc6
Junjie Wang
claimed_verified
2024-06-21T07:05:45.156Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
6674e8e6dbe75ba786077cc7
Yin Zhang
admin_assigned
2024-06-21T08:12:03.713Z
62be79222dd4214c062ec09d
/avatars/09f78f8fd533e13ab7e54b5468e22da4.svg
yin zhang
user
miracleyin
6674e8e6dbe75ba786077cc8
Yatai Ji
admin_assigned
2024-06-21T08:12:10.313Z
64c8b0a2f3d2a59a431dbb8e
/avatars/bf130fdca7a2a0fe394558e2bf22c920.svg
yatai ji
user
jiyatai
6674e8e6dbe75ba786077cc9
Yuxiang Zhang
claimed_verified
2024-07-22T07:12:27.425Z
6239386535384c2bcccd2a4f
/avatars/c62cefc6235210ca12b46beb62751d8b.svg
Yuxiang Zhang
user
Joelzhang
6674e8e6dbe75ba786077cca
Chunyang Jiang
6674e8e6dbe75ba786077ccb
Yubo Wang
claimed_verified
2025-01-30T08:41:38.245Z
636a35eff8d9af4aea181608
https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
yubo
user
ubowang
6674e8e6dbe75ba786077ccc
Kang Zhu
claimed_verified
2024-10-18T08:55:17.312Z
6578265ddea7e2122d02f6ba
https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
kang zhu
user
kangz
6674e8e6dbe75ba786077ccd
Zekun Wang
claimed_verified
2024-10-07T07:31:22.430Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
6674e8e6dbe75ba786077cce
Tiezhen Wang
admin_assigned
2024-06-21T08:24:20.397Z
62d22496c58f969c152bcefd
/avatars/76c3b70e312f25e1e610473475553c5c.svg
Tiezhen WANG
user
xianbao
6674e8e6dbe75ba786077ccf
Wenhao Huang
claimed_verified
2024-10-13T20:25:50.587Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
6674e8e6dbe75ba786077cd0
Jie Fu
claimed_verified
2024-06-24T07:33:11.409Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
6674e8e6dbe75ba786077cd1
Bei Chen
6674e8e6dbe75ba786077cd2
Qunshu Lin
6674e8e6dbe75ba786077cd3
Minghao Liu
claimed_verified
2024-06-21T11:47:38.165Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
user
Liam-Liu
6674e8e6dbe75ba786077cd4
Ge Zhang
claimed_verified
2024-06-21T07:05:47.149Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6674e8e6dbe75ba786077cd5
Wenhu Chen
PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
  Documents
Recent advancements in Large Multimodal Models (LMMs) have leveraged
extensive multimodal datasets to enhance capabilities in complex
knowledge-driven tasks. However, persistent challenges in perceptual and
reasoning errors limit their efficacy, particularly in interpreting intricate
visual data and deducing multimodal relationships. Addressing these issues, we
introduce a novel dataset format, PIN (Paired and INterleaved multimodal
documents), designed to significantly improve both the depth and breadth of
multimodal training. The PIN format is built on three foundational principles:
knowledge intensity, scalability, and support for diverse training modalities.
This innovative format combines markdown files and comprehensive images to
enrich training data with a dense knowledge structure and versatile training
strategies. We present PIN-14M, an open-source dataset comprising 14 million
samples derived from a diverse range of Chinese and English sources, tailored
to include complex web and scientific content. This dataset is constructed
meticulously to ensure data quality and ethical integrity, aiming to facilitate
advanced training strategies and improve model robustness against common
multimodal training pitfalls. Our initial results, forming the basis of this
technical report, suggest significant potential for the PIN format in refining
LMM performance, with plans for future expansions and detailed evaluations of
its impact on model capabilities.
A new dataset format called PIN is introduced to enhance LMMs by improving perceptual and reasoning capabilities through diverse, high-quality multimodal training data.
Large Multimodal Models LMMs perceptual errors reasoning errors multimodal datasets multimodal relationships PIN format knowledge intensity scalability training modalities markdown files comprehensive images dense knowledge structure versatile training strategies PIN-14M data quality ethical integrity model robustness common multimodal training pitfalls