SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
  Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
https://github.com/Alpha-VLLM/LLaMA2-Accessory
65c59a8fc83b7f1e35ed1a2e
Peng Gao
65c59a8fc83b7f1e35ed1a2f
Renrui Zhang
claimed_verified
2024-03-22T08:39:39.087Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
user
ZrrSkywalker
65c59a8fc83b7f1e35ed1a30
Chris Liu
claimed_verified
2025-04-14T09:48:09.458Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
65c59a8fc83b7f1e35ed1a31
Longtian Qiu
65c59a8fc83b7f1e35ed1a32
Siyuan Huang
claimed_verified
2024-09-17T07:14:08.354Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
65c59a8fc83b7f1e35ed1a33
Weifeng Lin
admin_assigned
2024-02-09T09:38:08.676Z
63483f3450fe0799927b9b0f
/avatars/66c2be2d03dafcb91cec0239ca7b8b44.svg
Weifeng Lin
user
Singaless
65c59a8fc83b7f1e35ed1a34
Shitian Zhao
admin_assigned
2024-02-09T09:38:16.388Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
65c59a8fc83b7f1e35ed1a35
Shijie Geng
claimed_verified
2024-04-03T09:04:27.952Z
62130cc0581b98bb1ad00a9d
https://cdn-avatars.huggingface.co/v1/production/uploads/1672648250385-62130cc0581b98bb1ad00a9d.jpeg
Shijie Geng
user
makitanikaze
65c59a8fc83b7f1e35ed1a36
Ziyi Lin
65c59a8fc83b7f1e35ed1a37
Peng Jin
claimed_verified
2024-10-17T11:31:15.753Z
651585ea18cfed0d30bee586
/avatars/579e468334102472d870875fe40302e6.svg
Peng Jin
user
Chat-UniVi
65c59a8fc83b7f1e35ed1a38
Kaipeng Zhang
admin_assigned
2024-02-09T09:41:43.454Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
65c59a8fc83b7f1e35ed1a39
Wenqi Shao
admin_assigned
2024-02-09T09:41:50.941Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
65c59a8fc83b7f1e35ed1a3a
Chao Xu
admin_assigned
2024-02-09T09:42:06.755Z
6468a48cff18750165a5aad2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
Chao Xu
user
chaoxu
65c59a8fc83b7f1e35ed1a3b
Conghui He
admin_assigned
2024-02-09T09:42:13.507Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
65c59a8fc83b7f1e35ed1a3c
Junjun He
65c59a8fc83b7f1e35ed1a3d
Hao Shao
admin_assigned
2024-02-09T09:42:41.197Z
651bda4337fecec1fe8bb09f
/avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
Hao Shao
user
deepcs233
65c59a8fc83b7f1e35ed1a3e
Pan Lu
admin_assigned
2024-02-09T09:42:59.342Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
65c59a8fc83b7f1e35ed1a3f
Hongsheng Li
admin_assigned
2024-02-09T09:43:07.274Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
65c59a8fc83b7f1e35ed1a40
Yu Qiao
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large
  Language Models
We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM)
series developed upon SPHINX. To improve the architecture and training
efficiency, we modify the SPHINX framework by removing redundant visual
encoders, bypassing fully-padded sub-images with skip tokens, and simplifying
multi-stage training into a one-stage all-in-one paradigm. To fully unleash the
potential of MLLMs, we assemble a comprehensive multi-domain and multimodal
dataset covering publicly available resources in language, vision, and
vision-language tasks. We further enrich this collection with our curated OCR
intensive and Set-of-Mark datasets, extending the diversity and generality. By
training over different base LLMs including TinyLlama1.1B, InternLM2-7B,
LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in
parameter size and multilingual capabilities. Comprehensive benchmarking
reveals a strong correlation between the multi-modal performance with the data
and parameter scales. Code and models are released at
https://github.com/Alpha-VLLM/LLaMA2-Accessory
SPHINX-X, an extensive Multimodality Large Language Model series, enhances training efficiency and performance by modifying the SPHINX framework and using comprehensive, curated datasets covering language, vision, and vision-language tasks.
SPHINX-X Multimodality Large Language Model MLLM visual encoders skip tokens one-stage all-in-one paradigm multi-domain multimodal dataset OCR intensive Set-of-Mark TinyLlama1.1B InternLM2-7B LLaMA2-13B Mixtral8x7B