TextSquare: Scaling up Text-Centric Visual Instruction Tuning
Text-centric visual question answering (VQA) has made great strides with the
development of Multimodal Large Language Models (MLLMs), yet open-source models
still fall short of leading models like GPT4V and Gemini, partly due to a lack
of extensive, high-quality instruction tuning data. To this end, we introduce a
new approach for creating a massive, high-quality instruction-tuning dataset,
Square-10M, which is generated using closed-source MLLMs. The data construction
process, termed Square, consists of four steps: Self-Questioning, Answering,
Reasoning, and Evaluation. Our experiments with Square-10M led to three key
findings: 1) Our model, TextSquare, considerably surpasses open-source previous
state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%).
It even outperforms top-tier models like GPT4V and Gemini in 6 of 10
text-centric benchmarks. 2) Additionally, we demonstrate the critical role of
VQA reasoning data in offering comprehensive contextual insights for specific
questions. This not only improves accuracy but also significantly mitigates
hallucinations. Specifically, TextSquare scores an average of 75.1% across four
general VQA and hallucination evaluation datasets, outperforming previous
state-of-the-art models. 3) Notably, the phenomenon observed in scaling
text-centric VQA datasets reveals a vivid pattern: the exponential increase of
instruction tuning data volume is directly proportional to the improvement in
model performance, thereby validating the necessity of the dataset scale and
the high quality of Square-10M.
6625f1688fee5ba324494021
Jingqun Tang
admin_assigned
2024-04-22T08:05:44.330Z
642bcc5ee8dfcc1fe4f4edea
/avatars/0e9142a0d5f6341fb3d4dcd76455eb3f.svg
jingqun tang
user
jingqun
6625f1688fee5ba324494022
Chunhui Lin
claimed_verified
2024-04-22T08:17:45.582Z
6363b6d561b008e03e7bfae4
/avatars/1af24131896a6c2be9e81e2997a8e0f7.svg
Chunhui Lin
user
Chunhuizx
6625f1688fee5ba324494023
Zhen Zhao
claimed_verified
2024-04-22T08:17:47.654Z
64c0b6cea73ac49b2c129d42
/avatars/4b2d993c0f176292ff032eec07889570.svg
zhaozhen
user
zhaozhen026
6625f1688fee5ba324494024
Shu Wei
6625f1688fee5ba324494025
Binghong Wu
6625f1688fee5ba324494026
Qi Liu
6625f1688fee5ba324494027
Hao Feng
6625f1688fee5ba324494028
Yang Li
6625f1688fee5ba324494029
Siqi Wang
6625f1688fee5ba32449402a
Lei Liao
6625f1688fee5ba32449402b
Wei Shi
6625f1688fee5ba32449402c
Yuliang Liu
6625f1688fee5ba32449402d
Hao Liu
6625f1688fee5ba32449402e
Yuan Xie
6625f1688fee5ba32449402f
Xiang Bai
admin_assigned
2024-04-22T11:43:47.494Z
641790e2f1e86908935d82a0
/avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
Xiang Bai
user
baixianger
6625f1688fee5ba324494030
Can Huang
TextSquare: Scaling up Text-Centric Visual Instruction Tuning
Text-centric visual question answering (VQA) has made great strides with the
development of Multimodal Large Language Models (MLLMs), yet open-source models
still fall short of leading models like GPT4V and Gemini, partly due to a lack
of extensive, high-quality instruction tuning data. To this end, we introduce a
new approach for creating a massive, high-quality instruction-tuning dataset,
Square-10M, which is generated using closed-source MLLMs. The data construction
process, termed Square, consists of four steps: Self-Questioning, Answering,
Reasoning, and Evaluation. Our experiments with Square-10M led to three key
findings: 1) Our model, TextSquare, considerably surpasses open-source previous
state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%).
It even outperforms top-tier models like GPT4V and Gemini in 6 of 10
text-centric benchmarks. 2) Additionally, we demonstrate the critical role of
VQA reasoning data in offering comprehensive contextual insights for specific
questions. This not only improves accuracy but also significantly mitigates
hallucinations. Specifically, TextSquare scores an average of 75.1% across four
general VQA and hallucination evaluation datasets, outperforming previous
state-of-the-art models. 3) Notably, the phenomenon observed in scaling
text-centric VQA datasets reveals a vivid pattern: the exponential increase of
instruction tuning data volume is directly proportional to the improvement in
model performance, thereby validating the necessity of the dataset scale and
the high quality of Square-10M.
Text-centric VQA performance is significantly improved by a large, high-quality instruction-tuning dataset, Square-10M, generated through a four-step process, leading to state-of-the-art results on multiple benchmarks.
Multimodal Large Language Models MLLMs VQA reasoning data Text-centric MLLMs OCRBench hallucination evaluation datasets scaling text-centric VQA datasets