Towards Self-Improving Systematic Cognition for Next-Generation
  Foundation MLLMs
Despite their impressive capabilities, Multimodal Large Language Models
(MLLMs) face challenges with fine-grained perception and complex reasoning.
Prevalent multimodal pre-training approaches focus on enhancing perception by
training on high-quality image captions due to the extremely high cost of
collecting chain-of-thought (CoT) reasoning data for improving reasoning. While
leveraging advanced MLLMs for caption generation enhances scalability, the
outputs often lack comprehensiveness and accuracy. In this paper, we introduce
Self-Improving cognition (SIcog), a self-learning framework designed to
construct next-generation foundation MLLMs by enhancing their systematic
cognitive capabilities through multimodal pre-training with self-generated
data. Specifically, we propose Chain-of-Description, an approach that improves
an MLLM's systematic perception by enabling step-by-step visual understanding,
ensuring greater comprehensiveness and accuracy. Additionally, we adopt a
structured CoT reasoning technique to enable MLLMs to integrate in-depth
multimodal reasoning. To construct a next-generation foundation MLLM with
self-improved cognition, SIcog first equips an MLLM with systematic perception
and reasoning abilities using minimal external annotations. The enhanced models
then generate detailed captions and CoT reasoning data, which are further
curated through self-consistency. This curated data is ultimately used for
multimodal pre-training to develop next-generation foundation models. Extensive
experiments on both low- and high-resolution MLLMs across diverse benchmarks
demonstrate that, with merely 213K self-generated pre-training samples, SIcog
produces next-generation foundation MLLMs with significantly improved
cognition, achieving benchmark-leading performance compared to prevalent
pre-training approaches.
67da52cd11b6db69208527b9
Xiaoying Zhang
67da52cd11b6db69208527ba
Da Peng
claimed_verified
2025-03-19T09:44:20.846Z
665b276ea132595fafc4a431
/avatars/f238582a506bc24a822360b4d4cbd625.svg
PengDa
user
PengDa02
67da52cd11b6db69208527bb
Yipeng Zhang
claimed_verified
2025-03-19T09:44:16.169Z
666e91b1623133f1ce35acc5
/avatars/cc78520e6cfb83817c1d0c1ac867ebdd.svg
YipengZhang
user
YipengZhang
67da52cd11b6db69208527bc
Zonghao Guo
extracted_pending
2025-03-19T05:14:57.514Z
6491af36c1741666238f3bff
/avatars/0ee7d2ec1566e2cc5e8f144140e17f00.svg
Zonghao Guo
user
guozonghao96
67da52cd11b6db69208527bd
Chengyue Wu
67da52cd11b6db69208527be
Chi Chen
67da52cd11b6db69208527bf
Wei Ke
67da52cd11b6db69208527c0
Helen Meng
67da52cd11b6db69208527c1
Maosong Sun
Towards Self-Improving Systematic Cognition for Next-Generation
  Foundation MLLMs
Despite their impressive capabilities, Multimodal Large Language Models
(MLLMs) face challenges with fine-grained perception and complex reasoning.
Prevalent multimodal pre-training approaches focus on enhancing perception by
training on high-quality image captions due to the extremely high cost of
collecting chain-of-thought (CoT) reasoning data for improving reasoning. While
leveraging advanced MLLMs for caption generation enhances scalability, the
outputs often lack comprehensiveness and accuracy. In this paper, we introduce
Self-Improving cognition (SIcog), a self-learning framework designed to
construct next-generation foundation MLLMs by enhancing their systematic
cognitive capabilities through multimodal pre-training with self-generated
data. Specifically, we propose Chain-of-Description, an approach that improves
an MLLM's systematic perception by enabling step-by-step visual understanding,
ensuring greater comprehensiveness and accuracy. Additionally, we adopt a
structured CoT reasoning technique to enable MLLMs to integrate in-depth
multimodal reasoning. To construct a next-generation foundation MLLM with
self-improved cognition, SIcog first equips an MLLM with systematic perception
and reasoning abilities using minimal external annotations. The enhanced models
then generate detailed captions and CoT reasoning data, which are further
curated through self-consistency. This curated data is ultimately used for
multimodal pre-training to develop next-generation foundation models. Extensive
experiments on both low- and high-resolution MLLMs across diverse benchmarks
demonstrate that, with merely 213K self-generated pre-training samples, SIcog
produces next-generation foundation MLLMs with significantly improved
cognition, achieving benchmark-leading performance compared to prevalent
pre-training approaches.
SIcog enhances MLLMs through self-generated data and structured reasoning to improve systematic perception and reasoning, achieving superior performance with limited annotations.
Multimodal Large Language Models (MLLMs) fine-grained perception complex reasoning chain-of-thought (CoT) reasoning self-improving cognition (SIcog) Chain-of-Description structured CoT reasoning multimodal pre-training self-generated data self-consistency