A Token-level Text Image Foundation Model for Document Understanding
In recent years, general visual foundation models (VFMs) have witnessed
increasing adoption, particularly as image encoders for popular multi-modal
large language models (MLLMs). However, without semantically fine-grained
supervision, these models still encounter fundamental prediction errors in the
context of downstream text-image-related tasks, i.e., perception, understanding
and reasoning with images containing small and dense texts. To bridge this gap,
we develop TokenOCR, the first token-level visual foundation model specifically
tailored for text-image-related tasks, designed to support a variety of
traditional downstream applications. To facilitate the pretraining of TokenOCR,
we also devise a high-quality data production pipeline that constructs the
first token-level image text dataset, TokenIT, comprising 20 million images and
1.8 billion token-mask pairs. Furthermore, leveraging this foundation with
exceptional image-as-text capability, we seamlessly replace previous VFMs with
TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document
understanding tasks. Finally, extensive experiments demonstrate the
effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be
available at https://token-family.github.io/TokenOCR_project.
67c7c17380815cf71fb9139d
Tongkun Guan
claimed_verified
2025-03-05T15:47:05.706Z
650d4a36cbd0c7d550d3b41b
/avatars/0f628ccbbad894098b201a188eb3bd0b.svg
TongkunGuan
user
TongkunGuan
67c7c17380815cf71fb9139e
Zining Wang
67c7c17380815cf71fb9139f
Pei Fu
67c7c17380815cf71fb913a0
Zhengtao Guo
67c7c17380815cf71fb913a1
Wei Shen
67c7c17380815cf71fb913a2
Kai Zhou
67c7c17380815cf71fb913a3
Tiezhu Yue
67c7c17380815cf71fb913a4
Chen Duan
67c7c17380815cf71fb913a5
Hao Sun
67c7c17380815cf71fb913a6
Qianyi Jiang
67c7c17380815cf71fb913a7
Junfeng Luo
67c7c17380815cf71fb913a8
Xiaokang Yang
A Token-level Text Image Foundation Model for Document Understanding
In recent years, general visual foundation models (VFMs) have witnessed
increasing adoption, particularly as image encoders for popular multi-modal
large language models (MLLMs). However, without semantically fine-grained
supervision, these models still encounter fundamental prediction errors in the
context of downstream text-image-related tasks, i.e., perception, understanding
and reasoning with images containing small and dense texts. To bridge this gap,
we develop TokenOCR, the first token-level visual foundation model specifically
tailored for text-image-related tasks, designed to support a variety of
traditional downstream applications. To facilitate the pretraining of TokenOCR,
we also devise a high-quality data production pipeline that constructs the
first token-level image text dataset, TokenIT, comprising 20 million images and
1.8 billion token-mask pairs. Furthermore, leveraging this foundation with
exceptional image-as-text capability, we seamlessly replace previous VFMs with
TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document
understanding tasks. Finally, extensive experiments demonstrate the
effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be
available at https://token-family.github.io/TokenOCR_project.
TokenOCR is a token-level visual foundation model designed to improve text-image understanding, constructed using a novel dataset for pretraining and integrated into a document-level MLLM named TokenVL, demonstrating superior performance in VQA tasks.
visual foundation models multi-modal large language models TokenOCR token-level image text dataset TokenIT image-as-text capability document-level MLLM TokenVL VQA-based document understanding