TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal
  Large Language Models
Multimodal Large Language Models (MLLMs) have shown impressive results on
various multimodal tasks. However, most existing MLLMs are not well suited for
document-oriented tasks, which require fine-grained image perception and
information compression. In this paper, we present TextHawk, a MLLM that is
specifically designed for document-oriented tasks, while preserving the general
capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
perception by designing four dedicated components. Firstly, a ReSampling and
ReArrangement (ReSA) module is proposed to reduce the redundancy in the
document texts and lower the computational cost of the MLLM. We explore
encoding the positions of each local feature by presenting Scalable Positional
Embeddings (SPEs), which can preserve the scalability of various image sizes. A
Query Proposal Network (QPN) is then adopted to initialize the queries
dynamically among different sub-images. To further enhance the fine-grained
visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
(MLCA) mechanism that captures the hierarchical structure and semantic
relations of document images. Furthermore, we create a new instruction-tuning
dataset for document-oriented tasks by enriching the multimodal document data
with Gemini Pro. We conduct extensive experiments on both general and
document-oriented MLLM benchmarks, and show that TextHawk outperforms the
state-of-the-art methods, demonstrating its effectiveness and superiority in
fine-grained document perception and general abilities.
661e1387f40801e72305d509
Ya-Qi Yu
661e1387f40801e72305d50a
Minghui Liao
661e1387f40801e72305d50b
Jihao Wu
661e1387f40801e72305d50c
Yongxin Liao
661e1387f40801e72305d50d
Xiaoyu Zheng
661e1387f40801e72305d50e
Wei Zeng
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal
  Large Language Models
Multimodal Large Language Models (MLLMs) have shown impressive results on
various multimodal tasks. However, most existing MLLMs are not well suited for
document-oriented tasks, which require fine-grained image perception and
information compression. In this paper, we present TextHawk, a MLLM that is
specifically designed for document-oriented tasks, while preserving the general
capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
perception by designing four dedicated components. Firstly, a ReSampling and
ReArrangement (ReSA) module is proposed to reduce the redundancy in the
document texts and lower the computational cost of the MLLM. We explore
encoding the positions of each local feature by presenting Scalable Positional
Embeddings (SPEs), which can preserve the scalability of various image sizes. A
Query Proposal Network (QPN) is then adopted to initialize the queries
dynamically among different sub-images. To further enhance the fine-grained
visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
(MLCA) mechanism that captures the hierarchical structure and semantic
relations of document images. Furthermore, we create a new instruction-tuning
dataset for document-oriented tasks by enriching the multimodal document data
with Gemini Pro. We conduct extensive experiments on both general and
document-oriented MLLM benchmarks, and show that TextHawk outperforms the
state-of-the-art methods, demonstrating its effectiveness and superiority in
fine-grained document perception and general abilities.
TextHawk, a multimodal large language model designed for document-oriented tasks, outperforms existing models by using specialized components for efficient text and image perception.
ReSampling and ReArrangement (ReSA) module Scalable Positional Embeddings (SPEs) Query Proposal Network (QPN) Multi-Level Cross-Attention (MLCA) mechanism