Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading of
text-intensive images. Pre-trained on large-scale text-intensive images,
Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
generating spatially-aware text blocks, where each block of text is assigned
its spatial coordinates within the image, and (2) producing structured text
output that captures styles and structures into the markdown format. This
unified multimodal literate capability is achieved through a shared Transformer
architecture, task-specific prompts, and flexible text representations. We
evaluate Kosmos-2.5 on end-to-end document-level text recognition and
image-to-markdown text generation. Furthermore, the model can be readily
adapted for any text-intensive image understanding task with different prompts
through supervised fine-tuning, making it a general-purpose tool for real-world
applications involving text-rich images. This work also paves the way for the
future scaling of multimodal large language models.
650bc0752ec41a31f31b13e8
Tengchao Lv
admin_assigned
2023-09-21T09:33:05.055Z
614c7f86cbb5e52274a4024b
/avatars/ff1a9079a1182d84397012fad746807c.svg
Tengchao Lv
user
estyle
650bc0752ec41a31f31b13e9
Yupan Huang
admin_assigned
2023-09-21T09:41:39.856Z
625d7fc87341c641426c7665
/avatars/3083ebe0343a03332b15f0e7f76ba1d7.svg
Yupan Huang
user
HYPJUDY
650bc0752ec41a31f31b13ea
Jingye Chen
admin_assigned
2023-09-21T09:44:20.792Z
63355049ce9a15ea6c4aa8cc
https://cdn-avatars.huggingface.co/v1/production/uploads/63355049ce9a15ea6c4aa8cc/aD4NI5d_ozAuQVv5nsNF4.jpeg
JingyeChen
user
JingyeChen
650bc0752ec41a31f31b13eb
Lei Cui
admin_assigned
2023-09-21T09:44:50.418Z
5f5a4fbe10b2753d9000c7fc
/avatars/0763b25c13733ef4eb757862ba6d603e.svg
Lei Cui
user
wolfshow
650bc0752ec41a31f31b13ec
Shuming Ma
admin_assigned
2023-09-21T09:44:56.951Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
650bc0752ec41a31f31b13ed
Yaoyao Chang
650bc0752ec41a31f31b13ee
Shaohan Huang
admin_assigned
2023-09-21T09:45:19.232Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
650bc0752ec41a31f31b13ef
Wenhui Wang
650bc0752ec41a31f31b13f0
Li Dong
claimed_verified
2024-10-08T15:20:34.657Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
650bc0752ec41a31f31b13f1
Weiyao Luo
admin_assigned
2023-09-21T09:46:17.438Z
63a409a584a6a25c65bb3e2c
/avatars/a08fb6052e7bde48cd3926c3d9bab30e.svg
Weiyao Luo
user
weiyao993
650bc0752ec41a31f31b13f2
Shaoxiang Wu
650bc0752ec41a31f31b13f3
Guoxin Wang
650bc0752ec41a31f31b13f4
Cha Zhang
650bc0752ec41a31f31b13f5
Furu Wei
admin_assigned
2023-09-21T09:46:49.821Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading of
text-intensive images. Pre-trained on large-scale text-intensive images,
Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
generating spatially-aware text blocks, where each block of text is assigned
its spatial coordinates within the image, and (2) producing structured text
output that captures styles and structures into the markdown format. This
unified multimodal literate capability is achieved through a shared Transformer
architecture, task-specific prompts, and flexible text representations. We
evaluate Kosmos-2.5 on end-to-end document-level text recognition and
image-to-markdown text generation. Furthermore, the model can be readily
adapted for any text-intensive image understanding task with different prompts
through supervised fine-tuning, making it a general-purpose tool for real-world
applications involving text-rich images. This work also paves the way for the
future scaling of multimodal large language models.
Kosmos-2.5, a unified multimodal model, generates spatially-aware and structured text from text-intensive images using a Transformer architecture and task-specific prompts.
multimodal literate model machine reading spatially-aware text blocks markdown format Transformer architecture task-specific prompts text representations end-to-end document-level text recognition image-to-markdown text generation supervised fine-tuning multimodal large language models