OmniCaptioner: One Captioner to Rule Them All
We propose OmniCaptioner, a versatile visual captioning framework for
generating fine-grained textual descriptions across a wide variety of visual
domains. Unlike prior methods limited to specific image types (e.g., natural
images or geometric visuals), our framework provides a unified solution for
captioning natural images, visual text (e.g., posters, UIs, textbooks), and
structured visuals (e.g., documents, tables, charts). By converting low-level
pixel information into semantically rich textual representations, our framework
bridges the gap between visual and textual modalities. Our results highlight
three key advantages: (i) Enhanced Visual Reasoning with LLMs, where
long-context captions of visual modalities empower LLMs, particularly the
DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)
Improved Image Generation, where detailed captions improve tasks like
text-to-image generation and image transformation; and (iii) Efficient
Supervised Fine-Tuning (SFT), which enables faster convergence with less data.
We believe the versatility and adaptability of OmniCaptioner can offer a new
perspective for bridging the gap between language and visual modalities.
67f7676d0ab78ef7b16a820f
Yiting Lu
admin_assigned
2025-04-10T07:58:03.992Z
6614fb3d5aed02b298a4b469
/avatars/d0ddb4f989ad1a3f24128cc843347bde.svg
yiting lu
user
yeeeeeyy
67f7676d0ab78ef7b16a8210
Jiakang Yuan
admin_assigned
2025-04-10T08:08:31.119Z
64a3d1ddb3239f3e3892b24b
/avatars/7ce585f5fc1d077fb1d70cc18c4da2c1.svg
Jiakang Yuan
user
JiakangYuan
67f7676d0ab78ef7b16a8211
Zhen Li
claimed_verified
2025-04-10T13:23:24.319Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
67f7676d0ab78ef7b16a8212
Shitian Zhao
claimed_verified
2025-04-11T08:23:49.529Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
67f7676d0ab78ef7b16a8213
Qi Qin
claimed_verified
2025-04-10T08:06:06.570Z
66bb136002fd8eb58bc84ffb
/avatars/122cb8f59c502392768099b3c2afe043.svg
qinqi
user
Dakerqi
67f7676d0ab78ef7b16a8214
Xinyue Li
claimed_verified
2025-04-11T07:25:55.970Z
66aba287b0f0b7411f511a47
/avatars/1450f182c38e80066ae5ea5df4fa218f.svg
Xinyue Li
user
Xxxy13
67f7676d0ab78ef7b16a8215
Le Zhuo
67f7676d0ab78ef7b16a8216
Licheng Wen
admin_assigned
2025-04-10T08:09:07.684Z
64a7c43ae940d769194055df
/avatars/441ccadd62e039fb8cb112f138ed917d.svg
Licheng Wen
user
Wayne-lc
67f7676d0ab78ef7b16a8217
Dongyang Liu
admin_assigned
2025-04-10T08:09:21.398Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
67f7676d0ab78ef7b16a8218
Yuewen Cao
67f7676d0ab78ef7b16a8219
Xiangchao Yan
admin_assigned
2025-04-10T08:09:35.388Z
65b88b92e0bde92c176a888a
/avatars/fc1cb54328ca93860e97fc73a3c1eb2f.svg
Xiangchao Yan
user
yxc97
67f7676d0ab78ef7b16a821a
Xin Li
67f7676d0ab78ef7b16a821b
Botian Shi
admin_assigned
2025-04-10T08:09:41.754Z
643df87f7cd64d872cb9fabd
/avatars/c53bfabcee08de448dde973915e8b31d.svg
Botian Shi
user
friskit
67f7676d0ab78ef7b16a821c
Tao Chen
67f7676d0ab78ef7b16a821d
Zhibo Chen
admin_assigned
2025-04-10T08:10:04.682Z
66d963e52e82d53d3b81031b
/avatars/302dbffc033ff47813a2435a2cec02f1.svg
Zhibo Chen
user
winhelp
67f7676d0ab78ef7b16a821e
Lei Bai
67f7676d0ab78ef7b16a821f
Bo Zhang
claimed_verified
2025-04-10T09:57:56.032Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
67f7676d0ab78ef7b16a8220
Peng Gao
OmniCaptioner: One Captioner to Rule Them All
We propose OmniCaptioner, a versatile visual captioning framework for
generating fine-grained textual descriptions across a wide variety of visual
domains. Unlike prior methods limited to specific image types (e.g., natural
images or geometric visuals), our framework provides a unified solution for
captioning natural images, visual text (e.g., posters, UIs, textbooks), and
structured visuals (e.g., documents, tables, charts). By converting low-level
pixel information into semantically rich textual representations, our framework
bridges the gap between visual and textual modalities. Our results highlight
three key advantages: (i) Enhanced Visual Reasoning with LLMs, where
long-context captions of visual modalities empower LLMs, particularly the
DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)
Improved Image Generation, where detailed captions improve tasks like
text-to-image generation and image transformation; and (iii) Efficient
Supervised Fine-Tuning (SFT), which enables faster convergence with less data.
We believe the versatility and adaptability of OmniCaptioner can offer a new
perspective for bridging the gap between language and visual modalities.
visual captioning fine-grained textual descriptions visual domains visual text natural images structured visuals semantically rich textual representations multimodal scenarios LLMs (Large Language Models) DeepSeek-R1 long-context captions text-to-image generation image transformation Efficient Supervised Fine-Tuning (SFT) faster convergence