OmniCaptioner: One Captioner to Rule Them All
We propose OmniCaptioner, a versatile visual captioning framework for
generating fine-grained textual descriptions across a wide variety of visual
domains. Unlike prior methods limited to specific image types (e.g., natural
images or geometric visuals), our framework provides a unified solution for
captioning natural images, visual text (e.g., posters, UIs, textbooks), and
structured visuals (e.g., documents, tables, charts). By converting low-level
pixel information into semantically rich textual representations, our framework
bridges the gap between visual and textual modalities. Our results highlight
three key advantages: (i) Enhanced Visual Reasoning with LLMs, where
long-context captions of visual modalities empower LLMs, particularly the
DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)
Improved Image Generation, where detailed captions improve tasks like
text-to-image generation and image transformation; and (iii) Efficient
Supervised Fine-Tuning (SFT), which enables faster convergence with less data.
We believe the versatility and adaptability of OmniCaptioner can offer a new
perspective for bridging the gap between language and visual modalities.
67f7676d0ab78ef7b16a820f
Yiting Lu
admin_assigned
2025-04-10T07:58:03.992Z
6614fb3d5aed02b298a4b469
/avatars/d0ddb4f989ad1a3f24128cc843347bde.svg
yiting lu
user
yeeeeeyy
67f7676d0ab78ef7b16a8210
Jiakang Yuan
admin_assigned
2025-04-10T08:08:31.119Z
64a3d1ddb3239f3e3892b24b
/avatars/7ce585f5fc1d077fb1d70cc18c4da2c1.svg
Jiakang Yuan
user
JiakangYuan
67f7676d0ab78ef7b16a8211
Zhen Li
claimed_verified
2025-04-10T13:23:24.319Z
6285a9133ab6642179158944
/avatars/6e10fa07c94141fcdbe0cab02bb731ca.svg
Zhen Li
user
Paper99
67f7676d0ab78ef7b16a8212
Shitian Zhao
claimed_verified
2025-04-11T08:23:49.529Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
67f7676d0ab78ef7b16a8213
Qi Qin
claimed_verified
2025-04-10T08:06:06.570Z
66bb136002fd8eb58bc84ffb
/avatars/122cb8f59c502392768099b3c2afe043.svg
qinqi
user
Dakerqi
67f7676d0ab78ef7b16a8214
Xinyue Li
claimed_verified
2025-04-11T07:25:55.970Z
66aba287b0f0b7411f511a47
/avatars/1450f182c38e80066ae5ea5df4fa218f.svg
Xinyue Li
user
Xxxy13
67f7676d0ab78ef7b16a8215
Le Zhuo
claimed_verified
2025-09-13T15:05:21.489Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
67f7676d0ab78ef7b16a8216
Licheng Wen
admin_assigned
2025-04-10T08:09:07.684Z
64a7c43ae940d769194055df
/avatars/441ccadd62e039fb8cb112f138ed917d.svg
Licheng Wen
user
Wayne-lc
67f7676d0ab78ef7b16a8217
Dongyang Liu
admin_assigned
2025-04-10T08:09:21.398Z
646f1bef075e11ca78da3bb7
https://cdn-avatars.huggingface.co/v1/production/uploads/646f1bef075e11ca78da3bb7/gNS-ikyZXYeMrf4a7HTQE.jpeg
Dongyang Liu (Chris Liu)
user
Cxxs
67f7676d0ab78ef7b16a8218
Yuewen Cao
67f7676d0ab78ef7b16a8219
Xiangchao Yan
admin_assigned
2025-04-10T08:09:35.388Z
65b88b92e0bde92c176a888a
/avatars/fc1cb54328ca93860e97fc73a3c1eb2f.svg
Xiangchao Yan
user
yxc97
67f7676d0ab78ef7b16a821a
Xin Li
67f7676d0ab78ef7b16a821b
Botian Shi
admin_assigned
2025-04-10T08:09:41.754Z
643df87f7cd64d872cb9fabd
/avatars/c53bfabcee08de448dde973915e8b31d.svg
Botian Shi
user
friskit
67f7676d0ab78ef7b16a821c
Tao Chen
67f7676d0ab78ef7b16a821d
Zhibo Chen
admin_assigned
2025-04-10T08:10:04.682Z
66d963e52e82d53d3b81031b
/avatars/302dbffc033ff47813a2435a2cec02f1.svg
Zhibo Chen
user
winhelp
67f7676d0ab78ef7b16a821e
Lei Bai
67f7676d0ab78ef7b16a821f
Bo Zhang
claimed_verified
2025-04-10T09:57:56.032Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
67f7676d0ab78ef7b16a8220
Peng Gao
OmniCaptioner: One Captioner to Rule Them All
We propose OmniCaptioner, a versatile visual captioning framework for
generating fine-grained textual descriptions across a wide variety of visual
domains. Unlike prior methods limited to specific image types (e.g., natural
images or geometric visuals), our framework provides a unified solution for
captioning natural images, visual text (e.g., posters, UIs, textbooks), and
structured visuals (e.g., documents, tables, charts). By converting low-level
pixel information into semantically rich textual representations, our framework
bridges the gap between visual and textual modalities. Our results highlight
three key advantages: (i) Enhanced Visual Reasoning with LLMs, where
long-context captions of visual modalities empower LLMs, particularly the
DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii)
Improved Image Generation, where detailed captions improve tasks like
text-to-image generation and image transformation; and (iii) Efficient
Supervised Fine-Tuning (SFT), which enables faster convergence with less data.
We believe the versatility and adaptability of OmniCaptioner can offer a new
perspective for bridging the gap between language and visual modalities.
OmniCaptioner generates detailed captions across various visual domains, enhancing visual reasoning with LLMs, improving image generation tasks, and enabling efficient supervised fine-tuning.
visual captioning framework low-level pixel information semantically rich textual representations LLMs DeepSeek-R1 long-context captions multimodal scenarios text-to-image generation image transformation supervised fine-tuning