One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
Zero-shot captioners are recently proposed models that utilize common-space
vision-language representations to caption images without relying on paired
image-text data. To caption an image, they proceed by textually decoding a
text-aligned image feature, but they limit their scope to global
representations and whole-image captions. We present , a
unified framework for zero-shot captioning that shifts from an image-centric to
a patch-centric paradigm, enabling the captioning of arbitrary regions without
the need of region-level supervision. Instead of relying on global image
representations, we treat individual patches as atomic captioning units and
aggregate them to describe arbitrary regions, from single patches to
non-contiguous areas and entire images. We analyze the key ingredients that
enable current latent captioners to work in our novel proposed framework.
Experiments demonstrate that backbones producing meaningful, dense visual
features, such as DINO, are key to achieving state-of-the-art performance in
multiple region-based captioning tasks. Compared to other baselines and
state-of-the-art competitors, our models achieve better performance on
zero-shot dense, region-set, and a newly introduced trace captioning task,
highlighting the effectiveness of patch-wise semantic representations for
scalable caption generation. Project page at https://paciosoft.com/Patch-ioner/ .
68e3c2e473e20ab5778421ed
Lorenzo Bianchi
claimed_verified
2025-10-08T08:03:13.038Z
63c67770b167901a39346c27
https://cdn-avatars.huggingface.co/v1/production/uploads/1673951039741-noauth.jpeg
Lorenzo Bianchi
user
lorebianchi98
68e3c2e473e20ab5778421ee
Giacomo Pacini
claimed_verified
2025-10-07T12:28:10.096Z
640d000904c679553cfa275b
https://cdn-avatars.huggingface.co/v1/production/uploads/640d000904c679553cfa275b/v18gSEHfT1FvEpYTkWJNj.png
Giacomo Pacini
user
Ruggero1912
68e3c2e473e20ab5778421ef
Fabio Carrara
claimed_verified
2025-10-14T14:24:34.662Z
64567c51263efcb14a5b83cf
/avatars/bc5d19a2b7d807fa8f4ec0958efeea96.svg
Fabio Carrara
user
fabiocarrara
68e3c2e473e20ab5778421f0
Nicola Messina
68e3c2e473e20ab5778421f1
Giuseppe Amato
68e3c2e473e20ab5778421f2
Fabrizio Falchi
One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
Zero-shot captioners are recently proposed models that utilize common-space
vision-language representations to caption images without relying on paired
image-text data. To caption an image, they proceed by textually decoding a
text-aligned image feature, but they limit their scope to global
representations and whole-image captions. We present , a
unified framework for zero-shot captioning that shifts from an image-centric to
a patch-centric paradigm, enabling the captioning of arbitrary regions without
the need of region-level supervision. Instead of relying on global image
representations, we treat individual patches as atomic captioning units and
aggregate them to describe arbitrary regions, from single patches to
non-contiguous areas and entire images. We analyze the key ingredients that
enable current latent captioners to work in our novel proposed framework.
Experiments demonstrate that backbones producing meaningful, dense visual
features, such as DINO, are key to achieving state-of-the-art performance in
multiple region-based captioning tasks. Compared to other baselines and
state-of-the-art competitors, our models achieve better performance on
zero-shot dense, region-set, and a newly introduced trace captioning task,
highlighting the effectiveness of patch-wise semantic representations for
scalable caption generation. Project page at https://paciosoft.com/Patch-ioner/ .
A patch-centric framework for zero-shot captioning achieves state-of-the-art performance by using dense visual features from models like DINO to caption arbitrary image regions.
zero-shot captioners common-space vision-language representations text-aligned image feature patch-centric paradigm region-level supervision atomic captioning units latent captioners dense visual features DINO region-based captioning tasks zero-shot dense captioning region-set captioning trace captioning task patch-wise semantic representations