DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
Dense visual prediction tasks have been constrained by their reliance on
predefined categories, limiting their applicability in real-world scenarios
where visual concepts are unbounded. While Vision-Language Models (VLMs) like
CLIP have shown promise in open-vocabulary tasks, their direct application to
dense prediction often leads to suboptimal performance due to limitations in
local feature representation. In this work, we present our observation that
CLIP's image tokens struggle to effectively aggregate information from
spatially or semantically related regions, resulting in features that lack
local discriminability and spatial consistency. To address this issue, we
propose DeCLIP, a novel framework that enhances CLIP by decoupling the
self-attention module to obtain ``content'' and ``context'' features
respectively. The ``content'' features are aligned with image crop
representations to improve local discriminability, while ``context'' features
learn to retain the spatial correlations under the guidance of vision
foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP
significantly outperforms existing methods across multiple open-vocabulary
dense prediction tasks, including object detection and semantic segmentation.
Code is available at magenta{https://github.com/xiaomoguhz/DeCLIP}.
681d615fbd89ba9ceb5e94bc
Junjie Wang
claimed_verified
2025-05-09T07:21:48.630Z
64a385281cbf675203fbb7df
/avatars/f259d080d3127c45bcf564a8d1fafcc6.svg
Junjie Wang
user
xiaomoguhzz
681d615fbd89ba9ceb5e94bd
Bin Chen
admin_assigned
2025-05-15T13:47:00.478Z
647d70d136e109abce415e0e
/avatars/c89864c663cbf3256e34785be85561bc.svg
Bin Chen
user
BinChen68
681d615fbd89ba9ceb5e94be
Yulin Li
681d615fbd89ba9ceb5e94bf
Bin Kang
admin_assigned
2025-05-15T13:47:31.037Z
67069836f95c7ec727df2806
/avatars/fbe87f41e680362595b5864e690b62b4.svg
bin kang
user
tygeer
681d615fbd89ba9ceb5e94c0
Yichi Chen
681d615fbd89ba9ceb5e94c1
Zhuotao Tian
admin_assigned
2025-05-15T13:48:12.084Z
64d104a37a7305c5895bd720
/avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
zt tian
user
tianzhuotao
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
Dense visual prediction tasks have been constrained by their reliance on
predefined categories, limiting their applicability in real-world scenarios
where visual concepts are unbounded. While Vision-Language Models (VLMs) like
CLIP have shown promise in open-vocabulary tasks, their direct application to
dense prediction often leads to suboptimal performance due to limitations in
local feature representation. In this work, we present our observation that
CLIP's image tokens struggle to effectively aggregate information from
spatially or semantically related regions, resulting in features that lack
local discriminability and spatial consistency. To address this issue, we
propose DeCLIP, a novel framework that enhances CLIP by decoupling the
self-attention module to obtain ``content'' and ``context'' features
respectively. The ``content'' features are aligned with image crop
representations to improve local discriminability, while ``context'' features
learn to retain the spatial correlations under the guidance of vision
foundation models, such as DINO. Extensive experiments demonstrate that DeCLIP
significantly outperforms existing methods across multiple open-vocabulary
dense prediction tasks, including object detection and semantic segmentation.
Code is available at magenta{https://github.com/xiaomoguhz/DeCLIP}.
DeCLIP enhances CLIP by separating content and context features, improving performance on open-vocabulary dense prediction tasks like object detection and semantic segmentation.
Vision-Language Models (VLMs) CLIP image tokens self-attention module content features context features image crop representations DINO spatial correlations object detection semantic segmentation