Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance
  for Self-supervised Monocular Depth Estimation
Current self-supervised monocular depth estimation (MDE) approaches encounter
performance limitations due to insufficient semantic-spatial knowledge
extraction. To address this challenge, we propose Hybrid-depth, a novel
framework that systematically integrates foundation models (e.g., CLIP and
DINO) to extract visual priors and acquire sufficient contextual information
for MDE. Our approach introduces a coarse-to-fine progressive learning
framework: 1) Firstly, we aggregate multi-grained features from CLIP (global
semantics) and DINO (local spatial details) under contrastive language
guidance. A proxy task comparing close-distant image patches is designed to
enforce depth-aware feature alignment using text prompts; 2) Next, building on
the coarse features, we integrate camera pose information and pixel-wise
language alignment to refine depth predictions. This module seamlessly
integrates with existing self-supervised MDE pipelines (e.g., Monodepth2,
ManyDepth) as a plug-and-play depth encoder, enhancing continuous depth
estimation. By aggregating CLIP's semantic context and DINO's spatial details
through language guidance, our method effectively addresses feature granularity
mismatches. Extensive experiments on the KITTI benchmark demonstrate that our
method significantly outperforms SOTA methods across all metrics, which also
indeed benefits downstream tasks like BEV perception. Code is available at
https://github.com/Zhangwenyao1/Hybrid-depth.
68ecedfcb0aa732b88f31fb9
Wenyao Zhang
68ecedfcb0aa732b88f31fba
Hongsi Liu
68ecedfcb0aa732b88f31fbb
Bohan Li
68ecedfcb0aa732b88f31fbc
Jiawei He
68ecedfcb0aa732b88f31fbd
Zekun Qi
68ecedfcb0aa732b88f31fbe
Yunnan Wang
68ecedfcb0aa732b88f31fbf
Shengyang Zhao
68ecedfcb0aa732b88f31fc0
Xinqiang Yu
68ecedfcb0aa732b88f31fc1
Wenjun Zeng
68ecedfcb0aa732b88f31fc2
Xin Jin
Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance
  for Self-supervised Monocular Depth Estimation
Current self-supervised monocular depth estimation (MDE) approaches encounter
performance limitations due to insufficient semantic-spatial knowledge
extraction. To address this challenge, we propose Hybrid-depth, a novel
framework that systematically integrates foundation models (e.g., CLIP and
DINO) to extract visual priors and acquire sufficient contextual information
for MDE. Our approach introduces a coarse-to-fine progressive learning
framework: 1) Firstly, we aggregate multi-grained features from CLIP (global
semantics) and DINO (local spatial details) under contrastive language
guidance. A proxy task comparing close-distant image patches is designed to
enforce depth-aware feature alignment using text prompts; 2) Next, building on
the coarse features, we integrate camera pose information and pixel-wise
language alignment to refine depth predictions. This module seamlessly
integrates with existing self-supervised MDE pipelines (e.g., Monodepth2,
ManyDepth) as a plug-and-play depth encoder, enhancing continuous depth
estimation. By aggregating CLIP's semantic context and DINO's spatial details
through language guidance, our method effectively addresses feature granularity
mismatches. Extensive experiments on the KITTI benchmark demonstrate that our
method significantly outperforms SOTA methods across all metrics, which also
indeed benefits downstream tasks like BEV perception. Code is available at
https://github.com/Zhangwenyao1/Hybrid-depth.
Hybrid-depth framework integrates CLIP and DINO with language guidance to enhance self-supervised monocular depth estimation by addressing semantic and spatial feature mismatches.
self-supervised monocular depth estimation foundation models CLIP DINO coarse-to-fine progressive learning contrastive language guidance proxy task depth-aware feature alignment camera pose information pixel-wise language alignment KITTI benchmark BEV perception