DepthLM: Metric Depth From Vision Language Models
Vision language models (VLMs) can flexibly address various vision tasks
through text interactions. Although successful in semantic understanding,
state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from
2D inputs. On the other hand, expert pure vision models achieve super-human
accuracy in metric depth estimation, a key 3D understanding task. However, they
require task-specific architectures and losses. Such difference motivates us to
ask: Can VLMs reach expert-level accuracy without architecture or loss change?
We take per-pixel metric depth estimation as the representative task and show
that the answer is yes! Surprisingly, comprehensive analysis shows that
text-based supervised-finetuning with sparse labels is sufficient for VLMs to
unlock strong 3D understanding, no dense prediction head or complex
regression/regularization loss is needed. The bottleneck for VLMs lies actually
in pixel reference and cross-dataset camera ambiguity, which we address through
visual prompting and intrinsic-conditioned augmentation. With much smaller
models, our method DepthLM surpasses the accuracy of most advanced VLMs by over
2x, making VLMs for the first time comparable with pure vision models.
Interestingly, without explicit enforcement during training, VLMs trained with
DepthLM naturally avoids over-smoothing, having much fewer flying points at
boundary regions than pure vision models. The simplicity of DepthLM also
enables a single VLM to cover various 3D tasks beyond metric depth. Our code
and model will be released at the link below.
68dc745a4159d1f2418f999c
Zhipeng Cai
claimed_verified
2025-10-15T17:03:51.120Z
6419fe794577db917f6d17fc
/avatars/ed10371a0082c5e02d32b76217200207.svg
zhipeng cai
user
zcai
68dc745a4159d1f2418f999d
Ching-Feng Yeh
68dc745a4159d1f2418f999e
Hu Xu
68dc745a4159d1f2418f999f
Zhuang Liu
68dc745a4159d1f2418f99a0
Gregory Meyer
68dc745a4159d1f2418f99a1
Xinjie Lei
68dc745a4159d1f2418f99a2
Changsheng Zhao
68dc745a4159d1f2418f99a3
Shang-Wen Li
68dc745a4159d1f2418f99a4
Vikas Chandra
68dc745a4159d1f2418f99a5
Yangyang Shi
DepthLM: Metric Depth From Vision Language Models
Vision language models (VLMs) can flexibly address various vision tasks
through text interactions. Although successful in semantic understanding,
state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from
2D inputs. On the other hand, expert pure vision models achieve super-human
accuracy in metric depth estimation, a key 3D understanding task. However, they
require task-specific architectures and losses. Such difference motivates us to
ask: Can VLMs reach expert-level accuracy without architecture or loss change?
We take per-pixel metric depth estimation as the representative task and show
that the answer is yes! Surprisingly, comprehensive analysis shows that
text-based supervised-finetuning with sparse labels is sufficient for VLMs to
unlock strong 3D understanding, no dense prediction head or complex
regression/regularization loss is needed. The bottleneck for VLMs lies actually
in pixel reference and cross-dataset camera ambiguity, which we address through
visual prompting and intrinsic-conditioned augmentation. With much smaller
models, our method DepthLM surpasses the accuracy of most advanced VLMs by over
2x, making VLMs for the first time comparable with pure vision models.
Interestingly, without explicit enforcement during training, VLMs trained with
DepthLM naturally avoids over-smoothing, having much fewer flying points at
boundary regions than pure vision models. The simplicity of DepthLM also
enables a single VLM to cover various 3D tasks beyond metric depth. Our code
and model will be released at the link below.
Text-based supervised fine-tuning with sparse labels enables vision language models to achieve expert-level accuracy in 3D depth estimation without requiring task-specific architectures or losses.
vision language models VLMs GPT-5 metric depth estimation text-based supervised-finetuning sparse labels visual prompting intrinsic-conditioned augmentation DepthLM over-smoothing flying points