Does DINOv3 Set a New Medical Vision Standard?
The advent of large-scale vision foundation models, pre-trained on diverse
natural images, has marked a paradigm shift in computer vision. However, how
the frontier vision foundation models' efficacies transfer to specialized
domains remains such as medical imaging remains an open question. This report
investigates whether DINOv3, a state-of-the-art self-supervised vision
transformer (ViT) that features strong capability in dense prediction tasks,
can directly serve as a powerful, unified encoder for medical vision tasks
without domain-specific pre-training. To answer this, we benchmark DINOv3
across common medical vision tasks, including 2D/3D classification and
segmentation on a wide range of medical imaging modalities. We systematically
analyze its scalability by varying model sizes and input image resolutions. Our
findings reveal that DINOv3 shows impressive performance and establishes a
formidable new baseline. Remarkably, it can even outperform medical-specific
foundation models like BiomedCLIP and CT-Net on several tasks, despite being
trained solely on natural images. However, we identify clear limitations: The
model's features degrade in scenarios requiring deep domain specialization,
such as in Whole-Slide Pathological Images (WSIs), Electron Microscopy (EM),
and Positron Emission Tomography (PET). Furthermore, we observe that DINOv3
does not consistently obey scaling law in the medical domain; performance does
not reliably increase with larger models or finer feature resolutions, showing
diverse scaling behaviors across tasks. Ultimately, our work establishes DINOv3
as a strong baseline, whose powerful visual features can serve as a robust
prior for multiple complex medical tasks. This opens promising future
directions, such as leveraging its features to enforce multiview consistency in
3D reconstruction.
68bf97aa207285de11b07ba5
Che Liu
68bf97aa207285de11b07ba6
Yinda Chen
68bf97aa207285de11b07ba7
Haoyuan Shi
68bf97aa207285de11b07ba8
Jinpeng Lu
68bf97aa207285de11b07ba9
Bailiang Jian
68bf97aa207285de11b07baa
Jiazhen Pan
68bf97aa207285de11b07bab
Linghan Cai
68bf97aa207285de11b07bac
Jiayi Wang
68bf97aa207285de11b07bad
Yundi Zhang
68bf97aa207285de11b07bae
Jun Li
68bf97aa207285de11b07baf
Cosmin I. Bercea
68bf97aa207285de11b07bb0
Cheng Ouyang
68bf97aa207285de11b07bb1
Chen Chen
68bf97aa207285de11b07bb2
Zhiwei Xiong
68bf97aa207285de11b07bb3
Benedikt Wiestler
68bf97aa207285de11b07bb4
Christian Wachinger
68bf97aa207285de11b07bb5
Daniel Rueckert
68bf97aa207285de11b07bb6
Wenjia Bai
68bf97aa207285de11b07bb7
Rossella Arcucci
Does DINOv3 Set a New Medical Vision Standard?
The advent of large-scale vision foundation models, pre-trained on diverse
natural images, has marked a paradigm shift in computer vision. However, how
the frontier vision foundation models' efficacies transfer to specialized
domains remains such as medical imaging remains an open question. This report
investigates whether DINOv3, a state-of-the-art self-supervised vision
transformer (ViT) that features strong capability in dense prediction tasks,
can directly serve as a powerful, unified encoder for medical vision tasks
without domain-specific pre-training. To answer this, we benchmark DINOv3
across common medical vision tasks, including 2D/3D classification and
segmentation on a wide range of medical imaging modalities. We systematically
analyze its scalability by varying model sizes and input image resolutions. Our
findings reveal that DINOv3 shows impressive performance and establishes a
formidable new baseline. Remarkably, it can even outperform medical-specific
foundation models like BiomedCLIP and CT-Net on several tasks, despite being
trained solely on natural images. However, we identify clear limitations: The
model's features degrade in scenarios requiring deep domain specialization,
such as in Whole-Slide Pathological Images (WSIs), Electron Microscopy (EM),
and Positron Emission Tomography (PET). Furthermore, we observe that DINOv3
does not consistently obey scaling law in the medical domain; performance does
not reliably increase with larger models or finer feature resolutions, showing
diverse scaling behaviors across tasks. Ultimately, our work establishes DINOv3
as a strong baseline, whose powerful visual features can serve as a robust
prior for multiple complex medical tasks. This opens promising future
directions, such as leveraging its features to enforce multiview consistency in
3D reconstruction.
DINOv3, a self-supervised vision transformer, demonstrates strong performance across various medical vision tasks without domain-specific pre-training, though it shows limitations in deeply specialized domains and does not consistently follow scaling laws in the medical domain.
self-supervised vision transformer ViT dense prediction tasks medical vision tasks 2D/3D classification segmentation medical imaging modalities scalability model sizes input image resolutions BiomedCLIP CT-Net Whole-Slide Pathological Images Electron Microscopy Positron Emission Tomography scaling law multiview consistency 3D reconstruction