Revisiting DETR Pre-training for Object Detection
Motivated by that DETR-based approaches have established new records on COCO
detection and segmentation benchmarks, many recent endeavors show increasing
interest in how to further improve DETR-based approaches by pre-training the
Transformer in a self-supervised manner while keeping the backbone frozen. Some
studies already claimed significant improvements in accuracy. In this paper, we
take a closer look at their experimental methodology and check if their
approaches are still effective on the very recent state-of-the-art such as
H-Deformable-DETR. We conduct thorough experiments on COCO object
detection tasks to study the influence of the choice of pre-training datasets,
localization, and classification target generation schemes. Unfortunately, we
find the previous representative self-supervised approach such as DETReg, fails
to boost the performance of the strong DETR-based approaches on full data
regimes. We further analyze the reasons and find that simply combining a more
accurate box predictor and Objects365 benchmark can significantly improve the
results in follow-up experiments. We demonstrate the effectiveness of our
approach by achieving strong object detection results of AP=59.3% on COCO
val set, which surpasses H-Deformable-DETR + Swin-L by +1.4%.
Last, we generate a series of synthetic pre-training datasets by combining the
very recent image-to-text captioning models (LLaVA) and text-to-image
generative models (SDXL). Notably, pre-training on these synthetic datasets
leads to notable improvements in object detection performance. Looking ahead,
we anticipate substantial advantages through the future expansion of the
synthetic pre-training dataset.
64cb33d500377e28486f8486
Yan Ma
64cb33d500377e28486f8487
Weicong Liang
64cb33d500377e28486f8488
Yiduo Hao
claimed_verified
2025-01-29T08:55:43.434Z
64d26a067a4b4217eff1effb
/avatars/b684a6061b71fde92939cb1f73351d21.svg
Yiduo Hao
user
Scherzando
64cb33d500377e28486f8489
Bohan Chen
64cb33d500377e28486f848a
Xiangyu Yue
64cb33d500377e28486f848b
Chao Zhang
64cb33d500377e28486f848c
Yuhui Yuan
claimed_verified
2025-03-28T08:39:35.114Z
631f108bb45367a05fe74260
/avatars/c20da6800b643728062712d9a2771648.svg
Researcher
user
YuanYuhui
Revisiting DETR Pre-training for Object Detection
Motivated by that DETR-based approaches have established new records on COCO
detection and segmentation benchmarks, many recent endeavors show increasing
interest in how to further improve DETR-based approaches by pre-training the
Transformer in a self-supervised manner while keeping the backbone frozen. Some
studies already claimed significant improvements in accuracy. In this paper, we
take a closer look at their experimental methodology and check if their
approaches are still effective on the very recent state-of-the-art such as
H-Deformable-DETR. We conduct thorough experiments on COCO object
detection tasks to study the influence of the choice of pre-training datasets,
localization, and classification target generation schemes. Unfortunately, we
find the previous representative self-supervised approach such as DETReg, fails
to boost the performance of the strong DETR-based approaches on full data
regimes. We further analyze the reasons and find that simply combining a more
accurate box predictor and Objects365 benchmark can significantly improve the
results in follow-up experiments. We demonstrate the effectiveness of our
approach by achieving strong object detection results of AP=59.3% on COCO
val set, which surpasses H-Deformable-DETR + Swin-L by +1.4%.
Last, we generate a series of synthetic pre-training datasets by combining the
very recent image-to-text captioning models (LLaVA) and text-to-image
generative models (SDXL). Notably, pre-training on these synthetic datasets
leads to notable improvements in object detection performance. Looking ahead,
we anticipate substantial advantages through the future expansion of the
synthetic pre-training dataset.
Thorough evaluation reveals that synthetic pre-training datasets derived from recent captioning and generative models significantly improve the performance of DETR-based object detection approaches.
DETR Transformer self-supervised pre-training backbone COCO object detection pre-training datasets localization classification target generation DETReg $\mathcal{H}$-Deformable-DETR Swin-L LLaVA SDXL synthetic pre-training datasets