Region-Aware Pretraining for Open-Vocabulary Object Detection with
  Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a
contrastive image-text pretraining recipe to bridge the gap between image-level
pretraining and open-vocabulary object detection. At the pretraining phase, we
propose to randomly crop and resize regions of positional embeddings instead of
using the whole image positional embeddings. This better matches the use of
positional embeddings at region-level in the detection finetuning phase. In
addition, we replace the common softmax cross entropy loss in contrastive
learning with focal loss to better learn the informative yet difficult
examples. Finally, we leverage recent advances in novel object proposals to
improve open-vocabulary detection finetuning. We evaluate our full model on the
LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer.
RO-ViT achieves a state-of-the-art 32.1 AP_r on LVIS, surpassing the best
existing approach by +5.8 points in addition to competitive zero-shot transfer
detection. Surprisingly, RO-ViT improves the image-level representation as well
and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr
image-text retrieval benchmarks, outperforming competitive approaches with
larger models.
645dc4767c7258d904e97553
Dahun Kim
claimed_verified
2023-05-15T07:34:03.283Z
6461d65b5ea1bc9a9da98b88
/avatars/fd6e3715ca8f918ca074356244bb9ce9.svg
Dahun Kim
user
mcahny
645dc4767c7258d904e97554
Anelia Angelova
645dc4767c7258d904e97555
Weicheng Kuo
Region-Aware Pretraining for Open-Vocabulary Object Detection with
  Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a
contrastive image-text pretraining recipe to bridge the gap between image-level
pretraining and open-vocabulary object detection. At the pretraining phase, we
propose to randomly crop and resize regions of positional embeddings instead of
using the whole image positional embeddings. This better matches the use of
positional embeddings at region-level in the detection finetuning phase. In
addition, we replace the common softmax cross entropy loss in contrastive
learning with focal loss to better learn the informative yet difficult
examples. Finally, we leverage recent advances in novel object proposals to
improve open-vocabulary detection finetuning. We evaluate our full model on the
LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer.
RO-ViT achieves a state-of-the-art 32.1 AP_r on LVIS, surpassing the best
existing approach by +5.8 points in addition to competitive zero-shot transfer
detection. Surprisingly, RO-ViT improves the image-level representation as well
and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr
image-text retrieval benchmarks, outperforming competitive approaches with
larger models.
Region-aware Open-vocabulary Vision Transformers improve open-vocabulary object detection and image-text retrieval by using focal loss and novel object proposals in contrastive learning.
Region-aware Open-vocabulary Vision Transformers RO-ViT contrastive image-text pretraining positional embeddings focal loss open-vocabulary object detection LVIS COCO image-text retrieval zero-shot transfer AP_r image-level representation novel object proposals