DeepAI AI Chat
Log In Sign Up

Open-Vocabulary Object Detection Using Captions

by   Alireza Zareian, et al.

Despite the remarkable accuracy of deep neural networks in object detection, they are costly to train and scale due to supervision requirements. Particularly, learning more object categories typically requires proportionally more bounding box annotations. Weakly supervised and zero-shot learning techniques have been explored to scale object detectors to more categories with less supervision, but they have not been as successful and widely adopted as supervised models. In this paper, we put forth a novel formulation of the object detection problem, namely open-vocabulary object detection, which is more general, more practical, and more effective than weakly supervised and zero-shot approaches. We propose a new method to train object detectors using bounding box annotations for a limited set of object categories, as well as image-caption pairs that cover a larger variety of objects at a significantly lower cost. We show that the proposed method can detect and localize objects for which no bounding box annotation is provided during training, at a significantly higher accuracy than zero-shot approaches. Meanwhile, objects with bounding box annotation can be detected almost as accurately as supervised methods, which is significantly better than weakly supervised baselines. Accordingly, we establish a new state of the art for scalable object detection.


page 1

page 4

page 11

page 12


Towards Open Vocabulary Object Detection without Human-provided Bounding Boxes

Despite great progress in object detection, most existing methods are li...

Exploiting Web Images for Weakly Supervised Object Detection

In recent years, the performance of object detection has advanced signif...

VEIL: Vetting Extracted Image Labels from In-the-Wild Captions for Weakly-Supervised Object Detection

The use of large-scale vision-language datasets is limited for object de...

Weakly Supervised Faster-RCNN+FPN to classify animals in camera trap images

Camera traps have revolutionized the animal research of many species tha...

ORF-Net: Deep Omni-supervised Rib Fracture Detection from Chest CT Scans

Most of the existing object detection works are based on the bounding bo...

Faster Bounding Box Annotation for Object Detection in Indoor Scenes

This paper proposes an approach for rapid bounding box annotation for ob...

Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection

Learning to localize and name object instances is a fundamental problem ...

Code Repositories


A new framework for open-vocabulary object detection, based on maskrcnn-benchmark

view repo