Learning Detection with Diverse Proposals at CVPR 2017
To predict a set of diverse and informative proposals with enriched representations, this paper introduces a differentiable Determinantal Point Process (DPP) layer that is able to augment the object detection architectures. Most modern object detection architectures, such as Faster R-CNN, learn to localize objects by minimizing deviations from the ground-truth but ignore correlation between multiple proposals and object categories. Non-Maximum Suppression (NMS) as a widely used proposal pruning scheme ignores label- and instance-level relations between object candidates resulting in multi-labeled detections. In the multi-class case, NMS selects boxes with the largest prediction scores ignoring the semantic relation between categories of potential election. In contrast, our trainable DPP layer, allowing for Learning Detection with Diverse Proposals (LDDP), considers both label-level contextual information and spatial layout relationships between proposals without increasing the number of parameters of the network, and thus improves location and category specifications of final detected bounding boxes substantially during both training and inference schemes. Furthermore, we show that LDDP keeps it superiority over Faster R-CNN even if the number of proposals generated by LDPP is only 30READ FULL TEXT VIEW PDF
Region based object detectors achieve the state-of-the-art performance, ...
Object detection is one of the most active areas in computer vision, whi...
Object proposals are an ensemble of bounding boxes with high potential t...
Large scale object detection with thousands of classes introduces the pr...
In this paper, we propose to learn a deep fitting degree scoring network...
Existing object proposal approaches use primarily bottom-up cues to rank...
Many high-level video understanding methods require input in the form of...
Learning Detection with Diverse Proposals at CVPR 2017
. However, object detection is still more challenging than image classification as it aims at both localizing and classifying objects. Accurate localization of objects in each image requires both well-processed “candidate” object locations and “selected refined” boxes with precise locations. Looking at the object detection problem as an extractive image summarization and representation task, the set of all predicted bounding boxes per image should be as informative and non-repetitive as possible.
proposed an efficient approach for object proposal classification and localization with a multi-task loss function during training. The training process in such methods contains a fine-tuning stage, which jointly optimizes a softmax classifier and a bounding-box regressor. Such a bounding box regressor tries to minimize the distance between the candidate object proposals with their corresponding ground-truth boxes for each category of objects. However, it does not consider relation “between” boxes in terms of location and context while learning a representation model. In this paper, we propose a new loss layer added to the other two softmax classifier and bounding-box regressor layers (all included in the multi-task loss for training the deep model) which formulates the discriminative contextual information as well as mutual relation between boxes into a Determinantal Point Process (DPP) loss function. This DPP loss finds a subset of diverse bounding boxes using the outputs of the other two loss functions (namely, the probability of each proposal to belong to each object category as well as the location information of the proposals) and will reinforce them in finding more accurate object instances in the end, as illustrated in Figure 1. We employ our DPP loss to maximize the likelihood of an accurate selection given the pool of overlapping background and non-background boxes over multiple categories.
Inference in state-of-the-art detection methods [20, 6, 18, 16] is generally based on Non-Maximum Suppression (NMS), which considers only the overlap between candidate boxes per class label and ignores their semantic relationship. We propose a DPP inference scheme to select a set of non-repetitive high-quality boxes per image taking into account spatial layout, category-level analogy between proposals, as well as their quality score obtained from deep trained model. We call our proposed model as “Learning Detection with Diverse Proposals Network – LDDP-Net”.
Our proposed loss function for representation enhancement and more accurate inference can be applied on any deep network architecture for object detection. In our experiments below we focus on the Faster R-CNN model to show the significant performance improvement added by our DPP model. We demonstrate the effect of our proposed DPP loss layer in accurate object localization during training as well as inference on the benchmark detection data sets PASCAL VOC and MS COCO based on average precision and average recall detection metrics.
To sum up, we make following contributions in this work:
We propose to explicitly pursue diversity on generated object proposals and introduce the strategy of learning detection with diverse proposals.
We introduce a DPP layer that is able to maximize diversity favorably in an end-to-end trainable way. Besides it is compatible with many existing state-of-the-art detection architectures and thus able to augment them effectively.
Experiments on Pascal VOC and MS COCO data sets clearly demonstrate the superiority of diverse proposals and effectiveness of our proposed method on producing diverse detections.
LDDP Code is available at https://github.com/azadis/LDDP.
Faster R-CNN 
as a unified deep convolutional framework for generating and refining region proposals alternates between fine-tuning for proposals using a fully convolutional Region Proposal Network (RPN) and fine-tuning for object detection by Fast R-CNN model. Keeping the object proposals generated from RPN fixed, they will be mapped into convolutional features through several convolutional and max-pooling layers. An RoI pooling layer converts the features inside each region of interest (RoI) into a fixed-length feature vector afterwards, which will be fed into a sequence of fully connected layers.
The loss function on the top layer of detection model is a multi-task loss dealing with both classification and localization of object proposals: the softmax loss layer outputs a discrete probability distribution over thecategories of objects in addition to the background for each object proposal, and the bounding box regressor layer determines location offsets per object proposal for all categories.
Applying a diversity-ignited model can reinforce the network to limit the boxes around each object while they have minimum overlap with other object bounding boxes in the image. It will also “select” boxes in such a way to make their collection as informative as possible given the minimum possible number of proposals. We define such a model through a DPP loss layer added to the other two loss functions introduced in the Faster R-CNN architecture.
On the other hand, inference in Region-based CNN models as well as other state-of-the-art networks is done through NMS which selects boxes with highest detection scores for each category. Giving a priority to the detection scores, it might finally end up in selecting overlapping detection boxes and miss the best possible set of non-overlapping ones with acceptable score. Besides, NMS neglects the semantic relations between categories as its selection is done category-wisely. We address all such problems through a probabilistic DPP inference scheme which jointly considers all spatial layout, contextual information, and semantic similarities between object candidates and selects the best probable ones. In the following section, we define our learn-able DPP loss layer, show how to back-propagate through this layer while training the model (summarized in Alg. 1), and then clarify how to infer the best non-overlapping boxes from the predicted proposals.
Determinantal Point Processes (DPPs) are natural models for diverse subset selection problems . In the selection problem, there is a trade-off between two influential metrics: The selected subset of items, or in other words their summary, should be “representative” and cover significant amount of information from the whole set. Besides, the information should be passed “efficiently” through this selection; the selection should be diverse and non-repetitive. We briefly explain a determinantal point process here and refer the readers to  for an in-depth discussion.
A point process on a discrete set is a probability measure on the set of all subsets of . is called a determinantal point process (DPP) if:
is an identity matrix,is the kernel matrix, and is a random subset of . The positive semi-definite kernel matrix indexed by elements of models diversity among items: the marginal probability of inclusion of each single item is proportional to the diagonal values of the kernel matrix , while correlation between each pair of items is proportional to the off-diagonal values of the kernel. As a result, subsets with higher diversity measured by the kernel have higher likelihood.
In the object detection scenario, items are indeed the set of proposals in the image produced by the region proposal network. Given the bounding box probability scores of softmax loss layer and location offsets from the bounding box regressor layer for the given set of proposals for image , , we seek for a precise and diverse set of boxes, . In other words, the probability of selecting informative non-redundant boxes given the whole set of background and non-background proposals should be maximized during training. Simultaneuosly, the probability of selecting background boxes, denoted by , should be minimized. We employ a learn-able determinantal point process layer by maximizing the log-likelihood of the training set and minimizing the log-likelihood of background proposals election:
where refers to the parameters of the deep network.
For simplicity, we assume that number of images per iteration is one and thus, remove index from our notations. We follow the same mini-batch setting as in Faster R-CNN network  where is the number of object proposals in each iteration or the size of mini-batch.
Given a list of object proposals as output of the RPN network,modeled as a determinantal point process would imply which boxes should be selected with a high probability:
The above distribution model considers relation among different proposals (indexed by ) through their similarity encoded by (which is the product of their spatial overlap rate and category similarity) as well as their individual quality score . We now proceed to explain each quantity in the distribution model.
We set with a small to make sure that the ensemble matrix is positive semi-definite, which is important for a proper DPP model definition. Here, is a normalizing factor, and is the Intersection-over-Union associated with each pair of proposals and , where is the area covered by the proposal . Motivated by [14, 17], we consider the semantic relation among proposals, , as the semantic similarity between the labels of each pair of proposals . Here refers to the lowest common subsumer of the category labels and in the WordNet hierarchy. The information content of class is computed as where is the probability of occurrence of an instance with label among all images. Posterior probability of selecting background boxes, , follows the same determinantal point process formulation as in Eq. (2.1.1). The difference between these two posterior probabilities lies in how we measure quality of proposals .
In general, the classification score over categories as well as overlap with the bounding box target can be used to define the quality score for proposal
. The classification scores are computed in the fully-connected layer before “softmax” layer in the Faster R-CNN architecture, and location of each bounding box is given by the inner product layer feeding into the “bounding box regressor”. The exact definition offor different proposals in the two terms of log-likelihood function Eq. (1) varies according to the general goal of increasing scores of high-quality boxes in their ground-truth label and increasing scores of background boxes in the background category.
One should note that is an ideal extractive summary of input which is the list of proposals in each mini-batch, (e.g. ) . Thus, we apply a maximum a posteriori (MAP) DPP inference, Alg. 2, in each iteration of the training algorithm to determine the set of representative boxes from the set of proposals. If ground-truth boxes exist among the proposals in each mini-batch, they will be automatically selected as the set through MAP. However, if they don’t exist among the proposals, MAP will select the best summarizing ones. To make sure that selected boxes, , are accurate and close to ground-truth boxes, we only select from a “subset” of proposals in the mini-batch with high overlap with their bounding box targets. Also, we define the quality of boxes only based on their overlap with their bounding box target in this step of applying MAP, as summarized in Alg. 1. Thus, by selecting as the set of best representations of ground-truth boxes in , maximizing results in maximizing selection of ground-truth boxes, which corresponds to maximizing the probability of training data.
On the other hand, to specify the set of more probable background proposals as , given the set of proposals in each mini-batch , we define as the set of all proposals in except those that have high overlap with ground-truth and their associated predicted label matches their ground-truth label. As mentioned before, the goal here is to minimize the probability of selecting “background proposals” as the representative boxes in each image.
To complete the DPP model description, we define the quality scores of the proposals as follows.
For the first term , we define the quality of boxes as:
where is the output of the inner product layer before the softmax loss layer, is the weight vector learned for category and is the fc7 feature learned for proposal . Moreover, denotes the weight vector for the corresponding ground-truth label of proposal , and shows the background category. Note that the goal is to increase the score of boxes in in their ground-truth label and the score of other boxes in the back-ground category. Derivatives of log-likelihood function with respect to in Eq. (5), (6) clarifies all above definitions.
The second term is designed for minimizing the probability of selection of background boxes which results in a boost in the scores of such proposals in their background category. We thus define the quality of proposals for this term as:
The effect of involving in quality scores during training appears in computing the gradient, Eq. (5),(6), where a larger gradient would be passed for boxes with higher overlap with their bbox target. It means more accurate boxes will move toward being selected (achieve higher softmax prediction scores) faster than others.
To avoid degrading the relatively accurate bounding boxes which are not selected in , in Eq. (1), during the learning process, we exclude from all the boxes that have high overlap with boxes in and their label matches their ground-truth category (in both , ).
We modify the negative log-likelihood function in Eq. (1) such that the two terms get balanced according to the number of selected proposals in . This DPP loss function, depends on the inputs of both softmax loss and bounding box regressor in the deep network. Since the ensemble matrix , Eq. (2.1.1), and the consequent log-likelihood function in Eq. (1) are functions of the parameters of deep network, we incorporate the gradient of our DPP loss layer into the back-propagation scheme of the detection network. We assume the location coordinates of proposals are fixed and only consider the outputs of the fully-connected layer before softmax loss as the parameters of our loss layer.
where . Therefore, minimizing the negative log-likelihood increases the scores of representative boxes in their ground-truth label and background boxes in background label. Similarly, according to the defined ’s in Eq. (4), the gradient of w.r.t is:
Consequently, the gradient of the above log-likelihood function with respect to will be added to the gradient of the other two loss functions in the backward pass while end-to-end training of parameters of the network. The proof for the above gradient derivations is provided in Appendix A.
Given the prediction scores and bounding box offsets from the learned network, we model the selection problem for unseen images as a maximum a posteriori (MAP) inference scheme where the probability of inclusion of each candidate box depends on the determinant of a kernel matrix. We define this kernel or similarity matrix such that it captures all spatial and contextual information between boxes all at once. We use the same kernel matrix as in Eq. (2.1.1) where is the list of all candidate proposals per image over all non-background categories with a score above an specific threshold (e.g. 0.05).
We capture quality of boxes, , by their per class prediction scores:
Similarly, we employ the spatial information by IoU and the semantic similarity between box labels by as shown in Eq. (2.1.1).
Thus, our kernel definition allows selection of a set of candidate boxes which have minimum possible overlapping as well as highest detection scores. In other words, the boxes with less location- and label-level similarity and higher detection scores would be more probable to be selected. To figure out which boxes should be selected, similar to , we use a greedy optimization algorithm, Alg. 2, which keeps the box with the highest probability found by Eq. (2.1.1) at each iteration.
@ IoU 0.5
@ IoU 0.7
Several works [2, 17, 13] have proposed a replacement for the conventional non-maximum suppression to optimally select among detection proposals. Desai et al.  proposed a unified model for multi-class object detection that both learns optimal NMS parameters and infers boxes by capturing different structured contextual interactions between object and their categories. In contrast to , the contextual information used in our model captures fixed label-level semantic similarities based on WordNet hierarchy as well as learned proposal probability scores. We also use IoU to capture spatial layout within a similarity kernel, while there is no strong notion of quality or overlap among boxes in 
and only a thresholded value is used in a (0/1) loss function. Unlike this method, the learnable deep features in our full end-to-end framework improve bounding box locations and category specifications through our proposed differentiable loss function.
Mrowca et al.  proposed a large-scale multi-class affinity propagation clustering (MAPC)  to improve both the localization and categorization of selected detected proposals, which simultaneously optimizes across all categories and all proposed locations in the image. Similar to our semantic similarity metric, they use WordNet relationships to capture highly related fine-grained categories in a large-scale detection setting. Lee et al.  use individualness to measure quality and similarity scores in a determinantal point process inference scheme focusing on the “binary” pedestrian detection problem. However, these methods are only applied for the inference paradigm and can neither improve proposal representations nor impose diversity among object bounding boxes while training the model.
@ IoU 0.5
We demonstrate the significant improvement obtained from our model on the representation of detected bounding boxes via a set of quantitative and qualitative experiments on Pascal VOC2007 and MS COCO benchmark data sets as explained in the following sections. We use the caffe deep learning framework in all our experiments. We use one image per iteration with mini-batch of 128 proposals. We replaced the semantic similarity matrix in Eq. (2.1.1) with its fourth power during LDDP inference as we observed improvement on detection performance on validation sets.
Our baseline in all experiments is the state-of-the-art object detection Faster R-CNN approach used as our training model. Moreover, final detections are inferred by the NMS scheme applied on top of the deep trained network, denoted as our inference baseline. We use two different NMS IoU threshold values for within class and across class suppressions: First, proposals labeled similarly are suppressed by an IoU threshold resulting in less overlapping within-class detections. Afterwards, we apply a second IoU threshold to suppress boxes across different categories. We do a grid-search on these two thresholds to find the best combination.
We evaluate the performance of our proposed LDPP model on Pascal VOC2007 data set with 5k trainval images and 5k test images categorized with 20 different labels. We use the ImageNet pre-trained fast Zeiler and Fergus (ZF) model with 5 convolutional layers and 3 fully-connected layers and add our DPP loss layer to the other two existing loss functions in the multi-task loss. We evaluate our results based on the mean Average Precision (mAP) metric for object detection for both training and inference schemes. Results in Table 1 show the significant improvement made by our proposed LDDP model over the state-of-the-art Faster R-CNN approach on per-category average precision scores and their corresponding mean average precision (mAP), both in training and inference steps.
|Method||Avg Precision @ IoU:||Avg Precision @ Area:||Avg Recall, #Dets:||Avg Recall @ Area:|
|(Faster RCNN, NMS)||15.0||31.5||12.7||3.6||15.1||23.8||16.4||23.2||23.6||6.0||24.2||38.9|
|(Faster RCNN, LDPP)||15.3||32.3||12.9||3.7||15.5||24.5||17.3||24.9||25.4||6.7||26.5||42.5|
In addition, to compare diversity among proposals generated by the two training models, Faster R-CNN and our proposed LDDP, we add up the intersection-over-union values for proposals among the categories with prediction scores above and compare the results in Figure 2. The total number of proposals generated by the two approaches are similar in this experiment. As expected, our LDDP model generates less overlapping proposals in most of the categories. For instance, the number of overlapping boxes identified as “horse” and “dog” is much larger in the Faster R-CNN approach than our LDDP model. This behavior is observed for other pairs of categories such as (dog, cat), (dog, cow), (person, sofa), (person, horse), (sofa, chair), etc revealing detection of diverse proposals through our proposed DPP loss layer.
We repeated similar experimental setup on Pascal VOC2012 data set containing 11.5K trainval images  and use the Imagenet pretrained ZF network  to train the end-to-end model and tested the results on Pascal VOC2007  data set. The results comparing our LDPP model used either in training, inference, or both (end-to-end LDDP) with the state-of-the-art Faster R-CNN detection network , in Table 2, show the out-performance of our proposed model.
Next, we demonstrate results on Microsoft COCO data set containing 80 different object categories. We train our model on the 80K training set and evaluate on the 40K validation set and use the VGG_CNN_M_1024 deep network architecture pre-trained on ImageNet data set [1, 6]. In addition to the Average Precision metric (AP) at multiple IoU threshold values and size of object proposals, we evaluate our proposed model based on Average Recall (AR) when allowing a different range for maximum number of detections or size of proposals. In all cases represented in Table 3, our proposed LDDP model outperforms the state-of-the-art Faster R-CNN network on COCO detection.
As an ablation study, we used different powers of the semantic similarity matrix during inference and evaluated the detection performance on the minival subset of MS COCO data set with 5K images. As shown in Table 4 in Appendix B.2, omitting semantic similarity from the kernel matrix drops the performance significantly. We use for inferring object proposals in all our experiments.
To show the effectiveness of LDDP in capturing informative proposals, we restrict the number of proposals generated by LDDP to a fraction of those generated by Faster R-CNN on VOC2007 data set. Limiting the number of proposals generated by our LDDP model to 100 drops mean AP on VOC2007 test set from to which is similar to the mean AP achieved by 300 proposals in Faster R-CNN. One can refer to Appendix B.1, Figure 5 for a complete plot representing performance versus number of proposals. This experiment approves high-confidence non-redundant localization through our end-to-end LDDP network.
We visualize the output of both our learning and inference LDDP models in Figures 3, 4 in comparison with the state-of-the-art detection model, Faster R-CNN, and NMS for pruning the proposals. Specifically, we illustrate the predictions of our proposed LDDP model as well as the baseline for the learning step on both Pascal VOC2007 and MS COCO data sets for prediction scores above 0.6 in Figure 3. We use NMS for the inference step in both predictions. Similarly, we compare the pruned detection boxes by the LDDP and NMS algorithms in Figure 4 using the Faster R-CNN training model. Bounding boxes found by our LDDP network are both more diverse and representative of the image content as illustrated.
We presented a deep network architecture for object detection which considers all contextual information, spatial layout, and label-level relationships between proposals during training and inference procedures and reinforces the parameters of the model to move toward a more accurate localization of bounding boxes and a better representation. The experimental results on PASCAL VOC and MS COCO benchmark data sets demonstrate the superiority of the LDDP detection performance in terms of average precision and recall. It is worth noting that our model does not add any new parameters to the deep network while boosting the performance and also results in the same detection performance as Faster R-CNN only by generating 1/3 proposals.
Learning semantic weights  instead of using a fixed semantic similarity matrix based on WordNet hierarchy can be investigated as a future study. As another possible future work, applying our DPP loss layer in the region proposal network besides the multi-task loss in the detection network can also reinforce more efficiency among generated proposals and boost the detection performance.
The authors would like to thank Marcus Rohrbach and Fisher Yu for helpful discussions. This work was supported in part by DARPA; NSF awards IIS-1212798, IIS-1427425, and IIS-1536003, Berkeley DeepDrive, and the Berkeley Artificial Intelligence Research Center.
Given the set of representative proposals, , and set of probable background proposals, , we can compute the log-likelihood loss function in Eq. (7) as well as its gradient with respect to the confidence scores.
where refers to the parameters of the deep network, and conditional probabilities are defined based on the DPP model  and our set of parameters:
where denotes the L-ensemble matrix, the similarity matrix, the quality measure, and denotes the restriction of to the entries indexed by elements of . Expanding the above probability distribution:
where is the normalizing factor as with as all possible sets of proposals selections.
Now, we take the gradient of each term of the above loss function with respect to the outputs of the inner product layer before softmax. As explained in the paper, for the first term, , we have:
), the first conditional probability distribution would be as:
where . The proposals indexed by and labeled as background are not involved in this log probability resulting in a zero gradient. The same result will be applied on the proposals indexed by and labeled by category . On the other hand:
|Similarity Matrix||Avg Precision @ IoU:||Avg Precision @ Area:||Avg Recall, #Dets:||Avg Recall @ Area:|
Similarly for the proposals indexed by and labeled as :
For the second log probability, , we change the quality measures as discussed in the paper. We skip the derivation of gradient of with respect to each , which can be achieved by following a similar scheme:
As explained in the paper, to approve the generation of high-confidence non-redundant proposals through our proposed LDDP network, we evaluate bounding box detection performance when we restrict the number of generated proposals to different values, as shown in Figure 5. Limiting the number of proposals generated by our LDDP model to 100 drops mean AP on VOC2007 test set from to which is similar to the mean AP achieved by 300 proposals in Faster R-CNN network (. Thus, our LDDP model is much more efficient than the state-of-the-art Faster R-CNN approach for the task of object detection.
To understand how the semantic similarity matrix used in the kernel matrix affects the performance of our LDDP model, we use its different powers during inference and evaluate the detection performance on the minival subset of MS COCO data set with 5K images. According to the results reported in Table 4, the semantic similarity matrix plays a crucial role in achieving accurate boxes.
We visualize the output of our end-to-end LDDP model as well as Faster R-CNN followed by NMS both on Pascal VOC2007 and MS COCO data sets  in Figures 6 and 7, respectively. We use the ZF model architecture for training the models on Pascal VOC2007 data set and the VGG_CNN_M_1024 deep network for training on MS COCO. The non-repetitive and accurate detections by the LDDP model reveal the superiority of our model against Faster R-CNN.
International journal of computer vision, 95(1):1–12, 2011.
Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
Imagenet classification with deep convolutional neural networks.In Advances in neural information processing systems, pages 1097–1105, 2012.