DeepAI
Log In Sign Up

DOOBNet: Deep Object Occlusion Boundary Detection from an Image

Object occlusion boundary detection is a fundamental and crucial research problem in computer vision. This is challenging to solve as encountering the extreme boundary/non-boundary class imbalance during training an object occlusion boundary detector. In this paper, we propose to address this class imbalance by up-weighting the loss contribution of false negative and false positive examples with our novel Attention Loss function. We also propose a unified end-to-end multi-task deep object occlusion boundary detection network (DOOBNet) by sharing convolutional features to simultaneously predict object boundary and occlusion orientation. DOOBNet adopts an encoder-decoder structure with skip connection in order to automatically learn multi-scale and multi-level features. We significantly surpass the state-of-the-art on the PIOD dataset (ODS F-score of .668) and the BSDS ownership dataset (ODS F-score of .555), as well as improving the detecting speed to as 0.037s per image.

READ FULL TEXT VIEW PDF

page 1

page 2

page 4

page 5

page 10

11/26/2019

DDNet: Dual-path Decoder Network for Occlusion Relationship Reasoning

Occlusion relationship reasoning based on convolution neural networks co...
08/12/2021

MT-ORL: Multi-Task Occlusion Relationship Learning

Retrieving occlusion relation among objects in a single image is challen...
03/17/2022

One-Stage Deep Edge Detection Based on Dense-Scale Feature Fusion and Pixel-Level Imbalance Learning

Edge detection, a basic task in the field of computer vision, is an impo...
06/04/2021

NMS-Loss: Learning with Non-Maximum Suppression for Crowded Pedestrian Detection

Non-Maximum Suppression (NMS) is essential for object detection and affe...
10/12/2018

DeepScores and Deep Watershed Detection: current state and open issues

This paper gives an overview of our current Optical Music Recognition (O...
03/16/2022

Zero Pixel Directional Boundary by Vector Transform

Boundaries are among the primary visual cues used by human and computer ...
05/26/2016

Dense Volume-to-Volume Vascular Boundary Detection

In this work, we present a novel 3D-Convolutional Neural Network (CNN) a...

Code Repositories

1 Introduction

A 2D image captures the projection of objects from a 3D scene, such that object occlusion appears as the depth discontinuities along the boundaries between different objects (or object and background). Figure 1 shows an example from the Pascal instance occlusion dataset (PIOD) [1], where one sheep is partially occluded by another one and each occludes part of the lawn background. Occlusion reasoning is both fundamental and crucial to a variety of computer vision research areas, including object detection [2], segmentation [3, 2], scene parsing [4] and 3D reconstruction [5]. Hoiem et al. [6]

argue that it lies at the core of scene understanding and must be addressed explicitly. In computer vision, the study of occlusion reasoning has been largely confined to the context of stereo, motion and other multi-view imaging problems

[7, 8, 9]. However, in single-view tasks, occlusion reasoning becomes challenging due to the unavailability of multiple images. In this paper, we focus on detecting occlusion boundary and boundary ownership from a single image.

The problem of object occlusion boundary detection relies on having precise object boundary. We argue that it is key to improve the performance of object boundary detector. Recent CNN-based boundary detection methods [10, 11, 12, 13, 14, 15, 16] have demonstrated promising F-score performance improvements on the BSDS500 dataset [17]. However, it is still unsatisfactory for them to handle higher-level object boundary, leaving a large room for improvement. A primary obstacle is extreme boundary/non-boundary pixels imbalance. Despite this can be partially resolved by class-balanced cross entropy loss function [10] and focal loss [18]

, the easily classified true positive and true negative still constitute the majority of loss and dominate the gradient during training of object boundary detector. Recently, FCN

[19], SegNet [20], U-Net [21] have been very successful for segmentation and related dense prediction visual tasks such as edge detection, which use an encoder-decoder structure to preserve precise localization and learn multi-scale and multi-level features. Meanwhile, dilated convolution [22, 23] has been used to systematically aggregate multi-scale contextual information without losing resolution.

(a) (b) (c) (d)
Figure 1: A ground truth from the PIOD dataset. Given an image (a), PIOD provides two annotated maps, namely (b) object boundary map and (c) occlusion orientation variable map. (d) Object occlusion boundary is represented by red arrows, each indicating an orientation . By the ”left” rule, the left side of each arrow indicates the foreground.

Motivated by these, we propose a novel loss function called Attention Loss tackling class imbalance. It is a dynamically scaled class-balanced cross entropy loss function, up-weighting the loss contribution of false negative and false positive examples. We also propose an encoder-decoder structure with skip connection and dilated convolution module, designing a unified end-to-end multi-task deep object occlusion boundary detection network (DOOBNet) by sharing convolutional features to simultaneously predict object boundary and occlusion orientation. Our model achieves a new state-of-the-art performance on both the PIOD and BSDS ownership dataset.

2 Related Work

Estimating the occlusion relationships from a single image is challenging. Early computer vision succeeded in simple domains, such as blocks world [24] and line drawings [25]. The 2.1D sketch [26] was a mid-level representation of images involving occlusion relations. Ren et al. [27] proposed a method for labeling occlusion boundaries in natural images on the BSDS border ownership dataset. They took a two-stage approach of image segmentation, followed by figure/ground labeling of each boundary fragment according to local image evidence and a learned MRF model. [28] addressed the border ownership problem based on the 2.1D model [26]. Teo et al. [29] embedded several different local cues (e.g. HoG, extremal edges) and semi-global grouping cues (e.g

. Gestalt-like) within a Structured Random Forest (SRF)

[30] to detect both boundaries and border ownership in a single-step. Maire et al. [31, 32] also designed and embedded the border ownership representation into the segmentation depth ordering inference. Hoiem et al. [6] introduced an approach to recover the occlusion boundaries and depth ordering of free-standing structures in a scene using the traditional edge and region cues together with 3D surface and depth cues. However, these methods were segmentation dependent, and that their performance dropped significantly without perfect segmentation. Recently, DOC [1] proposed a deep convolutional network architecture to detect object boundaries and estimate the occlusion relationships, which adapted a two streams network to perform two tasks separately. To train and test the network, it introduced the PASCAL instance occlusion dataset (PIOD), comprising a large-scale (10k images) instance of occlusion boundary dataset constructed by PASCAL VOC images.

Similar to DOC [1], our method detects object boundaries and estimates the occlusion relationships from a single image, which is referred as the object occlusion boundary detection. Notably, we adapt a single stream network architecture simultaneously predicting both object boundary and occlusion orientation in a single step by sharing convolutional features.

3 Problem Formulation

We use the representation of object occlusion boundary as in [1]. Occlusion relations represented by a per-pixel representation with two variables: (I) a binary edge variable to flag an object boundary pixel, and (II) a continuous-valued occlusion orientation variable (at each edge pixel) to indicate the occlusion relationship using the ”left” rule based on the tangent direction of the edge. As shown in Figure 1, we visualize the object occlusion boundaries with these two variable by red arrows, where the left side of each arrow indicates the foreground.

Given an input image, our goal is to compute the object boundary map and the corresponding occlusion orientation map. Formally, for an input image , we obtain a pair of object boundary map and occlusion orientation map , each having the same size as . Here, and . When at pixel , specifies the tangent of the boundary, where its direction indicates occlusion relationship using the ”left” rule. We do not use the when . In addition, we denote the ground truth by a label pair .

3.1 Class-balanced Cross Entropy and Focal Loss

As described in §1, CNN-based methods encounter the extreme boundary/non-boundary pixels imbalance during training an object boundary detector. As a priori knowledge, a typical natural image usually comprises not more than 0.97% boundary pixels111The statistics come from PIOD dataset.. CSCNN [33] proposed a cost-sensitive loss function with additional trade-off parameters introduced for biased sampling. HED [10] introduced a class-balancing weight on a per-pixel term basis to automatically balance the loss between positive/negative classes. It is formulated as:

(1)

where specifies the ground truth (non-boundary/boundary pixel) and

is the model’s estimated probability for the boundary pixel,

and . In addition, and denote the non-boundary and boundary ground truth label sets in a batch of images, respectively.

Recently, Lin et al. [18] introduced Focal Loss (FL) based on CE to object detection. FL is defined as , setting to reduce the relative loss for well-classified examples () and putting more focus on hard, misclassified examples.

The CCE loss and FL can be seen as the blue and brown curve in Figure 2, respectively, and the weight relates to the number of boundary and non-boundary pixels. For the edge detection, CCE and FL can easily classify edge pixels. However, these loss curves change slowly and the penalization has a small difference for , it is hard to discriminate both false negative and true positive examples for object boundary detection, where most of the edge pixels do not belong to object boundary pixels and are false negative in object boundary detection. see Figure 3.

3.2 Attention Loss for Object Boundaries

To address the problem, we propose a discriminating loss function motivated by FL, called the Attention Loss (AL), focusing the attention on class-agnostic object boundaries. Note that the true positive and true negative examples belong to well-classified examples after the loss are balanced by weight and FL can continue to partially solve the class-imbalance problem. However, the number of false negative and false positive examples is small and that their loss are still overwhelmed during training. Meanwhile, training is insufficient and inefficient, leading to degeneration of the model. The attention loss function explicitly up-weights the loss contribution of false negative and false positive examples so that it is more discriminating.

Formally, we propose to add two modulating factors and to the class-balanced cross entropy loss, with tunable parameters and . We define the attention loss as:

(2)

The attention loss is visualized for several values of and in Figure 2. The parameter adjusts true positive (true negative) and false negative (false positive) loss contributions. The attention loss strongly penalizes misclassified examples and only weakly penalizes the correctly classified ones, being more discriminating. Notably, the parameter gives a free degree to smoothly adjust the loss contribution at certain value. For instance, with , by reducing from 0.7 to 0.2, we can gradually enlarge the loss contribution. When , AL is equivalent to CCE. As our experiment results will show, we found setting and works the best in our experiments.

AL shares many similarities with FL, but AL makes the network to accept more misclassified signals () and to sufficiently back propagate, while FL reduces the well-classified signals and predicts a large number of false negatives (edge pixels, but not boundary pixels), see Figure 3. It is effective in terms of mAP metric () for object detection but not for object boundary detection ().

Figure 2: Attention loss distribution curves by varying and .
(a) image (b) GT (c) CCE (d) FL (e) AL
Figure 3: The predicted boundary maps with different loss functions (). CCE and FL predict a large number of false negatives (edge pixels, but not boundary pixels). In contrast to CCE and FL, AL focuses the attention on class-agnostic object boundaries.

3.3 Loss Function for Object Occlusion Boundary Detection

To perform occlusion orientation estimation, we adapt the loss function as defined in [34], which has demonstrated its simplicity yet effectiveness for regression task. Subsequently, our multi-task loss function for a batch of images is defined as:

(3)

where is mini-batch size, is the index of an image in a mini-batch, is the index of a pixel in an image, and

(4)
(5)

where adjusts the loss contribution curve and we explicitly penalize the predicted occlusion orientation values as we define the .

4 Network Architecture

Figure 4: DOOBNet Architecture.

DOOBNet Backbone

Encoder-decoder network structure has been successfully applied for semantic segmentation in [20, 21]. Inspired by this, we adopt an encoder-decoder structure as the backbone network in our DOOBNet, as illustrated in Figure 4. Typically, an encoder-decoder network comprises an encoder module and a decoder module. We use Res50 [35] (before pool5 layer) as the encoder module and design a simple yet effective decoder module to gradually recover spatial information for obtaining sharper object occlusion boundaries. In contrast to typical image classification, object occlusion boundary detection requires relatively large spatial resolution and receptive field to obtain precise object boundaries localization. For this reason, we use dilated convolutions () [22]

to increase the receptive field and remove sub-sampling (stride from 2 to 1) at

res5 stage to increase spatial resolution, where feature maps are 16 times smaller than the input image resolution now. Because using dilated convolutions can cause gridding artifacts [36], we add two conv block222The conv

block refers to convolution layer followed by batch normalization (BN)

[37]

and ReLU activation.

at the end of res5 to remove gradding artifacts. Besides, we reduce the number of channels to 256 for optimizing computation. Existing decoders [20, 21] operate based on the features from encoder, which are gradually bilinearly upsampled by a factor of 2 up to full input resolution. However, this significantly increases network parameters and computation cost. We hence propose a simple yet effective decoder module. The encoder features are first bilinearly upsampled by a factor 4 and then concatenated with the corresponding mid-level features from the end of res2, which have the same spatial resolution (see Figure 4). In this way, we explicitly learn multi-level (mid-level and high-level) features very effectively as our experiments will show. After the concatenation, we apply two residual conv blocks to refine the features followed by another simple bilinearly upsampling by a factor of 4.

Object Boundary Detection and Occlusion Orientation Estimation Subnet

In contrast to DOC [1], which relies on two separate stream networks, our DOOBNet adapts a single stream network by sharing backbone features with two subnets: one for object boundary detection and the other for occlusion orientation estimation. For the object boundary detection subnet (see Figure 4 top), we first apply three conv blocks to obtain low-level features, which have the same spatial resolution as the input image, and then concatenate them with the features from decoder. After concatenation, we add six extra conv blocks to learn specific task features. The final output feature is fed to a sigmoid classifier for pixel-wise classification. In parallel with the object boundary detection subnet, we attach the same subnet but exclude the sigmoid layer for occlusion orientation estimation (see Figure 4 bottom). Notably, our object boundary detection subnet does not share low-features with the occlusion orientation estimation subnet. We particularly design these low-features to improve generalization on low-level perceptual edges. Table 1 depicts our proposed network architecture in details.

layer name conv1 conv1_b/conv1_o res2_x res3_x res4_x
setup
res5_x conv6_x deconv7/deconv9 res8_1 res8_2 conv10_b / conv10_o
Table 1: DOOBNet Architecture. Building blocks are in brackets, with the numbers of blocks stacked. Down-sampling is performed by res3_1, res4_1 with a stride of 2. We apply a dilated rate of 2 to res5_x. Finally, _b/_o refers to object boundary detection and occlusion orientation estimation subnets, respectively.

Training Phase

For each training image , the corresponding ground truth comprises a binary object boundary map and an occlusion orientation map , as described above. We compute loss for object boundary detection subnet for every pixel but only compute occlusion orientation loss if .

Testing Phase

Given an input image , we obtain an object boundary map and an occlusion orientation map by simply forwarding the image through the network. To obtain the final object occlusion boundary map, we first perform non-maximum suppression (NMS) [30] on the object boundary map and then obtain the occlusion orientation for each boundary pixel from the orientation map (see Figure 4 right). Finally, similar to DOC [1], we adjust the orientation estimation to the direction of the tangent line estimated from the boundary map as we trust the accuracy of the estimated boundaries.

5 Experiments

5.1 Implementation

We implement DOOBNet by Caffe [38]. Our experiments initialize the encoder module with the pre-trained Res50 [35]

model on ImageNet and the other convolutional layers with the ”msra”

[39] initialization. All experiments are run on a single NVIDIA TITAN XP GPU.

Evaluation Criteria

For object occlusion boundary detection, we use three standard measures: fixed threshold for all images in the dataset (ODS), per-image best threshold (OIS), and average precision (AP). In contrast to DOC [1], which evaluated occlusion relations by measuring occlusion accuracy w.r.t. boundary recall (AOR). Occlusion Accuracy (A) is defined as the ratio of the total number of correct occlusion orientation pixels on correctly labelled boundaries to the total number of correctly labelled boundary pixels, while boundary Recall (R) is defined as the fraction of ground truth boundaries detected. However, it only evaluates the occlusion relations but not for occlusion boundary. In this paper, we instead measure occlusion precision w.r.t. boundary recall (OPR) to evaluate object occlusion boundaries as the occlusion relationships estimation relies on object boundaries, where occlusion precision is only computed at the correctly detected boundary pixels. Note that a standard non-maximal suppression (NMS) [30]

with default parameters is applied to obtain thinned boundaries and 99 thresholds are used to compute precision and recall for evaluation. We also refer readers to the original paper for the details about AOR curve.

Data augmentation

Data augmentation has proven to be a crucial technique in deep networks. We augment the PIOD data by horizontally flipping each image (two times), and additionally augment the BSDS ownership data by rotating each image to {0, 90, 180, 270} different angles (eight times). To save training time and improve generalization, we randomly crop the image to in every mini-batch during the training runtime. During testing, we operate on an input image at its original size.

Hyper-parameters

We use a validation set from the PIOD dataset and the CCE [10] loss function to tune the deep model hyper-parameters, including mini-batch size (5), iter size (3), learning rate (3e-5), momentum (0.9), weight decay (0.0002), sigma (3) in the loss function, lambda (0.5) in Equation 3. The number of training iterations for PIOD dataset (30,000; divide learning rate by 10 after 20,000 iterations) and BSDS ownership dataset (5,000; divide learning rate by 10 after every 2,000 iterations). In the following experiments, we set the values of these hyper-parameters as discussed above to explore DOOBNet variants.

Attention Loss

The attention loss introduces two new hyper-parameters, and , controlling the loss contribution. To demonstrate the effectiveness of the proposed attention loss for object boundary detection as described in §3.2, we adapt the grid search method to find an optimal parameter combination. To save training time, we change the iter size to 1 and the learning rate to 1e-5. Results for various and are shown in Table 2. When , our loss is equivalent to the CCE loss. AL shows large gains over CCE as is increased and slight gains by varying . With and , AL yields 8.3% ODS, 7.7% OIS and 11.6% AP improvement over the CCE loss. One notable property of AL, which can be easily seen in Figure 2 (right), is that we can adjust and to make two curves have similar loss contribution (). For example, and yield similar results as in Table 2. We use and for all the following experiments.

To understand the attention loss better, we empirically analyse that the CCE loss accepts the pixels as edge pixels when . However, the object boundary usually achieves a higher predicted probability, such as . We explicitly up-weight the loss contribution of false negative and false positive samples so that the model can focus on object boundaries. As the experiments shown, our design choices for AL is reasonable.

1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5
- .2 .3 .5 .7 .2 .3 .5 .7 .2 .3 .5 .7 .2 .3 .5 .7
ODS .633 .650 .679 .680 .687 .692 .698 .700 .706 .704 .710 .716 .712 .702 .713 .710 .714
OIS .649 .666 .691 .692 .698 .704 .708 .716 .714 .715 .719 .726 .721 .714 .721 .721 .726
AP .593 .511 .652 .644 .679 .680 .686 .696 .695 .699 .694 .709 .702 .695 .700 .611 .713
Table 2: Varying and for Attention Loss about object boundary detection on the PIOD.

5.2 PIOD Dataset

We evaluate DOOBNet on the PIOD dataset [1] which is composed of 9175 training images and 925 testing images. Each image has an object instance boundary map and the corresponding occlusion orientation map. We compare our method with the structured random forests algorithm SRF-OCC [29]

, and the state-of-the-art deep learning methods DOC-HED and DOC-DMLFOV

[1]. Results are shown in Table 3a and Figure 5a. Notably, DOOBNet performs the best, achieving ODS=0.702. It is 10.1%, 24.2% and 43.4% higher than DOC-DMLFOV, DOC-HED and SRF-OCC, respectively, with occlusion boundary precision being higher at every level of recall. On the other hand, we report the results of object boundary detection subnet in Table 4a and Figure 6a. DOOBNet obtains 0.736 ODS, 0.746 OIS and 0.723 AP, which are 6.7%, 6.2% and 4.6% higher than DOC-DMLFOV. We also visualize some of our results in Figure 7. It demonstrates DOOBNet has learned higher level features and can focus attention on class-agnostic object boundaries and estimate the corresponding occlusion orientations. For example, despite both the bird and the twig have similar color and texture, DOOBNet can correctly detect the bird boundaries and occlusion relationships.

(a) PIOD dataset (b) BSDS ownership dataset
Method ODS OIS AP FPS
SRF-OCC-BSDS .268 .286 .152 1/55.5
DOC-HED .460 .479 .405 18.5
DOC-DMLFOV .601 .611 .585 19.2
DOOBNet (w/o AL) .624 .636 .611 27
DOOBNet (w/o MLF) .607 .617 .568 29.2
DOOBNet (VGG16) .672 .683 .663 31.3
DOOBNet (FL) .652 .661 .631 27
DOOBNet .702 .712 .683 27
Method ODS OIS AP FPS
SRF-OCC .419 .448 .337 1/33.8
DOC-HED .522 .545 .428 20
DOC-DMLFOV .463 .491 .369 21.5
DOOBNet (w/o AL) .510 .528 .487 26.3
DOOBNet (w/o MLF) .443 .456 .324 29
DOOBNet (VGG16) .508 .523 .382 32.3
DOOBNet (FL) .536 .557 .510 26.3
DOOBNet .555 .570 .440 26.3
Table 3: Object occlusion boundary detection results on PIOD and BSDS ownership dataset. The term of MLF is multi-level features. SRF-OCC-BSDS trains on the BSDS ownership dataset and tests on the PIOD dataset. FL achieves the best performance using 6e-5 learnning rate. (Note: refers to GPU running time.)
(a) PIOD dataset (b) BSDS ownership dataset
Figure 5: Occlusion precision/recall (OPR) curves on PIOD and BSDS ownership dataset.

5.3 BSDS Ownership Dataset

We also evaluate our model on the BSDS ownership dataset [27] although its small size makes it challenging to train. It contains 100 training images and 100 testing images. Table 3b and Figure 5b show that DOOBNet can achieve the best performance but is lower than the one on PIOD. The main reason might be the case that there are only 100 training images, being insufficient for a complex deep network. We note that our DOOBNet is a more complex network (67-layer) and outperforms the complex model DOC-DMLFOV significantly by a margin of 9.2% ODS. DOOBNet is also 3.2% and 13.6% higher than DOC-HED and SRF-OCC, respectively. DOOBNet has a slightly low AP as setting in AL leads to lower recall (see Figure 5b). Reducing in AL from 4 to 2 can further improve ODS, OIS and AP to 0.565, 0.585 and 0.481, respectively. Object boundary detection results are shown in Table 4a and Figure 6b. DOOBNet is slightly lower than DOC-HED and still competitive with DOC-HED for recall below 0.65 (See Figure 6b). While it obtains a large margin over DOC-DMLFOV. Qualitative results are shown in Figure 8.

(a) PIOD dataset (b) BSDS ownership dataset
Method ODS OIS AP
SRF-OCC .345 .369 .207
DOC-HED .509 .532 .468
DOC-DMLFOV .669 .684 .677
DOOBNet (w/o AL) .655 .669 .646
DOOBNet (w/o MLF) .634 .643 .600
DOOBNet (VGG16) .705 .717 .706
DOOBNet (FL) .673 .683 .659
DOOBNet .736 .746 .723
Method ODS OIS AP
SRF-OCC .511 .544 .442
DOC-HED .658 .685 .602
DOC-DMLFOV .579 .609 .519
DOOBNet (w/o AL) .549 .598 .552
DOOBNet (w/o MLF) .526 .541 .422
DOOBNet (VGG16) .600 .617 .476
DOOBNet (FL) .596 .621 .583
DOOBNet .647 .668 .539
Table 4: Object boundary detection results on PIOD and BSDS ownership dataset. The term of MLF is multi-level features. SRF-OCC-BSDS trains on the BSDS ownership dataset and tests on the PIOD dataset. FL achieves the best performance using 6e-5 learnning rate.
(a) PIOD dataset (b) BSDS ownership dataset
Figure 6: Object boundary detection precision/recall curves on PIOD and BSDS ownership dataset.
Figure 7: Example results on PIOD dataset. Ground truth (Columns 1-3): visualization using ”left” rule with arrows, object boundaries, and object occlusion boundaries by a 2.1D relief sculpture. DOOBNet results (Columns 4-6). Note for column 4: ”red” pixels with arrows are correctly labelled occlusion boundaries; ”cyan” pixels are correctly labelled boundaries but with incorrect occlusion; ”green” pixels are false negative boundaries; ”orange” pixels are false positive boundaries. (Best viewed in color)
Figure 8: Example results on BSDS ownership dataset. (Best viewed in color)

5.4 Ablation Study

An ablation study is performed on both the PIOD and BSDS ownership datasets to confirm our design choices for DOOBNet. Four DOOBNet variants are tested: (I) DOOBNet (w/o attention loss), which adapts CCE loss from [10] instead of AL for the object boundary detection subnet, (II) DOOBNet (w/o multi-level features), which removes the low- and mid-features concatenation by skip connection, (III) DOOBNet (VGG16), which uses VGG16 [40] as the encoder module and the decoder module as the same as DOOBNet, (IV) DOOBNet (FL), which adapts FL () from [18] instead of AL for the object boundary detection subnet. We report the results in Table 3 and Figure 5. With AL, DOOBNet yields 7.8%/4.5% ODS, 7.6%/4.2% OIS, 7.2%/-4.7% AP improvements, while adding multi-level feature gives the gains of 9.5%/11.2% ODS, 9.5%/11.4% OIS, 11.5%/11.6% AP on the PIOD and BSDS ownership dataset, respectively. We observe that the improvement from AL on the PIOD dataset is higher than the one on the BSDS ownership dataset, especially to AP, but getting the opposite result for multi-level features learning. One of the main reasons is the PIOD dataset contains only object boundaries, while the BSDS ownership dataset includes many low-level edges. We also observe that DOOBNet(VGG16) is higher 7.1%/4.5% ODS, 7.2%/3.2% OIS, 7.8%/1.3% AP than DOC-DMLFOV on both datasets, while the later uses the same VGG16 as the encoder module. It demonstrates that the gains are come from our AL and decoder module design. DOOBNet with Res50 [35] improves performance by another 3.0%/4.7% ODS, 2.9%/4.7% OIS, 2.0%/5.8% AP over the DOOBNet(VGG16). Further more, compare to FL, DOOBNet is higher 5.0%/1.9% ODS, 5.1%/1.3% OIS, 5.2%/-7.0% AP than DOOBNet(FL) on both datasets. It demonstrates that AL is more discriminating than FL. We also report object boundary detection subnet results in Table 4 and Figure 6. All results of ablation study clearly show the effectiveness of our design choices for DOOBNet.

5.5 Additional AOR curves

To demonstrate the effectiveness of our proposed DOOBNet and ensure a fair comparison, we also report the AOR curve results in Figure 9. All our models outperform the methods under comparison. As described above, AOR curves only evaluate the accuracy of occlusion relations on correctly labelled boundaries so that a lower-performing object occlusion boundary detector may have a higher AOR curve. For example, the AOR curves of DOOBNet(FL) are higher than DOOBNet on both dataset. One of the reasons is that the lower-performing boundary detector have a smaller denominator when calculating the accuracy for two occlusion orientation estimators with the same performance.

(a) PIOD dataset (b) BSDS ownership dataset
Figure 9: Occlusion accuracy/recall (AOR) curves on PIOD and BSDS ownership dataset.

6 Conclusion

In this paper, we propose the Attention Loss to address the extreme positive/negative class imbalance, which we have suggested it as the primary obstacle in object occlusion boundary detection. We also design a unified end-to-end encoder-decoder structure multi-task object occlusion boundary detection network that simultaneously predicts object boundaries and estimates occlusion orientations. Our approach is simple and highly effective, surpassing the state-of-the-art methods with significant margins. In practice, Attention Loss is not specific to occlusion object boundary detection, and we plan to apply it to other tasks such as semantic edge detection and semantic segmentation in the future. Source code will be released.

7 Acknowledgement

This work is supported by National Key R&D Program of China (2017YFB1002702) and National Nature Science Foundation of China (61572058). We would like to thank Peng Wang for helping with generating DOC experimental results and valuable discussions.

References

  • [1] Wang, P., Yuille, A.: Doc: Deep occlusion estimation from a single image. In: European Conference on Computer Vision, Springer (2016) 545–561
  • [2] Gao, T., Packer, B., Koller, D.: A segmentation-aware object detection model with occlusion handling.

    In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE (2011) 1361–1368

  • [3] Zhang, Z., Schwing, A.G., Fidler, S., Urtasun, R.: Monocular object instance segmentation and depth ordering with cnns. arXiv preprint arXiv:1505.03159 (2015)
  • [4] Tighe, J., Niethammer, M., Lazebnik, S.: Scene parsing with object instances and occlusion ordering. In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE (2014) 3748–3755
  • [5] Shan, Q., Curless, B., Furukawa, Y., Hernandez, C., Seitz, S.M.: Occluding contours for multi-view stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2014) 4002–4009
  • [6] Hoiem, D., Stein, A.N., Efros, A.A., Hebert, M.: Recovering occlusion boundaries from a single image. In: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, IEEE (2007) 1–8
  • [7] He, X., Yuille, A.: Occlusion boundary detection using pseudo-depth. In: European Conference on Computer Vision, Springer (2010) 539–552
  • [8] Sundberg, P., Brox, T., Maire, M., Arbeláez, P., Malik, J.: Occlusion boundary detection and figure/ground assignment from optical flow. In: Computer Vision and Pattern Recognition, 2011 IEEE Conference on, IEEE (2011) 2233–2240
  • [9] Fu, H., Wang, C., Tao, D., Black, M.J.: Occlusion boundary detection via deep exploration of context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 241–250
  • [10] Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE international conference on computer vision. (2015) 1395–1403
  • [11] Kokkinos, I.: Pushing the boundaries of boundary detection using deep learning. arXiv preprint arXiv:1511.07386 (2015)
  • [12] Yang, J., Price, B., Cohen, S., Lee, H., Yang, M.H.: Object contour detection with a fully convolutional encoder-decoder network. (2016)
  • [13] Liu, Y., Lew, M.S.: Learning relaxed deep supervision for better edge detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 231–240
  • [14] Liu, Y., Cheng, M.M., Hu, X., Wang, K., Bai, X.: Richer convolutional features for edge detection. In: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE (2017) 5872–5881
  • [15] Hu, X., Liu, Y., Wang, K., Ren, B.: Learning hybrid convolutional features for edge detection. Neurocomputing (2018)
  • [16] Liu, Y., Cheng, M.M., Bian, J., Zhang, L., Jiang, P.T., Cao, Y.: Semantic edge detection with diverse deep supervision. arXiv preprint arXiv:1804.02864 (2018)
  • [17] Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE transactions on pattern analysis and machine intelligence 26(5) (2004) 530–549
  • [18] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv preprint arXiv:1708.02002 (2017)
  • [19] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 3431–3440
  • [20] Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39(12) (2017) 2481–2495
  • [21] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, Springer (2015) 234–241
  • [22] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)
  • [23] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40(4) (2018) 834–848
  • [24] Roberts, L.G.: Machine perception of three-dimensional solids. PhD thesis, Massachusetts Institute of Technology (1963)
  • [25] Cooper, M.C.: Interpreting line drawings of curved objects with tangential edges and surfaces. Image and Vision Computing 15(4) (1997) 263–276
  • [26] Nitzberg, M., Mumford, D.: The 2.1-d sketch. In: Computer Vision, 1990. Proceedings, Third International Conference on, IEEE (1990) 138–144
  • [27] Ren, X., Fowlkes, C.C., Malik, J.: Figure/ground assignment in natural images. In: European Conference on Computer Vision, Springer (2006) 614–627
  • [28] Leichter, I., Lindenbaum, M.: Boundary ownership by lifting to 2.1 d. In: Computer Vision, 2009 IEEE 12th International Conference on, IEEE (2009) 9–16
  • [29] Teo, C.L., Fermüller, C., Aloimonos, Y.: Fast 2d border ownership assignment. In: Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, IEEE (2015) 5117–5125
  • [30] Dollár, P., Zitnick, C.L.: Fast edge detection using structured forests. IEEE transactions on pattern analysis and machine intelligence 37(8) (2015) 1558–1570
  • [31] Maire, M.: Simultaneous segmentation and figure/ground organization using angular embedding. In: European Conference on Computer Vision, Springer (2010) 450–464
  • [32] Maire, M., Narihira, T., Yu, S.X.: Affinity cnn: Learning pixel-centric pairwise relations for figure/ground embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 174–182
  • [33] Hwang, J.J., Liu, T.L.: Pixel-wise deep learning for contour detection. arXiv preprint arXiv:1504.01989 (2015)
  • [34] Girshick, R.: Fast r-cnn. In: International Conference on Computer Vision. (2015)
  • [35] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 770–778
  • [36] Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In: Computer Vision and Pattern Recognition. Volume 1. (2017)
  • [37] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
  • [38] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  • [39] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. (2015) 1026–1034
  • [40] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)