Point-Set Anchors for Object Detection, Instance Segmentation and Pose Estimation

07/06/2020 ∙ by Fangyun Wei, et al. ∙ Microsoft Peking University 0

A recent approach for object detection and human pose estimation is to regress bounding boxes or human keypoints from a central point on the object or person. While this center-point regression is simple and efficient, we argue that the image features extracted at a central point contain limited information for predicting distant keypoints or bounding box boundaries, due to object deformation and scale/orientation variation. To facilitate inference, we propose to instead perform regression from a set of points placed at more advantageous positions. This point set is arranged to reflect a good initialization for the given task, such as modes in the training data for pose estimation, which lie closer to the ground truth than the central point and provide more informative features for regression. As the utility of a point set depends on how well its scale, aspect ratio and rotation matches the target, we adopt the anchor box technique of sampling these transformations to generate additional point-set candidates. We apply this proposed framework, called Point-Set Anchors, to object detection, instance segmentation, and human pose estimation. Our results show that this general-purpose approach can achieve performance competitive with state-of-the-art methods for each of these tasks.



There are no comments yet.


page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

A basic yet effective approach for object localization is to estimate keypoints. This has been performed for object detection by detecting points that can define a bounding box, e.g., corner points [20], and then grouping them together. An even simpler version of this approach that does not require grouping is to extract the center point of an object and regress the bounding box size from it. This method, called CenterNet [43], can be easily applied to human pose estimation as well, by regressing the offsets of keypoints instead.

While CenterNet is highly practical and potentially has broad application, its regression of keypoints from features at the center point can be considered an important drawback. Since keypoints might not lie in proximity of the center point, the features extracted at the center may provide little information for inferring their positions. This problem is exacerbated by the geometric variations an object can exhibit, including scale, orientation, and deformations, which make keypoint prediction even more challenging.

In this paper, we propose to address this issue by acquiring more informative features for keypoint regression. Rather than extract them at the center point, our approach is to obtain features at a set of points that are likely to lie closer to the regression targets. This point set is determined according to task. For instance segmentation, the points are placed along the edges of an implicit bounding box. For pose estimation, the arrangement of points follows modes in the pose distribution of the training data, such as that in Fig. 1 (b). As a good task-specific initialization, the point set can yield features that better facilitate keypoint localization.

It can be noted that a point set best serves its purpose when it is aligned in scale and aspect ratio with the target. To accomplish this, we adapt the anchor box scheme commonly used in object detection by expressing point sets as point-set anchors. Like their anchor box counterparts, point-set anchors are sampled at multiple scales, aspect ratios, and image positions. In addition, different point-set configurations may be enumerated, such as different modes in a pose estimation training set. With the generated point-set candidates, keypoint regression is conducted to find solutions for the given task.

The main contributions of this work can be summarized as:

  • A new object representation named Point-Set Anchors, which can be seen as a generalization and extension of classical box anchors. Point-set anchors can further provide informative features and better task-specific initializations for shape regression.

  • A network based on point-set anchors called PointSetNet, which is a modification of RetinaNet [23] that simply replaces the anchor boxes with the proposed point-set anchors and also attaches a parallel regression branch. Variants of this network are applied to object detection, human pose estimation, and also instance segmentation, for which the problem of defining specific regression targets is addressed.

It is shown that the proposed general-purpose approach achieves performance competitive with state-of-the-art methods on object detection, instance segmentation and pose estimation.

Figure 1: Illustration of our point-set anchors for instance segmentation, object detection and pose estimation. Instance mask point-set anchors contain an implicit bounding box and anchor points are uniformly sampled from the corresponding bounding box. Pose point-set anchors are initialized as the most frequent poses in the training set.

2 Related Work

Object representations. In object detection, rectangular anchors [34, 23, 22] are the most common representation used in locating objects. These anchors serve as initial bounding boxes, and an encoding is learned to refine the object localization or to provide intermediate proposals for top-down solutions [15, 8]. However, the anchor box is a coarse representation that is insufficient for finer degrees of localization required in tasks such as instance segmentation and pose estimation. An alternative is to represent objects in terms of specific points, including center points [38, 43], corner points [20, 12], extreme points [44], octagon points [32], point sets [41, 42], and radial points [40]. These point representations are designed to solve one or two tasks among object detection, instance segmentation and pose estimation. Polygon point based methods, such as corner points [20, 12] and extreme points [44], are hard to apply to instance segmentation and pose estimation due to their restricted shape. While center point representations [38, 43] are more flexible, as offsets from the object center to the corresponding bounding box corners or human joints can be directly predicted, we argue that features extracted from center locations are not as informative as from our task-specific point sets, illustrated in Fig. 1. In addition, how to define regression targets for instance segmentation is unclear for these representations. Another way to perform instance segmentation from a center point is by regressing mask boundary points in radial directions [40]; however, radial regressions at equal angular intervals are unsuitable for pose estimation. Other representations such as octagon points or point sets [41, 42] are specifically designed for one or two recognition tasks. The proposed point-set anchors combine benefits from anchor boxes and point representations, and its flexibility makes them applicable to object detection, instance segmentation and pose estimation.

Instance segmentation. Two-stage methods [15, 21, 17] formulate instance segmentation in a ‘Detect and Segment’ paradigm, which detects bounding boxes and then performs instance segmentation inside the boxes. Recently, there is much research focused on single-stage instance segmentation since two-stage methods are often slow in practice. PolarMask [40] uses a polar representation and rays at equal angular intervals are emitted from the polar center for dense distance regression. YOLACT [1] generates a set of prototype masks and predicts per-instance mask coefficients for linearly combining the prototype masks. ExtremeNet [44] detects four extreme points and a center point using a standard keypoint estimation network, which has the disadvantage of a long training time, and then grouping is applied to generate coarse octagonal masks. The recent Deep Snake [32] proposes a two-stage pipeline based on initial contours and contour deformation. Our method is different from Deep Snake in three ways. First, our method for instance segmentation operates in a single stage, without needing proposals generated by detectors. Second, Point-Set Anchors perform mask shape regression directly, in contrast to the iterative deformation in Deep Snake. Finally, our method is evaluated on the challenging MS COCO dataset [24] and is compared to state-of-the-art methods in object detection, instance segmentation and pose estimation.

Pose estimation. In pose estimation algorithms, most previous works follow the paradigm of estimating a heat map for each joint [2, 3, 5, 6, 7, 14, 15, 33, 18, 28, 39]. The heat map represents the confidence of a joint existing at each position in the image. Despite its good performance, a heat map representation has a few drawbacks such as no end-to-end training, a need for high resolution [37], and separate steps for joint localization and association. The joint association problem is typically addressed by an early stage of human bounding box detection [39, 15] or a post-processing step that groups the detected joints together with additionally learned joint relations [3, 27]. Recently, a different approach has been explored of directly regressing the offsets of joints from a center or root point [43]

. In contrast to the heat map based approaches, the joint association problem is naturally addressed by conducting a holistic regression of all joints from a single point. However, the holistic shape regression is generally more difficult than part based heat map learning due to optimization on a large, high-dimensional vector space. Nie et al. 

[29] address this problem by factorizing the long-range displacement with respect to the center position into accumulative shorter ones based on the articulated kinematics of human poses. They argue that modeling short-range displacements can alleviate the learning difficulty of mapping from an image representation to the vector domain. We follow the regression based paradigm and propose to address the long-range displacement problem by regressing from a set of points placed at more advantageous positions.

3 Our Method

In this section, we first formulate the proposed task-specific point-set anchors. Next, we show how to make use of these point-set anchors for regression-based instance segmentation and pose estimation. Finally, we introduce our PointSetNet, which is an extension of RetinaNet with a parallel branch attached for keypoint regression.

3.1 Point-Set Anchors

The point-set anchor contains a number of ordered points that are defined according to the specific task. We describe the definitions for the tasks of human pose estimation and instance segmentation that we will handle.

Pose point-set anchor. We naturally use the human keypoints to form the pose point-set anchor. For example, in the COCO [24] dataset, there are keypoints, and the pose point-set anchor is represented by a

-dimensional vector. At each image position, we use several point-set anchors. We initialize the point-set anchors as the most frequent poses in the training set. We use a standard k-means clustering algorithm 

[26] to partition all the training poses into clusters, and the mean pose of each cluster is used to form a point-set anchor. Fig. 1(b) illustrates one of the point-set anchors for pose estimation.

Instance mask point-set anchor. This anchor has two parts: one center point and ordered anchor points which are uniformly sampled from an implicit bounding box as illustrated in Fig. 1(a). is a hyper-parameter to control sampling density. Corner points in mask point-set anchor can be also served as a reference for object detection. At each image position, we form point-set anchors by varying the scale and aspect ratios of the implicit bounding box.

3.2 Shape Regression

We treat instance segmentation, object detection and pose estimation as a shape regression problem. We represent the object by a shape , i.e., a set of ordered points , where represents the -th polygon vertex for instance mask, -th corner vertex for bounding box, or the -th keypoint for pose estimation. Instead of regressing the shape point locations from the object center, we employ as a reference for shape regression. Our goal is to regress the offsets from the point-set anchor to the shape .

Offsets for pose estimation. Each keypoint in the human pose estimation task represents a joint with semantic meaning, e.g., head, right elbow or left knee. We use keypoints as the shape and the offsets are simply the difference: .

Offsets for instance segmentation. The shape of an instance mask also contains a set of ordered points, and the number of points might be different for different object instances. The point-set anchor is defined for all instances and contains a fixed number of points. To compute the offsets , we find the matching points , each of which has a one-to-one correspondence to each point in , from the shape , and then . The matching strategies, illustrated in Fig. 2, are described as follows.

Figure 2: Illustration of three matching strategies between point-set anchor and the ground-truth mask contour for instance segmentation. Yellow solid dots represent polygon points of the ground-truth mask. Green solid dots and green hollow dots denote valid and invalid anchor points, respectively. Orange and yellow lines indicate correspondences for corner and non-corner anchor points, respectively. Only valid anchor points are considered for training and inference.
  • Nearest point. The matching target of each point in the point-set anchor T is defined as the nearest polygon point in S based on distance. Thus, a single polygon point may be assigned to several anchor points, one anchor point, or none of them.

  • Nearest line. We treat the mask contour S as a sequence of line segments instead of discrete polygon vertices. Each of the anchor points is projected to all the polygon edges, and the closest projection point is assigned to the corresponding anchor point.

  • Corner point with projection. We first find the targets of the four corner anchor points by the Nearest Point strategy, and then the targets of these corner points are used to subdivide the mask contour into four parts, i.e., top, right, bottom and left parts, that are to be matched with anchor points on the corresponding side. For each of the four parts, the target of each anchor point is the nearest intersection point between the horizontal (for left and right parts) or vertical (for top and bottom parts) projection line and the line segments of the mask contour. If a matched point lies outside of the corresponding contour segment delimited by the matched corner points, we mark it as invalid and it is ignored in training and testing. The remaining anchor points and their matches are marked as valid for mask regression learning.

Offsets for object detection. The bounding box shape S in object detection can be denoted as two keypoints, i.e., the top-left and bottom-right point. The offsets are the distance from the target points to the corresponding corner points in mask point-set anchors.

Positive and negative samples. In assigning positive or negative labels to point-set anchors, we directly employ IoU for object detection and instance segmentation, and Object Keypoint Similarity (OKS) for pose estimation. Formally, in instance segmentation and object detection, a point-set anchor is assigned a positive label if has the highest IoU for a given ground-truth box or an IoU over 0.6 with any ground-truth box, and a negative label if it has IoU lower than 0.4 for all ground-truth boxes. In practice, we use the IoU between the implicit bounding box of the mask point-set anchors and ground-truth bounding boxes instead of masks in instance segmentation, to reduce computation cost. For human pose estimation, a point-set anchor is assigned a positive label if it has the highest OKS for a given ground-truth pose or an OKS over 0.5 with any ground-truth pose, and a negative label if it has OKS lower than 0.4 for all ground-truth poses.

Mask construction. In instance segmentation, a mask is constructed from the regressed anchor points as a post-processing step during inference. For the matching methods Nearest Point and Nearest Line, we choose an arbitrary point as the origin and connect adjacent points sequentially. For Corner Point with Projection, a mask is similarly constructed from only the valid points.

3.3 PointSetNet

Figure 3: Network architecture. The left part represents the backbone and feature pyramid network to extract features from different levels. The right part shows the shared heads for classification and mask/pose estimation with point-set anchors. We omit the bounding box regression branch for clearer illustration.

Architecture. PointSetNet is an intuitive and natural extension of RetinaNet [23]. Conceptually, it simply replaces the classical rectangular anchor with the proposed point-set anchor, and attaches a parallel regression branch for instance segmentation or pose estimation in the head. Fig. 3 illustrates its network structure. Following RetinaNet, we use a multi-scale feature pyramid for detecting objects at different scales. Specifically, we make use of five levels of feature maps, denoted as . , and are generated by backbone feature maps , and followed by a convolutional layer and lateral connections as in FPN [22]. and are generated by a stride-2 convolutional layer on and , respectively. As a result, the stride of is . The head contains several subnetworks for classification, mask or pose regression, and bounding box regression. Each subnetwork contains four stride-1 convolutional layers, a feature aggregation layer which is used only for the pose estimation task, and an output layer. Table 1 lists the output dimensions from the three subnetworks for instance segmentation and pose estimation. Following [22, 23], we also share the head among .

Task Classification Shape Regression Bounding Box Regression
Instance Segmentation
Pose Estimation -
Table 1: Output dimensions from different subnetworks for instance segmentation and pose estimation. , and denote the number of point-set anchors, points number in each of the point-set anchors, and class number for the training/testing dataset, respectively.

Point-set anchor density. One of the most important design factors in anchor based detection frameworks [34, 22, 23, 25] is how densely the space of possible anchors is sampled. For instance segmentation, following [23], we simply replace classical rectangular anchors by our mask point-set anchors, and use 3 scales and 3 aspect ratios per location on each of the feature maps. Specifically, we make use of the implicit bounding box in point-set anchors, where each bounding box has three octave scales and three aspect ratios , and the base scale for feature maps is . The combination of octave scales, aspect ratios and base scales will generate 9 bounding boxes per location on each of the feature maps. Anchor points are uniformly sampled on the four sides of the generated bounding boxes. For pose estimation, we use 3 mean poses generated by the k-means clustering algorithm. Then we translate them to each position of the feature maps as the point-set anchors. We further use 3 scales and 3 rotations for each anchor, yielding 27 anchors per location. The other feature map settings are the same as in instance segmentation.

Loss function.

We define our training loss function as follows:


where is the Focal loss in [23] and is the L1 loss for shape regression. and represent classification and regression targets, respectively. denotes the number of positive samples and is the balance weight, which is set to and for instance segmentation and pose estimation, respectively. denotes the indicator function, being 1 if and 0 otherwise. The loss is calculated over all locations and all feature maps.

Elements specific to pose estimation. Besides target normalization and the embedding of prior knowledge in the anchor shapes, we further show how feature aggregation with point-set anchors achieves a certain feature transformation invariance and how point-set anchors can be extended to multi-stage learning.

– Deep shape indexed features.

Learning of shape/transformation invariant features has been a fundamental problem in computer vision 

[36, 11], as they provide consistent and robust image information that is independent of geometric configuration. A point-set anchor acts as a shape hypothesis of the object to be localized. Though it reflects a coarse estimate of the target object shape, it still achieves a certain feature invariance to object shape, as it extracts the feature in accordance with the ordered point-set. The blue dashed rectangle in Fig. 3 depicts the feature aggregation module. In principle, the closer the anchor points are to the object shape, the better shape invariance of the feature. The deep shape indexed feature is implemented by DCN [9]. Specifically, we replace the learnable offset in DCN with the location of points in a point-set anchor.

– Multi-stage refinement. Holistic shape regression is generally more difficult than part based heat map learning [37]

. This is on one hand because of the large and continuous solution space of poses. On the other hand, it is due to the extremely unbalanced transformation variance between different keypoints. To address this, a classic paradigm is to estimate the pose progressively via a sequence of weak regressors where each weak regressor uses features that depend on the estimated pose from the previous stage 

[11, 36].

To this end, we use an additional refinement stage for pose estimation. While the mean poses in the training set are used as the initial anchors for the first stage, we can directly use the pose predictions of the first stage as the point-set anchors for the second stage. Since the joint positions in the point-set anchors are well-initialized in the first stage, the point-set anchors for the second stage are much closer to the ground truth shapes. This facilitates learning since the distance of the regression target becomes much smaller and better shape-invariant features can be extracted by using the more accurate anchor shapes.

Conceptually, this head network can be stacked for multi-stage refinement, but we find the use of a single refinement to be most effective. Hence, we use one-step refinement for simplicity and efficiency.

4 Experiments

4.1 Instance Segmentation Settings

Dataset. We present experimental results for instance segmentation and object detection on the MS-COCO [24] benchmark. We use COCO trainval35k (115k images) for training and the minival split (5k images) for ablations. Comparisons to the state-of-the-art are reported on the test-dev split (20k images).

Matching strategy
Nearest Point 21.9 42.1 20.6 11.4 24.6 29.8
Nearest Line 23.2 46.5 21.0 12.5 26.2 32.0
Corner Point with Projection 27.0 49.1 26.6 13.8 30.6 36.7
Table 2: Comparison of different matching strategies between anchor points and mask contours on the instance segmentation task.

Training details. All our ablation experiments, except when specifically noted, are conducted on the minival split with ResNet-50 [16]

and FPN. Our network is trained with synchronized stochastic gradient descent (SGD) over 4 GPUs with a mini-batch of 16 images (4 images per GPU). We adopt the

training setting, with 12 epochs in total and the learning rate initialized to 0.01 and then divided by 10 at epochs 8 and 11. The weight decay and momentum parameters are set to


, respectively. We initialize our backbone network with the weights pre-trained on ImageNet 

[10]. For the newly added layers, we keep the same initialization as in [23]. Unless specified, the input images are resized to have a shorter side of 800 and a longer side less than or equal to 1333. Samples with IoU higher than 0.6 and lower than 0.4 are defined as positive samples and negative samples, respectively. We use the Corner Point with Projection matching strategy and set the number of anchor points to 36 by default.

Inference details. We forward the input image through the network and obtain the classification scores and the corresponding predicted classes. According to the classification scores, the top-1k anchors from each level of the feature maps are sent for mask construction. Then the top predictions from all levels are merged and non-maximum suppression (NMS)111IoU is calculated by predicted bounding boxes and ground-truth rectangles due to the high computational cost of mask IoU. If there is no bounding box branch in the network, we use the smallest rectangle that encompasses the predicted mask instead. with a threshold of 0.5 is employed as post-processing.

4.2 Experiments on Instance Segmentation

24 26.7 48.8 26.1 13.6 30.2 36.4
36 27.0 49.1 26.6 13.8 30.6 36.7
48 27.2 49.2 26.8 13.9 30.7 36.7
60 28.0 49.8 27.9 13.9 31.4 38.6
72 28.0 49.6 27.9 14.6 31.5 38.8
Table 3: Comparison on different numbers of anchor points.

Mask matching strategies. First, we compare the results from the three matching strategies between point-set anchor and the corresponding mask contour as shown in Table 2. Nearest Point has the worst performance, perhaps because each polygon point may be assigned to multiple anchor points and this inconsistency may misguide training. Both Nearest Line and Corner Point with Projection treat the ground-truth mask as a whole contour instead of discrete polygon points in the mask matching step. However, there still exist ambiguous anchor points for the Nearest Line method as shown in Fig. 2(b). Corner Point with Projection eliminates inconsistency, as the subdivision of the mask contour into segments leads to better-defined matches, and it achieves the best performance among the three methods.

Center Point 26.0 48.4 24.8 13.6 29.3 35.4
Anchor Points 27.0 49.1 26.6 13.8 30.6 36.7
Table 4: Comparison of two regression origins.

Effect of point-set anchors. The number of anchor points can greatly affect instance segmentation. From Table 3, it can be seen that the more accurate representation from more anchor points leads to better performance. Also, the performance is seen to saturate beyond a certain number of anchor points, e.g., 72 points. We also demonstrate the benefits of using point-set anchors as the regression origin as shown in Table 4. It can be seen that with the same regression targets, using point-set anchors as the regression origin outperforms center point based regression by 1.0 AP.

Comparison with state-of-the-art methods. We evaluate PointSetNet on the COCO test-dev split and compare with other state-of-the-art object detection and instance segmentation methods. We use 60 anchor points for contour representation and ResNext-101 with DCN [9] as backbone. For data augmentation, we randomly scale the shorter side of images in the range of 480 to 960 during training and increase the number of training epochs to 24 ( training setting). An image pyramid with a shorter side of is applied during the inference. Table 5

reports the results. On both object detection and instance segmentation, PointSetNet achieves performance competitive with the state-of-the-art algorithms. The gap between TensorMask and PointSetNet arises from the tensor bipyramid head which brings a +5.1 AP improvement. We do not plug this into our framework due to its heavy time and memory cost. Although the AP is 0.2 lower than PolarMask on instance segmentation, our proposed point-set anchor has the benefit of being applicable to human pose estimation.

Method Backbone Regression Segmentation Detection
Mask RCNN [15] ResNeXt-101 37.1 60.0 39.4 39.8 62.3 43.4
TensorMask [4] ResNet-101 37.1 59.3 39.4 - - -
FCIS [21] ResNet-101 29.2 49.5 - - - -
YOLACT [1] ResNet-101 31.2 50.6 32.8 33.7 54.3 35.9
ExtremeNet[44] Hourglass-104 18.9 44.5 13.7 43.7 60.5 47.0
CornerNet [20] Hourglass-104 - - - 42.2 57.8 45.2
RetinaNet [23] ResNext-101 - - - 40.8 61.1 44.1
FCOS [38] ResNext-101 - - - 42.1 62.1 45.2
CenterNet[43] Hourglass-104 - - - 45.1 63.9 49.3
RepPoints[41] ResNeXt-101-DCN - - - 46.5 67.4 50.9
PolarMask [40] ResNeXt-101-DCN 36.2 59.4 37.7 - - -
Dense RepPoints [42] ResNeXt-101-DCN 33.7 59.9 34.3 45.8 66.7 49.6
Ours ResNeXt-101-DCN 34.6 60.1 34.9 45.1 66.1 48.9
Ours ResNeXt-101-DCN 36.0 61.5 36.6 46.2 67.0 50.5
Table 5: Results on the MS COCO test-dev compared to state-of-the-art instance segmentation and object detection methods. ‘’ denotes the multi-scale testing.

4.3 Pose Estimation Settings

Dataset. We conduct comprehensive comparison experiments on the MS COCO Keypoint dataset [24] to evaluate the effectiveness of point-set anchors on multi-person human pose estimation. The COCO train, validation, and test sets contain more than 200k images and 250k person instances labeled with keypoints. 150k of the instances are publicly available for training and validation. Our models are trained only on the COCO train2017 dataset (which includes 57K images and 150K person instances) with no extra data. Ablations are conducted on the val2017 set, and final results are reported on the test-dev2017 set for a fair comparison to published state-of-the-art results [3, 13, 15, 27, 30, 31, 43, 29].

Training details. For pose estimation, we use the Adam [19] optimizer for training. The base learning rate is 1e-4. It drops to 1e-5 at 80 epochs and 1e-6 at 90. There are 100 epochs in total. Samples with OKS higher than 0.5 and lower than 0.4 are defined as positive samples and negative samples, respectively. The other training details are the same as for instance segmentation including the backbone network, network initialization, batch size and image resolution.

4.4 Experiments on Pose Estimation

Anchor Type Matched GT(%) Pos/Neg()
Center Point 0 16.9 48.7 6.0
Rectangle 4.0 12.0 33.9 5.9
Mean Pose 18.0 40.9 69.4 42.0
K-means_3 27.7 43.3 70.7 45.5
K-means_5 32.1 43.8 71.3 46.6
K-means_7 34.6 43.8 71.2 46.5
Scale (0.8:0.2:1.2) 25.2 42.6 69.8 44.9
Scale (0.6:0.2:1.4) 27.6 42.6 69.7 44.5
Scale (0.4:0.2:1.6) 29.6 39.4 65.4 41.2
Rotation (-10:10:10) 27.2 42.6 70.6 44.2
Rotation (-20:10:20) 36.0 43.5 71.6 45.9
Rotation (-30:10:30) 40.5 42.6 70.6 44.4
Table 6: Comparing different point-set anchors. Better results are achieved when anchors efficiently cover more ground truth shapes with relatively large positive/negative sample ratio.

Effect of point-set anchors. We first compare the proposed point-set anchors with strong prior knowledge of pose shapes to other point-based anchors like the center point and points on a rectangle. We denote as mean pose the use of the average pose in the training set as the canonical shape. Then we translate this mean pose to every position in the image as the point-set anchors for pose estimation. No other transformation is used to augment the anchor distribution for a fair comparison to the center point and rectangle anchors. A ground truth pose is assigned to anchors with OKS higher than 0.5. If no anchor is found higher than OKS 0.5, the ground truth is assigned to the closest anchor.

In Table 6, it is shown that the mean pose anchor outperforms the center point and rectangle anchors by a large margin with more ground truth poses assigned with OKS greater than 0.5. Specifically, it surpasses center point anchors by 24 AP and rectangle anchors by 28.9 AP. This indicates that an anchor that better approximates the target shape is more effective for shape regression.

Feature Types Loss_cls Loss_reg
Center Feature 0.31 5.92 40.9 69.4 42.0
Box Corner Feature 0.31 6.22 42.6 68.2 45.6
Box Region Feature 0.30 5.99 42.8 69.0 46.0
Deep-SIF-9 for Cls 0.30 6.29 41.6 68.2 44.0
Deep-SIF-9 for Reg 0.32 5.93 45.5 69.6 49.1
Deep-SIF-9 for Cls Reg 0.29 5.92 46.0 71.7 49.1
 Deep-SIF-25 for Cls Reg 0.29 5.79 47.5 72.0 51.6
Table 7: Comparing the deep shape indexed feature with other feature extraction methods. Deep-SIF- denotes deep shape indexed feature with points used for feature extraction.

We obtain further improvements by using additional canonical pose shapes generated by the K-Means clustering algorithm or by augmenting the mean pose shape with additional sampling of rotation and scaling transformations. However, more anchor shapes and transformation augmentations also introduce more negative anchors that are not assigned to any of the ground truth poses, which makes learning less efficient. Better performance can be attained with a better trade-off between covering more ground truth shapes and incurring fewer negative samples. Empirically, we achieve the best performance by using 5 canonical pose shapes (+2.9AP), 5 scale transformations (+1.7AP) and 5 rotation transformations (+2.6AP).

Effect of deep shape indexed feature. Table 7 compares the deep shape indexed feature with other feature extraction methods. First, three regular feature extraction methods, namely from the center point, 4 corner points of the bounding box and 9 grid points in the bounding box region are compared. With more feature extraction points, slightly better performance is obtained.

Then, we use the deep shape indexed feature which uses the point set in the anchors for feature extraction. A point set based on pose shape priors extracts more informative features that greatly improve learning and performance. Specifically, with the same number of points (part of the 17 joints) as in the box region feature (i.e., 9), the deep shape indexed feature improves the AP from 42.8 to 46.0 (+3.2 AP, relative 7.5% improvement). Further improvement can be obtained by using more joint points for feature extraction. Note that if the shape indexed feature is used only for the person classification sub-network, the improvement is not as great as using it only in the pose regression sub-network. This indicates that it mainly enhances pose regression learning.

Stage OKS Matched GT(%) Pos/Neg() Loss_cls Loss_reg
Stage-1 0.5 45.7 12.7 0.25 5.52 48.3 74.3 52.5
Stage-2 0.99 81.5 11.5 0.23 4.28 58.0 80.8 62.4
Table 8: Effect of multi-stage refinement.
Backbone Multi-Test
ResNet-50 58.0 80.8 62.4
ResNeXt-101-DCN 62.5 83.1 68.3
ResNeXt-101-DCN 65.7 85.4 71.8
HRNet 69.8 88.8 76.3
Table 9: Effect of backbone network and multi-scale testing.

Effect of multi-stage refinement. Table 8 shows the result of using a second stage for refinement. Since the anchors for the second stage are much closer to the ground truth, we use a much higher OKS threshold (0.99) for positive sample selection. Even though the second stage uses a much higher OKS threshold, it is found that many more ground truth poses are covered. Both the person classification and the pose regression losses are decreased. We thus obtain significant improvement from the second stage, specifically, 9.7 AP (relative 20.1% improvement) over the first stage.

Effect of stronger backbone network and multi-scale testing. Table 9 shows the result of using stronger backbones and multi-scale testing. Specifically, we obtain 4.5 AP improvement from using the ResNeXt-101-DCN backbone network. A 3.2 AP improvement is found from using multi-scale testing. We obtain the further improvement by using HRNet [35] as backbone (+4.1 AP).

Comparison with state-of-the-art methods. Finally, we test our model (with HRNet backbone and multi-scale testing) on the MSCOCO test-dev2017 dataset and compare the result to other state-of-the-art methods in Table 10. PointSetNet outperforms CenterNet [43] by 5.7 AP and achieves results comparable to the state-of-the-art [30].

Method Backbone
Heat Map Based
CMU-Pose [3] 3CM-3PAF (102) 61.8 84.9 67.5 57.1 68.2
RMPE  [13] Hourglass-4 stacked 61.8 83.7 69.8 58.6 67.6
Mask-RCNN  [15] ResNet50 63.1 87.3 68.7 57.8 71.4
G-RMI  [31] ResNet-101+ResNet-50 64.9 85.5 71.3 62.3 70.0
AE [27] Hourglass-4 stacked 65.5 86.8 72.3 60.6 72.6
PersonLab [30] ResNet-152 68.7 89.0 75.4 64.1 75.5
Regression Based
CenterNet [43] Hourglass-2 stacked (104) 63.0 86.8 69.6 58.9 70.4
SPM [29] Hourglass-8 stacked 66.9 88.5 72.9 62.6 73.1
Ours HRNet 68.7 89.9 76.3 64.8 75.3
Table 10: Results on the MS COCO test-dev2017 compared to state-of-the-art pose estimation methods.

5 Conclusion

In this paper, we propose Point-Set Anchors which can be seen as a generalization and extension of classical anchors for high-level recognition tasks such as instance segmentation and pose estimation. Point-set anchors provide informative features and good task-specific initializations which are beneficial for keypoint regression. Moreover, we propose PointSetNet by simply replacing the anchor boxes with the proposed point-set anchors in RetinaNet and attaching a parallel branch for keypoint regression. Competitive experimental results on object detection, instance segmentation and human pose estimation show the generality of our point-set anchors.


  • [1] D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee (2019) YOLACT: real-time instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9157–9166. Cited by: §2, Table 5.
  • [2] A. Bulat and G. Tzimiropoulos (2016) Human pose estimation via convolutional part heatmap regression. In European Conference on Computer Vision, pp. 717–732. Cited by: §2.
  • [3] Z. Cao, T. Simon, S. Wei, and Y. Sheikh (2017) Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, Cited by: §2, §4.3, Table 10.
  • [4] X. Chen, R. Girshick, K. He, and P. Dollár (2019) Tensormask: a foundation for dense object segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2061–2069. Cited by: Table 5.
  • [5] Y. Chen, C. Shen, X. Wei, L. Liu, and J. Yang (2017) Adversarial posenet: a structure-aware convolutional network for human pose estimation. arXiv preprint arXiv:1705.00389. Cited by: §2.
  • [6] C. Chou, J. Chien, and H. Chen (2017) Self adversarial training for human pose estimation. arXiv preprint arXiv:1707.02439. Cited by: §2.
  • [7] X. Chu, W. Yang, W. Ouyang, C. Ma, A. L. Yuille, and X. Wang (2017) Multi-context attention for human pose estimation. arXiv preprint arXiv:1702.07432. Cited by: §2.
  • [8] J. Dai, K. He, and J. Sun (2016) Instance-aware semantic segmentation via multi-task network cascades. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    pp. 3150–3158. Cited by: §2.
  • [9] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei (2017) Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764–773. Cited by: §3.3, §4.2.
  • [10] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Cited by: §4.1.
  • [11] P. Dollár, P. Welinder, and P. Perona (2010) Cascaded pose regression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1078–1085. Cited by: §3.3, §3.3.
  • [12] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian (2019) Centernet: keypoint triplets for object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6569–6578. Cited by: §2.
  • [13] H. Fang, S. Xie, Y. Tai, and C. Lu (2017) Rmpe: regional multi-person pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2334–2343. Cited by: §4.3, Table 10.
  • [14] G. Gkioxari, A. Toshev, and N. Jaitly (2016)

    Chained predictions using convolutional neural networks

    In European Conference on Computer Vision, pp. 728–743. Cited by: §2.
  • [15] K. He, G. Gkioxari, P. Dollar, and R. Girshick (2017) Mask r-cnn. In International Conference on Computer Vision, Cited by: §2, §2, §2, §4.3, Table 10, Table 5.
  • [16] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §4.1.
  • [17] Z. Huang, L. Huang, Y. Gong, C. Huang, and X. Wang (2019) Mask scoring r-cnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6409–6418. Cited by: §2.
  • [18] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and B. Schiele (2016) DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer Vision, pp. 34–50. Cited by: §2.
  • [19] D. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §4.3.
  • [20] H. Law and J. Deng (2018) Cornernet: detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 734–750. Cited by: §1, §2, Table 5.
  • [21] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei (2017) Fully convolutional instance-aware semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2359–2367. Cited by: §2, Table 5.
  • [22] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie (2017) Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125. Cited by: §2, §3.3, §3.3.
  • [23] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár (2017) Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980–2988. Cited by: 2nd item, §2, §3.3, §3.3, §3.3, §4.1, Table 5.
  • [24] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, pp. 740–755. Cited by: §2, §3.1, §4.1, §4.3.
  • [25] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, and A. C. Berg (2016) Ssd: single shot multibox detector. In European conference on computer vision, pp. 21–37. Cited by: §3.3.
  • [26] S. Lloyd (1982) Least squares quantization in pcm. IEEE transactions on information theory 28 (2), pp. 129–137. Cited by: §3.1.
  • [27] A. Newell, Z. Huang, and J. Deng (2017) Associative embedding: end-to-end learning for joint detection and grouping. In Advances in Neural Information Processing Systems, pp. 2277–2287. Cited by: §2, §4.3, Table 10.
  • [28] A. Newell, K. Yang, and J. Deng (2016) Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision, pp. 483–499. Cited by: §2.
  • [29] X. Nie, J. Feng, J. Zhang, and S. Yan (2019) Single-stage multi-person pose machines. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6951–6960. Cited by: §2, §4.3, Table 10.
  • [30] G. Papandreou, T. Zhu, L. Chen, S. Gidaris, J. Tompson, and K. Murphy (2018) Personlab: person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 269–286. Cited by: §4.3, §4.4, Table 10.
  • [31] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy (2017) Towards accurate multi-person pose estimation in the wild. arXiv preprint arXiv:1701.01779. Cited by: §4.3, Table 10.
  • [32] S. Peng, W. Jiang, H. Pi, H. Bao, and X. Zhou (2020) Deep snake for real-time instance segmentation. arXiv preprint arXiv:2001.01629. Cited by: §2, §2.
  • [33] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, and B. Schiele (2016) Deepcut: joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4929–4937. Cited by: §2.
  • [34] S. Ren, K. He, R. Girshick, and J. Sun (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), Cited by: §2, §3.3.
  • [35] K. Sun, B. Xiao, D. Liu, and J. Wang (2019) Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5693–5703. Cited by: §4.4.
  • [36] X. Sun, Y. Wei, S. Liang, X. Tang, and J. Sun (2015) Cascaded hand pose regression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 824–832. Cited by: §3.3, §3.3.
  • [37] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei (2018) Integral human pose regression. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 529–545. Cited by: §2, §3.3.
  • [38] Z. Tian, C. Shen, H. Chen, and T. He (2019) Fcos: fully convolutional one-stage object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9627–9636. Cited by: §2, Table 5.
  • [39] B. Xiao, H. Wu, and Y. Wei (2018) Simple baselines for human pose estimation and tracking. arXiv preprint arXiv:1804.06208. Cited by: §2.
  • [40] E. Xie, P. Sun, X. Song, W. Wang, X. Liu, D. Liang, C. Shen, and P. Luo (2019) Polarmask: single shot instance segmentation with polar representation. arXiv preprint arXiv:1909.13226. Cited by: §2, §2, Table 5.
  • [41] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin (2019) Reppoints: point set representation for object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9657–9666. Cited by: §2, Table 5.
  • [42] Z. Yang, Y. Xu, H. Xue, Z. Zhang, R. Urtasun, L. Wang, S. Lin, and H. Hu (2019) Dense reppoints: representing visual objects with dense point sets. arXiv preprint arXiv:1912.11473. Cited by: §2, Table 5.
  • [43] X. Zhou, D. Wang, and P. Krähenbühl (2019) Objects as points. arXiv preprint arXiv:1904.07850. Cited by: §1, §2, §2, §4.3, §4.4, Table 10, Table 5.
  • [44] X. Zhou, J. Zhuo, and P. Krahenbuhl (2019) Bottom-up object detection by grouping extreme and center points. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 850–859. Cited by: §2, §2, Table 5.