High-Resolution Representations for Labeling Pixels and Regions

High-resolution representation learning plays an essential role in many vision problems, e.g., pose estimation and semantic segmentation. The high-resolution network (HRNet) SunXLW19, recently developed for human pose estimation, maintains high-resolution representations through the whole process by connecting high-to-low resolution convolutions in parallel and produces strong high-resolution representations by repeatedly conducting fusions across parallel convolutions. In this paper, we conduct a further study on high-resolution representations by introducing a simple yet effective modification and apply it to a wide range of vision tasks. We augment the high-resolution representation by aggregating the (upsampled) representations from all the parallel convolutions rather than only the representation from the high-resolution convolution as done in SunXLW19. This simple modification leads to stronger representations, evidenced by superior results. We show top results in semantic segmentation on Cityscapes, LIP, and PASCAL Context, and facial landmark detection on AFLW, COFW, 300W, and WFLW. In addition, we build a multi-level representation from the high-resolution representation and apply it to the Faster R-CNN object detection framework and the extended frameworks. The proposed approach achieves superior results to existing single-model networks on COCO object detection. The code and models have been publicly available at <https://github.com/HRNet>.



There are no comments yet.


page 1

page 2

page 3

page 4


Deep High-Resolution Representation Learning for Visual Recognition

High-resolution representations are essential for position-sensitive vis...

Lite-HRNet: A Lightweight High-Resolution Network

We present an efficient high-resolution network, Lite-HRNet, for human p...

A Holistically-Guided Decoder for Deep Representation Learning with Applications to Semantic Segmentation and Object Detection

Both high-level and high-resolution feature representations are of great...

OctNet: Learning Deep 3D Representations at High Resolutions

We present OctNet, a representation for deep learning with sparse 3D dat...

HRViT: Multi-Scale High-Resolution Vision Transformer

Vision transformers (ViTs) have attracted much attention for their super...

ResT V2: Simpler, Faster and Stronger

This paper proposes ResTv2, a simpler, faster, and stronger multi-scale ...

Fast and accurate object detection in high resolution 4K and 8K video using GPUs

Machine learning has celebrated a lot of achievements on computer vision...

Code Repositories


Pytorch implementation for Semantic Segmentation/Scene Parsing on MIT ADE20K dataset

view repo


Train the HRNet model on ImageNet

view repo


tensorflow implementation for "High-Resolution Representations for Labeling Pixels and Regions"

view repo



view repo


Simple inference implementation with trained HRNet.

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Deeply-learned representations have been demonstrated to be strong and achieved state-of-the-art results in many vision tasks. There are two main kinds of representations: low-resolution representations that are mainly for image classification, and high-resolution representations that are essential for many other vision problems, e.g., semantic segmentation, object detection, human pose estimation, etc. The latter one, the interest of this paper, remains unsolved and is attracting a lot of attention.

There are two main lines for computing high-resolution representations. One is to recover high-resolution representations from low-resolution representations outputted by a network (e.g., ResNet) and optionally intermediate medium-resolution representations, e.g., Hourglass [72], SegNet [2], DeconvNet [74], U-Net [83], and encoder-decoder [77]. The other one is to maintain high-resolution representations through high-resolution convolutions and strengthen the representations with parallel low-resolution convolutions [91, 30, 132, 86]

. In addition, dilated convolutions are used to replace some strided convolutions and associated regular convolutions in classification networks to compute medium-resolution representations 

[13, 126].

We go along the research line of maintaining high-resolution representations and further study the high-resolution network (HRNet), which is initially developed for human pose estimation [91], for a broad range of vision tasks. An HRNet maintains high-resolution representations by connecting high-to-low resolution convolutions in parallel and repeatedly conducting multi-scale fusions across parallel convolutions. The resulting high-resolution representations are not only strong but also spatially precise.

Figure 1: A simple example of a high-resolution network. There are four stages. The st stage consists of high-resolution convolutions. The nd (rd, th) stage repeats two-resolution (three-resolution, four-resolution) blocks. The detail is given in Section 3.

We make a simple modification by exploring the representations from all the high-to-low resolution parallel convolutions other than only the high-resolution representations in the original HRNet  [91]. This modification adds a small overhead and leads to stronger high-resolution representations. The resulting network is named as HRNetV. We empirically show the superiority to the original HRNet.

We apply our proposed network to semantic segmentation/facial landmark detection through estimating segmentation maps/facial landmark heatmaps from the output high-resolution representations. In semantic segmentation, the proposed approach achieves state-of-the-art results on PASCAL Context, Cityscapes, and LIP with similar model sizes and lower computation complexity. In facial landmark detection, our approach achieves overall best results on four standard datasets: AFLW, COFW, W, and WFLW.

In addition, we construct a multi-level representation from the high-resolution representation, and apply it to the Faster R-CNN object detection framework and its extended frameworks, Mask R-CNN [38] and Cascade R-CNN [9]. The results show that our method gets great detection performance improvement and in particular dramatic improvement for small objects. With single-scale training and testing, the proposed approach achieves better COCO object detection results than existing single-model methods.

2 Related Work

Strong high-resolution representations play an essential role in pixel and region labeling problems, e.g., semantic segmentation, human pose estimation, facial landmark detection, and object detection. We review representation learning techniques developed mainly in the semantic segmentation, facial landmark detection [92, 50, 69, 104, 123, 94, 119] and object detection areas111The techniques developed for human pose estimation are reviewed in [91]., from low-resolution representation learning, high-resolution representation recovering, to high-resolution representation maintaining.

Learning low-resolution representations. The fully-convolutional network (FCN) approaches [67, 87] compute low-resolution representations by removing the fully-connected layers in a classification network, and estimate from their coarse segmentation confidence maps. The estimated segmentation maps are improved by combining the fine segmentation score maps estimated from intermediate low-level medium-resolution representations [67], or iterating the processes [50]. Similar techniques have also been applied to edge detection, e.g., holistic edge detection [106].

The fully convolutional network is extended, by replacing a few (typically two) strided convolutions and the associated convolutions with dilated convolutions, to the dilation version, leading to medium-resolution representations [126, 13, 115, 12, 57]. The representations are further augmented to multi-scale contextual representations [126, 13, 15] through feature pyramids for segmenting objects at multiple scales.

Recovering high-resolution representations. An upsample subnetwork, like a decoder, is adopted to gradually recover the high-resolution representations from the low-resolution representations outputted by the downsample process. The upsample subnetwork could be a symmetric version of the downsample subnetwork, with skipping connection over some mirrored layers to transform the pooling indices, e.g., SegNet [2] and DeconvNet [74], or copying the feature maps, e.g., U-Net [83] and Hourglass [72, 111, 7, 22, 6], encoder-decoder [77], FPN [62], and so on. The full-resolution residual network [78] introduces an extra full-resolution stream that carries information at the full image resolution, to replace the skip connections, and each unit in the downsample and upsample subnetworks receives information from and sends information to the full-resolution stream.

The asymmetric upsample process is also widely studied. RefineNet [60] improves the combination of upsampled representations and the representations of the same resolution copied from the downsample process. Other works include: light upsample process [5]; light downsample and heavy upsample processes [97], recombinator networks [40]; improving skip connections with more or complicated convolutional units [76, 125, 42], as well as sending information from low-resolution skip connections to high-resolution skip connections [133] or exchanging information between them [36]; studying the details the upsample process [100]; combining multi-scale pyramid representations [16, 105]; stacking multiple DeconvNets/U-Nets/Hourglass [31, 101] with dense connections [93].

Maintaining high-resolution representations. High-resolution representations are maintained through the whole process, typically by a network that is formed by connecting multi-resolution (from high-resolution to low-resolution) parallel convolutions with repeated information exchange across parallel convolutions. Representative works include GridNet [30], convolutional neural fabrics [86], interlinked CNNs [132], and the recently-developed high-resolution networks (HRNet) [91] that is our interest.

The two early works, convolutional neural fabrics [86] and interlinked CNNs [132]

, lack careful design on when to start low-resolution parallel streams and how and when to exchange information across parallel streams, and do not use batch normalization and residual connections, thus not showing satisfactory performance.

GridNet [30] is like a combination of multiple U-Nets and includes two symmetric information exchange stages: the first stage only passes information from high-resolution to low-resolution, and the second stage only passes information from low-resolution to high-resolution. This limits its segmentation quality.

Figure 2: Multi-resolution block: (a) multi-resolution group convolution and (b) multi-resolution convolution. (c) A normal convolution (left) is equivalent to fully-connected multi-branch convolutions (right).

(a)             (b)             (c)  

Figure 3: (a) The high-resolution representation proposed in [91] (HRNetV ); (b) Concatenating the (upsampled) representations that are from all the resolutions for semantic segmentation and facial landmark detection (HRNetV ); (c) A feature pyramid formed over (b) for object detection (HRNetVp). The four-resolution representations at the bottom in each sub-figure are outputted from the network in Figure 1, and the gray box indicates how the output representation is obtained from the input four-resolution representations.

3 Learning High-Resolution Representations

The high-resolution network [91], which we named HRNetV for convenience, maintains high-resolution representations by connecting high-to-low resolution convolutions in parallel, where there are repeated multi-scale fusions across parallel convolutions.

Architecture. The architecture is illustrated in Figure 1. There are four stages, and the nd, rd and th stages are formed by repeating modularized multi-resolution blocks. A multi-resolution block consists of a multi-resolution group convolution and a multi-resolution convolution which is illustrated in Figure 2 (a) and (b). The multi-resolution group convolution is a simple extension of the group convolution, which divides the input channels into several subsets of channels and performs a regular convolution over each subset over different spatial resolutions separately.

The multi-resolution convolution is depicted in Figure 2 (b). It resembles the multi-branch full-connection manner of the regular convolution, illustrated in in Figure 2 (c). A regular convolution can be divided as multiple small convolutions as explained in [122]. The input channels are divided into several subsets, and the output channels are also divided into several subsets. The input and output subsets are connected in a fully-connected fashion, and each connection is a regular convolution. Each subset of output channels is a summation of the outputs of the convolutions over each subset of input channels.

The differences lie in two-fold. (i) In a multi-resolution convolution each subset of channels is over a different resolution. (ii) The connection between input channels and output channels needs to handle The resolution decrease is implemented in [91] by using several -strided convolutions. The resolution increase is simply implemented in [91] by bilinear (nearest neighbor) upsampling.

Modification. In the original approach HRNetV, only the representation (feature maps) from the high-resolution convolutions in [91] are outputted, which is illustrated in Figure 3 (a). This means that only a subset of output channels from the high-resolution convolutions is exploited and other subsets from low-resolution convolutions are lost.

We make a simple yet effective modification by exploiting other subsets of channels outputted from low-resolution convolutions. The benefit is that the capacity of the multi-resolution convolution is fully explored. This modification only adds a small parameter and computation overhead.

We rescale the low-resolution representations through bilinear upsampling to the high resolution, and concatenate the subsets of representations, illustrated in Figure 3 (b), resulting in the high-resolution representation, which we adopt for estimating segmentation maps/facial landmark heatmaps. In application to object detection, we construct a multi-level representation by downsampling the high-resolution representation with average pooling to multiple levels, which is depicted in Figure 3 (c). We name the two modifications as HRNetV and HRNetVp, respectively, and empirically compare them in Section 4.4.

Instantiation We instantiate the network using a similar manner as HRNetV [91]222https://github.com/leoxiaobin/deep-high-resolution-net.pytorch. The network starts from a stem that consists of two strided convolutions decreasing the resolution to . The st stage contains residual units where each unit is formed by a bottleneck with the width , and is followed by one convolution reducing the width of feature maps to . The nd, rd, th stages contain , , multi-resolution blocks, respectively. The widths (number of channels) of the convolutions of the four resolutions are , , , and , respectively. Each branch in the multi-resolution group convolution contains residual units and each unit contains two convolutions in each resolution.

In applications to semantic segmentation and facial landmark detection, we mix the output representations (Figure 3 (b)), from all the four resolutions through a convolution, and produce a

-dimensional representation. Then, we pass the mixed representation at each position to a linear classifier/regressor with the softmax/MSE loss to predict the segmentation maps/facial landmark heatmaps. For semantic segmentation, the segmentation maps are upsampled (

times) to the input size by bilinear upsampling for both training and testing. In application to object detection, we reduce the dimension of the high-resolution representation to , similar to FPN [62], through a convolution before forming the feature pyramid in Figure 3 (c).

4 Experiments

4.1 Semantic Segmentation

Semantic segmentation is a problem of assigning a class label to each pixel. We report the results over two scene parsing datasets, PASCAL Context [71] and Cityscapes [19], and a human parsing dataset, LIP [34]

. The mean of class-wise intersection over union (mIoU) is adopted as the evaluation metric.

backbone #param. GFLOPs mIoU
UNet++ [133] ResNet- M
DeepLabv3 [14] Dilated-ResNet- M
DeepLabv3+ [16] Dilated-Xception- M
PSPNet [126] Dilated-ResNet- M
Our approach HRNetV-W M
Our approach HRNetV-W M
Table 1: Segmentation results on Cityscapes val (single scale and no flipping). The GFLOPs is calculated on the input size .

Cityscapes. The Cityscapes dataset [19] contains high quality pixel-level finely annotated scene images. The finely-annotated images are divided into images for training, validation and testing. There are classes, and classes among them are used for evaluation. In addition to the mean of class-wise intersection over union (mIoU), we report other three scores on the test set: IoU category (cat.), iIoU class (cla.) and iIoU category (cat.).

We follow the same training protocol [126, 127]. The data are augmented by random cropping (from to ), random scaling in the range of , and random horizontal flipping. We use the SGD optimizer with the base learning rate of , the momentum of and the weight decay of . The poly learning rate policy with the power of is used for dropping the learning rate. All the models are trained for iterations with the batch size of on GPUs and syncBN.

backbone mIoU iIoU cla. IoU cat. iIoU cat.
Model learned on the train set
PSPNet [126] Dilated-ResNet-
PSANet [127] Dilated-ResNet- - - -
PAN [54] Dilated-ResNet- - - -
AAF [45] Dilated-ResNet- - - -
Our approach HRNetV-W
Model learned on the train+valid set
GridNet [30] -
LRR-4x [33] -
DeepLab [13] Dilated-ResNet-
LC [55] - - - -
Piecewise [61] VGG-
FRRN [78] -
RefineNet [60] ResNet-
PEARL [43] Dilated-ResNet-
DSSPN [59] Dilated-ResNet-
LKM [76] ResNet- - - -
DUC-HDC [99] -
SAC [120] Dilated-ResNet- - - -
DepthSeg [47] Dilated-ResNet- - - -
ResNet38 [103] WResNet-38
BiSeNet [113] ResNet- - - -
DFN [114] ResNet- - - -
PSANet [127] Dilated-ResNet- - - -
PADNet [108] Dilated-ResNet-
DenseASPP [126] WDenseNet-
Our approach HRNetV-W
Table 2: Semantic segmentation results on Cityscapes test.

Table 1 provides the comparison with several representative methods on the Cityscapes validation set in terms of parameter and computation complexity and mIoU class. (i) HRNetV-W ( indicates the width of the high-resolution convolution), with similar model size to DeepLabv+ and much lower computation complexity, gets better performance: points gain over UNet++, points gain over DeepLabv3 and about points gain over PSPNet, DeepLabv3+. (ii) HRNetV-W, with similar model size to PSPNet and much lower computation complexity, achieves much significant improvement: points gain over UNet++, points gain over DeepLabv3 and about points gain over PSPNet, DeepLabv3+. In the following comparisons, we adopt HRNetV-W

that is pretrained on ImageNet

333The description about ImageNet pretraining is given in the Appendix. and has similar model size as most Dilated-ResNet- based methods.

Table 2 provides the comparison of our method with state-of-the-art methods on the Cityscapes test set. All the results are with six scales and flipping. Two cases w/o using coarse data are evaluated: One is about the model learned on the train set, and the other is about the model learned on the train+valid set. In both cases, HRNetV-W achieves the best performance and outperforms the previous state-of-the-art by point.

backbone mIoU ( classes) mIoU ( classes)
FCN-[88] VGG- -
BoxSup [20] - -
HO_CRF [1] - -
Piecewise [61] VGG- -
DeepLab-v [13] Dilated-ResNet- -
RefineNet [60] ResNet- -
UNet++ [133] ResNet- -
PSPNet [126] Dilated-ResNet- -
Ding et al. [23] ResNet- -
EncNet [117] Dilated-ResNet- -
Our approach HRNetV-W
Table 3: Semantic segmentation results on PASCAL-context. The methods are evaluated on classes and classes.
backbone extra. pixel acc. avg. acc. mIoU
Attention+SSL [34] VGG Pose
DeepLabV[16] Dilated-ResNet- -
MMAN [68] Dilated-ResNet- - - -
SS-NAN [128] ResNet- Pose
MuLA [73] Hourglass Pose
JPPNet [58] Dilated-ResNet- Pose
CE2P [66] Dilated-ResNet- Edge
Our approach HRNetV-W N
Table 4: Semantic segmentation results on LIP. Our method doesn’t exploit any extra information, e.g., pose or edge.

PASCAL context. The PASCAL context dataset [71] includes scene images for training and images for testing with semantic labels and background label.

The data augmentation and learning rate policy are the same as Cityscapes. Following the widely-used training strategy [117, 23], we resize the images to and set the initial learning rate to and weight decay to . The batch size is and the number of iterations is .

We follow the standard testing procedure [117, 23]. The image is resized to and then fed into our network. The resulting label maps are then resized to the original image size. We evaluate the performance of our approach and other approaches using six scales and flipping.

Table 3 provides the comparison of our method with state-of-the-art methods. There are two kinds of evaluation schemes: mIoU over classes and classes ( classes + background). In both cases, HRNetV-W performs superior to previous state-of-the-arts.

LIP. The LIP dataset [34] contains elaborately annotated human images, which are divided into training images, and validation images. The methods are evaluated on categories ( human part labels and background label). Following the standard training and testing settings [66], the images are resized to and the performance is evaluated on the average of the segmentation maps of the original and flipped images.

The data augmentation and learning rate policy are the same as Cityscapes. The training strategy follows the recent setting [66]. We set the initial learning rate to and the momentum to and the weight decay to . The batch size is and the number of iterations is K.

Table 4 provides the comparison of our method with state-of-the-art methods. The overall performance of HRNetV-W performs the best with fewer parameters and lighter computation cost. We also would like to mention that our networks do not use extra information such as pose or edge.

R- H- R- H- R- H- X- H-
#param. (M)
Table 5: GFLOPs and #parameters of Faster R-CNN for COCO object detection. The numbers are obtained with the input size and proposals fed into R-CNN. ResNet--FPN (R-), X--d (X-101), HRNetV2p-W (H-).
backbone LS AP AP AP AP AP AP
Table 6: Object detection results evaluated on COCO val in the Faster R-CNN framework. LS = learning schedule.
backbone LS mask bbox
Table 7: Object detection results evaluated on COCO val in the Mask R-CNN framework. LS = learning schedule.
backbone size LS AP AP AP AP AP AP
MLKP [98] VGG - -
STDN [131] DenseNet- -
DES [124] VGG -
CoupleNet [137] ResNet- - -
DeNet [95] ResNet- -
RFBNet [64] VGG -
DFPR [48] ResNet- - - -
PFPNet [46] VGG -
RefineDet[121] ResNet- -
Relation Net [41] ResNet- - - - -
C-FRCNN [18] ResNet-
RetinaNet [63] ResNet--FPN
Deep Regionlets [109] ResNet- -
FitnessNMS [96] ResNet- -
DetNet [57] DetNet-FPN
CornerNet [52] Hourglass- -
M2Det [129] VGG
Faster R-CNN [62] ResNet--FPN
Faster R-CNN HRNetVp-W
Faster R-CNN [62] ResNet--FPN
Faster R-CNN HRNetVp-W
Faster R-CNN [62] ResNet--FPN
Faster R-CNN HRNetVp-W
Faster R-CNN [11] X--d-FPN
Faster R-CNN HRNetVp-W
Cascade R-CNN [9] ResNet--FPN
Cascade R-CNN ResNet--FPN
Cascade R-CNN HRNetVp-W
Table 8: Comparison with the state-of-the-art single-model object detectors on COCO test-dev without mutli-scale training and testing. We obtain the results of Faster R-CNN and Cascade R-CNN by using our implementations publicly available from the mmdetection platform[11] except that is from the original paper [9].

4.2 COCO Object Detection

We apply our multi-level representations (HRNetVp)444Same as FPN [63], we also use levels., shown in Figure 3 (c), in the Faster R-CNN [82] and Mask R-CNN [38] frameworks. We perform the evaluation on the MS-COCO detection dataset, which contains k images for training, k for validation (val) and k testing without provided annotations (test-dev). The standard COCO-style evaluation is adopted.

We train the models for both our HRNetVp and the ResNet on the public mmdetection platform [11] with the provided training setup, except that we use the learning rate schedule suggested in [37] for . The data is augmented by standard horizontal flipping. The input images are resized such that the shorter edge is 800 pixels [62]. Inference is performed on a single image scale.

Table 5 summarizes #parameters and GFLOPs. Table 6 and Table 7 report the detection results on COCO val. There are several observations. (i) The model size and computation complexity of HRNetVp-W (HRNetVp-W) are smaller than ResNet--FPN (ResNet--FPN). (ii) With , HRNetV2p-W performs better than ResNet--FPN. HRNetV2p-W performs worse than ResNet--FPN, which might come from insufficient optimization iterations. (iii) With , HRNetV2p-W and HRNetV2p-W perform better than ResNet--FPN and ResNet--FPN, respectively.

Table 8 reports the comparison of our network to state-of-the-art single-model object detectors on COCO test-dev without using multi-scale training and multi-scale testing that are done in [65, 79, 56, 90, 89, 75]. In the Faster R-CNN framework, our networks perform better than ResNets with similar parameter and computation complexity: HRNetVp-W vs. ResNet--FPN, HRNetVp-W vs. ResNet--FPN, HRNetVp-W vs. X--d-FPN. In the Cascade R-CNN framework, our HRNetVp-W performs better.

4.3 Facial Landmark Detection

Facial landmark detection a.k.a. face alignment is a problem of detecting the keypoints from a face image. We perform the evaluation over four standard datasets: WFLW [101], AFLW [49], COFW [8], and [85]. We mainly use the normalized mean error (NME) for evaluation. We use the inter-ocular distance as normalization for WFLW, COFW, and W, and the face bounding box as normalization for AFLW. We also report area-under-the-curve scores (AUC) and failure rates.

We follow the standard scheme [101] for training. All the faces are cropped by the provided boxes according to the center location and resized to . We augment the data by degrees in-plane rotation, scaling, and randomly flipping. The base learning rate is and is dropped to and at the th and

th epochs. The models are trained for

epochs with the batch size of on one GPU. Different from semantic segmentation, the heatmaps are not upsampled from

to the input size, and the loss function is optimized over the


At testing, each keypoint location is predicted by transforming the highest heatvalue location from to the original image space and adjusting it with a quarter offset in the direction from the highest response to the second highest response [17].

We adopt HRNetV-W for face landmark detection whose parameter and computation cost are similar to or smaller than models with widely-used backbones: ResNet- and Hourglass [72]. HRNetV-W: #parameters M, GFLOPs G; ResNet-: #parameters M, GFLOPs G; Hourglass: #parameters M, GFLOPs G. The numbers are obtained on the input size . It should be noted that the facial landmark detection methods adopting ResNet- and Hourglass as backbones introduce extra parameter and computation overhead.

WFLW. The WFLW dataset [101] is a recently-built dataset based on the WIDER Face [112]. There are training and testing images with manual annotated landmarks. We report the results on the test set and several subsets: large pose ( images), expression ( images), illumination ( images), make-up ( images), occlusion ( images) and blur ( images).

Table 9 provides the comparison of our method with state-of-the-art methods. Our approach is significantly better than other methods on the test set and all the subsets, including LAB that exploits extra boundary information [101] and PDB that uses stronger data augmentation [28].

backbone test pose expr. illu. mu occu. blur
ESR [10] -
SDM [107] -
CFSS [134] -
DVLN [102] VGG-16
Our approach HRNetV-W
Model trained with extra info.
LAB (w/ B) [101] Hourglass
PDB (w/ DA) [28] ResNet-
Table 9: Facial landmark detection results (NME) on WFLW test and subsets: pose, expression (expr.), illumination (illu.), make-up (mu.), occlusion (occu.) and blur. LAB [101] is trained with extra boundary information (B). PDB [28] adopts stronger data augmentation (DA). Lower is better.

AFLW. The AFLW [49] dataset is a widely used benchmark dataset, where each image has facial landmarks. Following [134, 101], we train our models on training images, and report the results on the AFLW-Full set ( testing images) and the AFLW-Frontal set ( testing images selected from testing images).

Table 10 provides the comparison of our method with state-of-the-art methods. Our approach achieves the best performance among methods without extra information and stronger data augmentation and even outperforms DCFE with extra D information. Our approach performs slightly worse than LAB that uses extra boundary information [101] and PDB [28] that uses stronger data augmentation.

backbone full frontal
RCN [40] -
CDM [116] -
ERT [44] -
LBF [80] -
SDM [107] -
CFSS [134] -
RCPR [8] -
CCL [135] -
DAC-CSR [29]
TSR [69] VGG-S -
CPM + SBR [25] CPM -
SAN [24] ResNet-
DSRN [70] - -
LAB (w/o B) [101] Hourglass
Our approach HRNetV2-W
Model trained with extra info.
DCFE (w/ D) [97] - -
PDB (w/ DA) [28] ResNet- -
LAB (w/ B) [101] Hourglass
Table 10: Facial landmark detection results (NME) on AFLW. DCFE [97] uses extra D information (D). Lower is better.

COFW. The COFW dataset [8] consists of training and testing faces with occlusions, where each image has facial landmarks.

Table 11 provides the comparison of our method with state-of-the-art methods. HRNetV outperforms other methods by a large margin. In particular, it achieves the better performance than LAB with extra boundary information and PDB with stronger data augmentation.

backbone NME FR
Human - -
ESR [10] -
RCPR [8] -
HPM [32] -
CCR [27] -
DRDA [118] -
RAR [104] -
DAC-CSR [29] -
LAB (w/o B) [101] Hourglass
Our approach HRNetV-W
Model trained with extra info.
PDB (w/ DA) [28] ResNet-
LAB (w/ B) [101] Hourglass
Table 11: Facial landmark detection results on COFW test. The failure rate is calculated at the threshold . Lower is better for NME and FR.

W. The dataset [85] is a combination of HELEN [53], LFPW  [4], AFW [136], XM2VTS  and IBUG datasets, where each face has landmarks. Following [81], we use the training images, which contains the training subsets of HELEN and LFPW and the full set of AFW. We evaluate the performance using two protocols, full set and test set. The full set contains images and is further divided into a common subset ( images) from HELEN and LFPW, and a challenging subset ( images) from IBUG. The official test set, used for competition, contains images ( indoor and outdoor images).

Table 12 provides the results on the full set, and its two subsets: common and challenging. Table 13 provides the results on the test set. In comparison to Chen et al. [17] that uses Hourglass with large parameter and computation complexity as the backbone, our scores are better except the AUC scores. Our HRNetV gets the overall best performance among methods without extra information and stronger data augmentation, and is even better than LAB with extra boundary information and DCFE [97] that explores extra D information.

backbone common challenging full
RCN [40] -
DSRN [70] -
PCD-CNN [51] -
CPM + SBR [25] CPM
SAN [24] ResNet-152
DAN [50] -
Our approach HRNetV-W
Model trained with extra info.
LAB (w/ B) [101] Hourglass
DCFE (w/ D) [97] -
Table 12: Facial landmark detection results (NME) on W: common, challenging and full. Lower is better.
backbone NME AUC AUC FR FR
Balt. et al. [3] - - - -
ESR [10] - - -
ERT [44] - - -
LBF [80] - - -
Face++ [130] - - - -
SDM [107] - - -
CFAN [119] - - -
Yan et al. [110] - - - -
CFSS [134] - - -
MDM [94] - - -
DAN [50] - - -
Chen et al. [17] Hourglass - -
Deng et al. [21] - - - -
Fan et al. [26] - - - -
DReg + MDM [35] ResNet101 - - -
JMFA [22] Hourglass - - -
Our approach HRNetV-W 52.09
Model trained with extra info.
LAB (w/ B) [101] Hourglass - - -
DCFE (w/ D) [97] - - -
Table 13: Facial landmark detection results on W test. DCFE [97] uses extra 3D information (3D). LAB [101] is trained with extra boundary information (B). Lower is better for NME, FR and FR, and higher is better for AUC and AUC.

4.4 Empirical Analysis

We compare the modified networks, HRNetV and HRNetVp, to the original network [91] (shortened as HRNetV) on semantic segmentation and COCO object detection. The segmentation and object detection results, given in Figure 4 (a) and Figure 4 (b), imply that HRNetV outperforms HRNetV significantly, except that the gain is minor in the large model case in segmentation for Cityscapes. We also test a variant (denoted by HRNetVh), which is built by appending a convolution to increase the dimension of the output high-resolution representation. The results in Figure 4 (a) and Figure 4 (b) show that the variant achieves slight improvement to HRNetV, implying that aggregating the representations from low-resolution parallel convolutions in our HRNetV is essential for increasing the capability.

(a)  (b) 

Figure 4: Empirical analysis. (a) Segmentation on Cityscapes val and PASCAL-Context test for comparing HRNetV and its variant HRNetVh, and HRNetV (single scale and no flipping). (b) Object detection on COCO val for comparing HRNetV and its variant HRNetVh, and HRNetVp (LS = learning schedule).

5 Conclusions

In this paper, we empirically study the high-resolution representation network in a broad range of vision applications with introducing a simple modification. Experimental results demonstrate the effectiveness of strong high-resolution representations and multi-level representations learned by the modified networks on semantic segmentation, facial landmark detection as well as object detection. The project page is https://jingdongwang2017.github.io/Projects/HRNet/.

Appendix: Network Pretraining

We pretrain our network, which is augmented by a classification head shown in Figure 5, on ImageNet [84]. The classification head is described as below. First, the four-resolution feature maps are fed into a bottleneck and the output channels are increased from , , , and to , , , and , respectively. Then, we downsample the high-resolution representation by a -strided convolution outputting channels and add it to the representation of the second-high-resolution. This process is repeated two times to get feature channels over the small resolution. Last, we transform the channels to channels through a convolution, followed by a global average pooling operation. The output -dimensional representation is fed into the classifier.

We adopt the same data augmentation scheme for training images as in [39], and train our models for epochs with a batch size of . The initial learning rate is set to and is reduced by times at epoch , and . We use SGD with a weight decay of

and a Nesterov momentum of

. We adopt standard single-crop testing, so that pixels are cropped from each image. The top- and top- error are reported on the validation set.

Table 14 shows our ImageNet classification results. As a comparison, we also report the results of ResNets. We consider two types of residual units: One is formed by a bottleneck, and the other is formed by two

convolutions. We follow the PyTorch implementation of ResNets and replace the

convolution in the input stem with two -strided convolutions decreasing the resolution to as in our networks. When the residual units are formed by two convolutions, an extra bottleneck is used to increase the dimension of output feature maps from to . One can see that under similar #parameters and GFLOPs, our results are comparable to and slightly better than ResNets.

In addition, we look at the results of two alternative schemes: (i) the feature maps on each resolution go through a global pooling separately and then are concatenated together to output a

-dimensional representation vector, named HRNet-W

-Ci; (ii) the feature maps on each resolution are fed into several -strided residual units (bottleneck, each dimension is increased to the double) to increase the dimension to , and concatenate and average-pool them together to reach a -dimensional representation vector, named HRNet-W-Cii, which is used in [91]. Table 15 shows such an ablation study. One can see that the proposed manner is superior to the two alternatives.

Figure 5: Representation for ImageNet classification. The input of the box is the representations of four resolutions.
#Params. GFLOPs top-1 err. top-5 err.
Residual branch formed by two convolutions
ResNet- M
ResNet- M
ResNet- M
Residual branch formed by a bottleneck
ResNet- M
ResNet- M
ResNet- M
Table 14: ImageNet Classification results of HRNet and ResNets. The proposed method is named HRNet-W-C.
#Params. GFLOPs top-1 err. top-5 err.
HRNet-W-Ci M
HRNet-W-Cii M
HRNet-W-Ci M
HRNet-W-Cii M
HRNet-W-Ci M
HRNet-W-Cii M
Table 15: Ablation study on ImageNet classification by comparing our approach (abbreviated as HRNet-W-C) with two alternatives: HRNet-W-Ci and HRNet-W-Cii (residual branch formed by two convolutions).


  • [1] A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr.

    Higher order conditional random fields in deep neural networks.

    In ECCV, pages 524–540, 2016.
  • [2] V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(12):2481–2495, 2017.
  • [3] T. Baltrusaitis, P. Robinson, and L. Morency. Constrained local neural fields for robust facial landmark detection in the wild. In ICCVW, pages 354–361, 2013.
  • [4] P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, and N. Kumar. Localizing parts of faces using a consensus of exemplars. IEEE Trans. Pattern Anal. Mach. Intell., 35(12):2930–2940, 2013.
  • [5] A. Bulat and G. Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV, volume 9911 of Lecture Notes in Computer Science, pages 717–732. Springer, 2016.
  • [6] A. Bulat and G. Tzimiropoulos. Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources. In ICCV, pages 3726–3734. IEEE Computer Society, 2017.
  • [7] A. Bulat and G. Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks). In ICCV, pages 1021–1030, 2017.
  • [8] X. P. Burgos-Artizzu, P. Perona, and P. Dollár. Robust face landmark estimation under occlusion. In ICCV, pages 1513–1520, 2013.
  • [9] Z. Cai and N. Vasconcelos. Cascade R-CNN: delving into high quality object detection. In CVPR, pages 6154–6162, 2018.
  • [10] X. Cao, Y. Wei, F. Wen, and J. Sun. Face alignment by explicit shape regression. In CVPR, pages 2887–2894, 2012.
  • [11] K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang, C. C. Loy, and D. Lin. mmdetection. https://github.com/open-mmlab/mmdetection, 2018.
  • [12] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062, 2014.
  • [13] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834–848, 2018.
  • [14] L. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, abs/1706.05587, 2017.
  • [15] L. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille. Attention to scale: Scale-aware semantic image segmentation. In CVPR, pages 3640–3649, 2016.
  • [16] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 833–851, 2018.
  • [17] Y. Chen, C. Shen, X. Wei, L. Liu, and J. Yang. Adversarial posenet: A structure-aware convolutional network for human pose estimation. In ICCV, pages 1221–1230, 2017.
  • [18] Z. Chen, S. Huang, and D. Tao. Context refinement for object detection. In ECCV, pages 74–89, 2018.
  • [19] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele.

    The cityscapes dataset for semantic urban scene understanding.

    In CVPR, pages 3213–3223, 2016.
  • [20] J. Dai, K. He, and J. Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, pages 1635–1643, 2015.
  • [21] J. Deng, Q. Liu, J. Yang, and D. Tao. M CSR: multi-view, multi-scale and multi-component cascade shape regression. Image Vision Comput., 47:19–26, 2016.
  • [22] J. Deng, G. Trigeorgis, Y. Zhou, and S. Zafeiriou. Joint multi-view face alignment in the wild. CoRR, abs/1708.06023, 2017.
  • [23] H. Ding, X. Jiang, B. Shuai, A. Q. Liu, and G. Wang. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In CVPR, pages 2393–2402, 2018.
  • [24] X. Dong, Y. Yan, W. Ouyang, and Y. Yang. Style aggregated network for facial landmark detection. In CVPR, pages 379–388, 2018.
  • [25] X. Dong, S. Yu, X. Weng, S. Wei, Y. Yang, and Y. Sheikh. Supervision-by-registration: An unsupervised approach to improve the precision of facial landmark detectors. In CVPR, pages 360–368, 2018.
  • [26] H. Fan and E. Zhou.

    Approaching human level facial landmark localization by deep learning.

    Image Vision Comput., 47:27–35, 2016.
  • [27] Z. Feng, P. Huber, J. Kittler, W. J. Christmas, and X. Wu. Random cascaded-regression copse for robust facial landmark detection. IEEE Signal Process. Lett., 22(1):76–80, 2015.
  • [28] Z. Feng, J. Kittler, M. Awais, P. Huber, and X. Wu.

    Wing loss for robust facial landmark localisation with convolutional neural networks.

    In CVPR, pages 2235–2245. IEEE Computer Society, 2018.
  • [29] Z. Feng, J. Kittler, W. J. Christmas, P. Huber, and X. Wu. Dynamic attention-controlled cascaded shape regression exploiting training data augmentation and fuzzy-set sample weighting. In CVPR, pages 3681–3690. IEEE Computer Society, 2017.
  • [30] D. Fourure, R. Emonet, É. Fromont, D. Muselet, A. Trémeau, and C. Wolf. Residual conv-deconv grid network for semantic segmentation. In BMVC, 2017.
  • [31] J. Fu, J. Liu, Y. Wang, and H. Lu. Stacked deconvolutional network for semantic segmentation. CoRR, abs/1708.04943, 2017.
  • [32] G. Ghiasi and C. C. Fowlkes. Occlusion coherence: Localizing occluded faces with a hierarchical deformable part model. In CVPR, pages 1899–1906. IEEE Computer Society, 2014.
  • [33] G. Ghiasi and C. C. Fowlkes. Laplacian pyramid reconstruction and refinement for semantic segmentation. In ECCV, pages 519–534, 2016.
  • [34] K. Gong, X. Liang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and A new benchmark for human parsing. CoRR, abs/1703.05446, 2017.
  • [35] R. A. Güler, G. Trigeorgis, E. Antonakos, P. Snape, S. Zafeiriou, and I. Kokkinos. Densereg: Fully convolutional dense shape regression in-the-wild. In CVPR, pages 2614–2623, 2017.
  • [36] J. Guo, J. Deng, N. Xue, and S. Zafeiriou. Stacked dense u-nets with dual transformers for robust face alignment. In BMVC, page 44, 2018.
  • [37] K. He, R. B. Girshick, and P. Dollár. Rethinking imagenet pre-training. CoRR, abs/1811.08883, 2018.
  • [38] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In ICCV, pages 2980–2988, 2017.
  • [39] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  • [40] S. Honari, J. Yosinski, P. Vincent, and C. J. Pal. Recombinator networks: Learning coarse-to-fine feature aggregation. In CVPR, pages 5743–5752, 2016.
  • [41] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networks for object detection. In CVPR, pages 3588–3597, 2018.
  • [42] M. A. Islam, M. Rochan, N. D. B. Bruce, and Y. Wang. Gated feedback refinement network for dense image labeling. In CVPR, pages 4877–4885, 2017.
  • [43] X. Jin, X. Li, H. Xiao, X. Shen, Z. Lin, J. Yang, Y. Chen, J. Dong, L. Liu, Z. Jie, J. Feng, and S. Yan. Video scene parsing with predictive feature learning. In ICCV, pages 5581–5589, 2017.
  • [44] V. Kazemi and J. Sullivan. One millisecond face alignment with an ensemble of regression trees. In CVPR, pages 1867–1874, 2014.
  • [45] T. Ke, J. Hwang, Z. Liu, and S. X. Yu. Adaptive affinity fields for semantic segmentation. In ECCV, pages 605–621, 2018.
  • [46] S. Kim, H. Kook, J. Sun, M. Kang, and S. Ko. Parallel feature pyramid network for object detection. In ECCV, pages 239–256, 2018.
  • [47] S. Kong and C. C. Fowlkes. Recurrent scene parsing with perspective understanding in the loop. In CVPR, pages 956–965, 2018.
  • [48] T. Kong, F. Sun, W. Huang, and H. Liu. Deep feature pyramid reconfiguration for object detection. In ECCV, pages 172–188, 2018.
  • [49] M. Köstinger, P. Wohlhart, P. M. Roth, and H. Bischof. Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization. In ICCV, pages 2144–2151, 2011.
  • [50] M. Kowalski, J. Naruniec, and T. Trzcinski. Deep alignment network: A convolutional neural network for robust face alignment. CoRR, abs/1706.01789, 2017.
  • [51] A. Kumar and R. Chellappa. Disentangling 3d pose in a dendritic CNN for unconstrained 2d face alignment. In CVPR, pages 430–439. IEEE Computer Society, 2018.
  • [52] H. Law and J. Deng. Cornernet: Detecting objects as paired keypoints. In ECCV, pages 765–781, 2018.
  • [53] V. Le, J. Brandt, Z. Lin, L. D. Bourdev, and T. S. Huang. Interactive facial feature localization. In ECCV (3), volume 7574 of Lecture Notes in Computer Science, pages 679–692. Springer, 2012.
  • [54] H. Li, P. Xiong, J. An, and L. Wang. Pyramid attention network for semantic segmentation. In BMVC, page 285, 2018.
  • [55] X. Li, Z. Liu, P. Luo, C. C. Loy, and X. Tang. Not all pixels are equal: Difficulty-aware semantic segmentation via deep layer cascade. In CVPR, pages 6459–6468, 2017.
  • [56] Z. Li, Y. Chen, G. Yu, and Y. Deng. R-FCN++: towards accurate region-based fully convolutional networks for object detection. In AAAI, pages 7073–7080, 2018.
  • [57] Z. Li, C. Peng, G. Yu, X. Zhang, Y. Deng, and J. Sun. Detnet: Design backbone for object detection. In ECCV, pages 339–354, 2018.
  • [58] X. Liang, K. Gong, X. Shen, and L. Lin. Look into person: Joint body parsing & pose estimation network and A new benchmark. CoRR, abs/1804.01984, 2018.
  • [59] X. Liang, H. Zhou, and E. Xing. Dynamic-structured semantic propagation network. In CVPR, pages 752–761, 2018.
  • [60] G. Lin, A. Milan, C. Shen, and I. D. Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In CVPR, pages 5168–5177, 2017.
  • [61] G. Lin, C. Shen, A. van den Hengel, and I. D. Reid. Efficient piecewise training of deep structured models for semantic segmentation. In CVPR, pages 3194–3203, 2016.
  • [62] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie. Feature pyramid networks for object detection. In CVPR, pages 936–944, 2017.
  • [63] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, pages 2999–3007, 2017.
  • [64] S. Liu, D. Huang, and Y. Wang. Receptive field block net for accurate and fast object detection. In ECCV, pages 404–419, 2018.
  • [65] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia. Path aggregation network for instance segmentation. In CVPR, pages 8759–8768, 2018.
  • [66] T. Liu, T. Ruan, Z. Huang, Y. Wei, S. Wei, Y. Zhao, and T. Huang. Devil in the details: Towards accurate single and multiple human parsing. CoRR, abs/1809.05996, 2018.
  • [67] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431–3440, 2015.
  • [68] Y. Luo, Z. Zheng, L. Zheng, T. Guan, J. Yu, and Y. Yang. Macro-micro adversarial network for human parsing. In ECCV, pages 424–440, 2018.
  • [69] J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection. In CVPR, pages 3691–3700, 2017.
  • [70] X. Miao, X. Zhen, X. Liu, C. Deng, V. Athitsos, and H. Huang. Direct shape regression networks for end-to-end face alignment. In CVPR, pages 5040–5049, 2018.
  • [71] R. Mottaghi, X. Chen, X. Liu, N. Cho, S. Lee, S. Fidler, R. Urtasun, and A. L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891–898, 2014.
  • [72] A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV, pages 483–499, 2016.
  • [73] X. Nie, J. Feng, and S. Yan. Mutual learning to adapt for joint human parsing and pose estimation. In ECCV, pages 519–534, 2018.
  • [74] H. Noh, S. Hong, and B. Han. Learning deconvolution network for semantic segmentation. In ICCV, pages 1520–1528, 2015.
  • [75] C. Peng, T. Xiao, Z. Li, Y. Jiang, X. Zhang, K. Jia, G. Yu, and J. Sun. Megdet: A large mini-batch object detector. In CVPR, pages 6181–6189, 2018.
  • [76] C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun. Large kernel matters - improve semantic segmentation by global convolutional network. In CVPR, pages 1743–1751, 2017.
  • [77] X. Peng, R. S. Feris, X. Wang, and D. N. Metaxas. A recurrent encoder-decoder network for sequential face alignment. In ECCV (1), volume 9905 of Lecture Notes in Computer Science, pages 38–56. Springer, 2016.
  • [78] T. Pohlen, A. Hermans, M. Mathias, and B. Leibe. Full-resolution residual networks for semantic segmentation in street scenes. In CVPR, pages 3309–3318, 2017.
  • [79] L. Qi, S. Liu, J. Shi, and J. Jia. Sequential context encoding for duplicate removal. In NeurIPS, pages 2053–2062, 2018.
  • [80] S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment at 3000 FPS via regressing local binary features. In CVPR, pages 1685–1692, 2014.
  • [81] S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment via regressing local binary features. IEEE Trans. Image Processing, 25(3):1233–1245, 2016.
  • [82] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal networks. volume abs/1506.01497, 2015.
  • [83] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241, 2015.
  • [84] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition challenge.

    International Journal of Computer Vision

    , 115(3):211–252, 2015.
  • [85] C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In ICCV Workshops, pages 397–403. IEEE Computer Society, 2013.
  • [86] S. Saxena and J. Verbeek. Convolutional neural fabrics. In NIPS, pages 4053–4061, 2016.
  • [87] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR, abs/1312.6229, 2013.
  • [88] E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):640–651, 2017.
  • [89] B. Singh and L. S. Davis. An analysis of scale invariance in object detection SNIP. In CVPR, pages 3578–3587, 2018.
  • [90] B. Singh, M. Najibi, and L. S. Davis. SNIPER: efficient multi-scale training. In NeurIPS, pages 9333–9343, 2018.
  • [91] K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
  • [92] Y. Sun, X. Wang, and X. Tang. Deep convolutional network cascade for facial point detection. In CVPR, pages 3476–3483. IEEE Computer Society, 2013.
  • [93] Z. Tang, X. Peng, S. Geng, L. Wu, S. Zhang, and D. N. Metaxas. Quantized densely connected u-nets for efficient landmark localization. In ECCV, pages 348–364, 2018.
  • [94] G. Trigeorgis, P. Snape, M. A. Nicolaou, E. Antonakos, and S. Zafeiriou. Mnemonic descent method: A recurrent process applied for end-to-end face alignment. In CVPR, pages 4177–4187, 2016.
  • [95] L. Tychsen-Smith and L. Petersson. Denet: Scalable real-time object detection with directed sparse sampling. In ICCV, pages 428–436, 2017.
  • [96] L. Tychsen-Smith and L. Petersson. Improving object localization with fitness NMS and bounded iou loss. In CVPR, pages 6877–6885, 2018.
  • [97] R. Valle, J. M. Buenaposada, A. Valdés, and L. Baumela. A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment. In ECCV, pages 609–624, 2018.
  • [98] H. Wang, Q. Wang, M. Gao, P. Li, and W. Zuo. Multi-scale location-aware kernel representation for object detection. In CVPR, pages 1248–1257, 2018.
  • [99] P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. W. Cottrell. Understanding convolution for semantic segmentation. In WACV, 2018.
  • [100] Z. Wojna, J. R. R. Uijlings, S. Guadarrama, N. Silberman, L. Chen, A. Fathi, and V. Ferrari. The devil is in the decoder. In BMVC, 2017.
  • [101] W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, and Q. Zhou. Look at boundary: A boundary-aware face alignment algorithm. In CVPR, pages 2129–2138, 2018.
  • [102] W. Wu and S. Yang. Leveraging intra and inter-dataset variations for robust face alignment. In CVPR, pages 2096–2105, 2017.
  • [103] Z. Wu, C. Shen, and A. van den Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. CoRR, abs/1611.10080, 2016.
  • [104] S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, and A. A. Kassim. Robust facial landmark detection via recurrent attentive-refinement networks. In ECCV, pages 57–72, 2016.
  • [105] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 432–448, 2018.
  • [106] S. Xie and Z. Tu. Holistically-nested edge detection. In ICCV, pages 1395–1403, 2015.
  • [107] X. Xiong and F. D. la Torre. Supervised descent method and its applications to face alignment. In CVPR, pages 532–539. IEEE Computer Society, 2013.
  • [108] D. Xu, W. Ouyang, X. Wang, and N. Sebe. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In CVPR, pages 675–684, 2018.
  • [109] H. Xu, X. Lv, X. Wang, Z. Ren, N. Bodla, and R. Chellappa. Deep regionlets for object detection. In ECCV, pages 827–844, 2018.
  • [110] J. Yan, Z. Lei, D. Yi, and S. Z. Li. Learn to combine multiple hypotheses for accurate face alignment. In ICCVW, pages 392–396, 2013.
  • [111] J. Yang, Q. Liu, and K. Zhang. Stacked hourglass network for robust facial landmark localisation. In CVPR, pages 2025–2033, 2017.
  • [112] S. Yang, P. Luo, C. C. Loy, and X. Tang.

    WIDER FACE: A face detection benchmark.

    In CVPR, pages 5525–5533. IEEE Computer Society, 2016.
  • [113] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV, pages 334–349, 2018.
  • [114] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang. Learning a discriminative feature network for semantic segmentation. In CVPR, pages 1857–1866, 2018.
  • [115] F. Yu, V. Koltun, and T. A. Funkhouser. Dilated residual networks. CoRR, abs/1705.09914, 2017.
  • [116] X. Yu, J. Huang, S. Zhang, W. Yan, and D. N. Metaxas. Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In ICCV, pages 1944–1951. IEEE Computer Society, 2013.
  • [117] H. Zhang, K. J. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal. Context encoding for semantic segmentation. In CVPR, pages 7151–7160, 2018.
  • [118] J. Zhang, M. Kan, S. Shan, and X. Chen.

    Occlusion-free face alignment: Deep regression networks coupled with de-corrupt autoencoders.

    In CVPR, pages 3428–3437. IEEE Computer Society, 2016.
  • [119] J. Zhang, S. Shan, M. Kan, and X. Chen. Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In ECCV (2), volume 8690 of Lecture Notes in Computer Science, pages 1–16. Springer, 2014.
  • [120] R. Zhang, S. Tang, Y. Zhang, J. Li, and S. Yan. Scale-adaptive convolutions for scene parsing. In ICCV, pages 2050–2058, 2017.
  • [121] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li. Single-shot refinement neural network for object detection. In CVPR, pages 4203–4212, 2018.
  • [122] T. Zhang, G. Qi, B. Xiao, and J. Wang. Interleaved group convolutions. In ICCV, pages 4383–4392, 2017.
  • [123] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial landmark detection by deep multi-task learning. In ECCV, pages 94–108, 2014.
  • [124] Z. Zhang, S. Qiao, C. Xie, W. Shen, B. Wang, and A. L. Yuille. Single-shot object detection with enriched semantics. In CVPR, pages 5813–5821, 2018.
  • [125] Z. Zhang, X. Zhang, C. Peng, X. Xue, and J. Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In ECCV, pages 273–288, 2018.
  • [126] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In CVPR, pages 6230–6239, 2017.
  • [127] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, pages 270–286, 2018.
  • [128] J. Zhao, J. Li, X. Nie, F. Zhao, Y. Chen, Z. Wang, J. Feng, and S. Yan. Self-supervised neural aggregation networks for human parsing. In CVPRW, pages 1595–1603, 2017.
  • [129] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, and H. Ling. M2det: A single-shot object detector based on multi-level feature pyramid network. CoRR, abs/1811.04533, 2018.
  • [130] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin. Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In ICCVW, pages 386–391, 2013.
  • [131] P. Zhou, B. Ni, C. Geng, J. Hu, and Y. Xu. Scale-transferrable object detection. In CVPR, pages 528–537, 2018.
  • [132] Y. Zhou, X. Hu, and B. Zhang. Interlinked convolutional neural networks for face parsing. In ISNN, pages 222–231, 2015.
  • [133] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang. Unet++: A nested u-net architecture for medical image segmentation. In MICCAI, pages 3–11, 2018.
  • [134] S. Zhu, C. Li, C. C. Loy, and X. Tang. Face alignment by coarse-to-fine shape searching. In CVPR, pages 4998–5006. IEEE Computer Society, 2015.
  • [135] S. Zhu, C. Li, C. C. Loy, and X. Tang. Unconstrained face alignment via cascaded compositional learning. In CVPR, pages 3409–3417, 2016.
  • [136] X. Zhu and D. Ramanan. Face detection, pose estimation, and landmark localization in the wild. In CVPR, pages 2879–2886. IEEE Computer Society, 2012.
  • [137] Y. Zhu, C. Zhao, J. Wang, X. Zhao, Y. Wu, and H. Lu. Couplenet: Coupling global structure with local parts for object detection. In ICCV, pages 4146–4154, 2017.