Despite the noticeable progress in perceptual tasks like detection, instance segmentation and human parsing, computers still perform unsatisfactorily on visually understanding humans in crowded scenes, such as group behavior analysis, person re-identification and autonomous driving, etc. To this end, models need to comprehensively perceive the semantic information and the differences between instances in a multi-human image, which is recently defined as the multi-human parsing task. In this paper, we present a new large-scale database "Multi-Human Parsing (MHP)" for algorithm development and evaluation, and advances the state-of-the-art in understanding humans in crowded scenes. MHP contains 25,403 elaborately annotated images with 58 fine-grained semantic category labels, involving 2-26 persons per image and captured in real-world scenes from various viewpoints, poses, occlusion, interactions and background. We further propose a novel deep Nested Adversarial Network (NAN) model for multi-human parsing. NAN consists of three Generative Adversarial Network (GAN)-like sub-nets, respectively performing semantic saliency prediction, instance-agnostic parsing and instance-aware clustering. These sub-nets form a nested structure and are carefully designed to learn jointly in an end-to-end way. NAN consistently outperforms existing state-of-the-art solutions on our MHP and several other datasets, and serves as a strong baseline to drive the future research for multi-human parsing.READ FULL TEXT VIEW PDF
|Datasets||Instance Aware?||# Total||# Training||# Validation||# Testing||# Category|
|MHP v1.0 ||✓||4,980||3,000||1,000||980||19|
One of the primary goals of intelligent human-computer interaction is understanding the humans in visual scenes. It involves several perceptual tasks including detection, i.e. localizing different persons at a coarse, bounding box level (Fig. 1 (a)), instance segmentation, i.e. labelling each pixel of each person uniquely (Fig. 1 (b)), and human parsing, i.e. decomposing persons into their semantic categories (Fig. 1
(c)). Recently, deep learning based methods have achieved remarkable sucess in these perceptual tasks thanks to the availability of plentiful annotated images for training and evaluation purposes[10, 11, 30, 17].
Though exciting, current progress is still far from the utimate goal of visually understanding humans. As Fig. 1 shows, previous efforts on understanding humans in visual scenes either only consider coarse information or are agnostic to different instances. In the real-world scenarios, it is more likely that there simutaneously exist multiple persons, with various human interactions, poses and occlusion. Thus, it is more practically demanded to parse human body parts and fashion items at the instance level, which is recently defined as the multi-human parsing task . Multi-human parsing enables more detailed understanding of humans in crowded scenes and aligns better with many real-world applications, such as group behavior analysis , person re-identification , e-commerce , image editing , video surveillance , autonomous driving  and virtual reality . However, the existing benchmark datasets [10, 11, 30, 17] are not suitable for such a new task. Even though Li et al.  proposed a preliminary Multi-Human Parsing (MHP v1.0) dataset, it only contains 4,980 images annotated with 18 semantic labels. In this work, we propose a new large-scale benchmark “Multi-Human Parsing (MHP v2.0)”, aiming to push the frontiers of multi-human parsing research towards holistically understanding humans in crowded scenes. The data in MHP v2.0 cover wide variability and complexity w.r.t. viewpoints, poses, occlusion, human interactions and background. It in total includes 25,403 human images with pixel-wise annotations of 58 semantic categories.
We further propose a novel deep Nested Adversarial Network (NAN) model for solving the challenging multi-human parsing problem. Unlike most existing methods [25, 21, 26] which rely on separate stages of instance localization, human parsing and result refinement, the proposed NAN parses semantic categories and differentiates different person instances simultaneously in an effective and time-efficient manner. NAN consists of three Generative Adversarial Network (GAN)-like sub-nets, respectively performing semantic saliency prediction, instance-agnostic parsing and instance-aware clustering. Each sub-task is simpler than the original multi-human parsing task, and is more easily addressed by the corresponding sub-net. Unlike many multi-task learning applications, in our method the sub-nets depend on each other, forming a causal nest by dynamically boosting each other through an adversarial strategy (See Fig. 5), which is hence called a “nested adversarial learning” structure. Such a structure enables effortless gradient BackproPagation (BP) in NAN such that it can be trained in a holistic, end-to-end way, which is favorable to both accuracy and speed. We conduct qualitative and quantitative experiments on the MHP v2.0 dataset proposed in this work, as well as the MHP v1.0 , PASCAL-Person-Part  and Buffy  benchmark datasets. The results demonstrate the superiority of NAN on multi-human parsing over the state-of-the-arts.
Our contributions are summarized as follows111 The dataset, annotation tools, and source codes for NAN and evaluation metrics are available at
The dataset, annotation tools, and source codes for NAN and evaluation metrics are available athttps://github.com/ZhaoJ9014/Multi-Human-Parsing_MHP.
We propose a new large-scale benchmark and evaluation server to advance understanding of humans in crowded scenes, which contains 25,403 images annotated pixel-wisely with 58 semantic category labels.
We propose a novel deep Nested Adversarial Network (NAN) model for multi-human parsing, which serves as a strong baseline to inspire more future research efforts on this task.
The statistics of popular publicly available datasets for human parsing are summarized in Tab. 1. The Buffy  dataset was released in 2011 for human parsing and instance segmentation. It contains only 748 images annotated with 13 semantic categories. The Fashionista  dataset was released in 2012 for human parsing, containing limited images annotated with 56 fashion categories. The PASCAL-Person-Part  dataset was initially annotated by Chen et al.  from the PASCAL-VOC-2010  dataset. Chen et al.  extended it for human parsing with 7 coarse body part labels. The ATR  dataset was released in 2015 for human parsing with a large number of images annotated with 18 semantic categories. The LIP  dataset further extended ATR  by cropping person instances from Microsoft COCO . It is a large-scale human parsing dataset with densely pixel-wise annotations of 20 semantic categories. But it has two limitations. 1) Despite the large data size, it contains limited semantic category annotations, which restricts the fine-grained understanding of humans in visual scenes. 2) In LIP , only a small proportion of images involve multiple persons with interactions. Such an instance-agnostic setting severely deviates from reality. Even in the MHP v1.0 dataset proposed by Li et al.  for multi-human parsing, only 4,980 images are included and annotated with 18 semantic labels. Comparatively, our MHP v2.0 dataset contains 25,403 elaborately annotated images with 58 fine-grained semantic part labels. It is the largest and most comprehensive multi-human parsing dataset to date, to our best knowledge. Visual comparisons between LIP , MHP v1.0  and our MHP v2.0 are provided in Fig. 2.
Recently, many research efforts have been devoted to human parsing [27, 17, 31, 46, 19, 9, 25, 21, 26] due to its wide range of potential applications. For example, Liang et al.  proposed a proposal-free network for instance segmentation by directly predicting the instance numbers of different categories and the pixel-level information. Gong et al.  proposed a self-supervised structure-sensitive learning approach, which imposes human pose structures to parsing results without resorting to extra supervision. Liu et al. 
proposed a single frame video parsing method which integrates frame parsing, optical flow estimation and temporal fusion into a unified network. Zhaoet al.  proposed a self-supervised neural aggregation network, which learns to aggregate the multi-scale features and incorporates a self-supervised joint loss to ensure the consistency between parsing and pose. He et al.  proposed the Mask R-CNN, which is extended from Faster R-CNN  by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Brabandere et al. 
proposed to tackle instance segmentation with a discriminative loss function, operating at the pixel level, which encourages a convolutional network to produce a representation of the image that can be easily clustered into instances with a simple post-processing step. However, these methods either only consider coarse semantic information or are agnostic to different instances. To enable more detailed human-centric analysis, Liet al.  initially proposed the multi-human parsing task, which aligns better with the realistic scenarios. They also proposed a novel MH-Parser model as a reference method which generates parsing maps and instance masks simutaneously in a bottom-up fashion. Jiang et al.  proposed a new approach to segment human instances and label their body parts using region assembly. Li et al.  proposed a framework with a human detector and a category-level segmentation module to segment the parts of objects at the instance level. These methods involve mutiple separate stages for instance localization, human parsing and result refinement. In comparison, the proposed NAN produces accurate multi-human parsing results through a single forward-pass in a time-efficient manner without tedious pre- or post-processing.
In this section, we introduce the “Multi-Human Parsing (MHP v2.0)”, a new large-scale dataset focusing on semantic understanding of humans in crowded scenes with several appealing properties. 1) It contains 25,403 elaborately annotated images with 58 fine-grained labels on body parts, fashion items and one background label, which is larger and more comprehensive than previous similar attempts [39, 25]. 2) The images within MHP v2.0 are collected from real-world scenarios, involving humans with various viewpoints, poses, occlusion, interactions and resolution. 3) The background of images in MHP v2.0 is more complex and diverse than previous datasets. Some examples are showed in Fig. 2. The MHP v2.0 dataset is expected to provide a new benchmark suitable for multi-human parsing together with a standard evaluation server where the test set will be kept secret to avoid overfitting.
We manually specify some underlying relationships (such as family, couple, team, etc.) and possible scenes (such as sports, conferences, banquets, etc.) to ensure the diversity of returned results. Based on any one of these specifications, corresponding multi-human images are located by performing Internet searches over Creative Commons licensed imagery. For each identified image, the contained human number and the corresponding URL are stored in a spreadsheet. Automated scrapping software is used to download the multi-human imagery and stores all relevant information in a relational database. Moreover, a pool of images containing clearly visible persons with interactions and rich fashion items is also constructed from the existing human-centric datasets [44, 5, 45, 36, 22]222PASCAL-VOC-2012  and Microsoft COCO  are not included due to limited percent of crowd-scene images with fine details of persons. to augment and complement Internet scraping results.
After curating the imagery, manual annotation is conducted by professional data annotators, which includes two distinct tasks. The first task is manually counting the number of foreground persons and duplicating each image to several copies according to the count number. Each duplicated image is marked with the image ID, the contained person number and a self-index. The second is assigning the fine-grained pixel-wise label to each semantic category for each person instance. We implement an annotation tool and generate multi-scale superpixels of images based on  to speed up the annotation. See Fig. 4 for an example. Each multi-human image contains at least two instances. The annotation for each instance is done in a left-to-right order, corresponding to the duplicated image with the self-index from beginning to end. For each instance, 58 semantic categories are defined and annotated, including cap/hat, helmet, face, hair, left-arm, right-arm, left-hand, right-hand, protector, bikini/bra, jacket/windbreaker/hoodie, t-shirt, polo-shirt, sweater, singlet, torso-skin, pants, shorts/swim-shorts, skirt, stockings, socks, left-boot, right-boot, left-shoe, right-shoe, left-highheel, right-highheel, left-sandal, right-sandal, left-leg, right-leg, left-foot, right-foot, coat, dress, robe, jumpsuits, other-full-body-clothes, headwear, backpack, ball, bats, belt, bottle, carrybag, cases, sunglasses, eyewear, gloves, scarf, umbrella, wallet/purse, watch, wristband, tie, other-accessaries, other-upper-body-clothes and other-lower-body-clothes. Each instance has a complete set of annotations whenever the corresponding category appears in the current image. When annotating one instance, others are regarded as background. Thus, the resulting annotation set for each image consists of instance-level parsing masks, where is the number of persons in the image.
After annotation, manual inspection is performed on all images and corresponding annotations to verify the correctness. In cases where annotations are erroneous, the information is manually rectified by 5 well informed analysts. The whole work took around three months to accomplish by 25 professional data annotators.
In total, there are 25,403 images in the MHP v2.0 dataset. Each image contains 2-26 person instances, with 3 on average. The resolution of the images ranges from 85100 to 4,5116,919, with 644718 on average. We spit the images into training, validation and testing sets. Following random selection, we arrive at a unique split consisting of 15,403 training and 5,000 validation images with publicly available annotations, as well as 5,000 testing images with annotations withheld for benchmarking purpose.
The statistics w.r.t. data distribution on 59 semantic categories, the average semantic category number per image and the average instance number per image in the MHP v2.0 dataset are illustrated in Fig. 3 (a), (b) and (c), respectively. In general, face, arms and legs are the most remarkable parts of a human body. However, understanding humans in crowded scenes needs to analyze fine-grained details of each person of interest, including different body parts, clothes and accessaries. We therefore define 11 body parts, and 47 clothes and accessaries. Among these 11 body parts, we divide arms, hands, legs and feet into left and right side for more precise analysis, which also increases the difficulty of the task. We define hair, face and torso-skin as the remaining three body parts, which can be used as auxiliary guidance for more comprehensive instance-level analysis. As for clothing categories, we have common clothes like coat, jacket/windbreaker/hoodie, sweater, singlet, pants, shorts/swim-shorts and shoes, confusing categories such as t-shirt v.s. polo-shirt, stockings v.s. socks, skirt v.s. dress and robe, and boots v.s. sandals and highheels, and infrequent categories such as cap/hat, helmet, protector, bikini/bra, jumpsuits, gloves and scarf. Furthermore, accessaries like sunglasses, belt, tie, watch and bags are also taken into account, which are common but hard to predict, especially for the small-scale ones.
To summarize, the pre-defined semantic categories of MHP v2.0 involve most body parts, clothes and accessaries of different styles for men, women and children in all seasons. The images in the MHP v2.0 dataset contain diverse instance numbers, viewpoints, poses, occlusion, interactions and background complexities. MHP v2.0 aligns better with real-world scenarios and serves as a more realistic benchmark for human-centric analysis, which pushes the frontiers of fine-grained multi-human parsing research.
As shown in Fig. 5, the proposed deep Nested Adversarial Network (NAN) model consists of three GAN-like sub-nets that jointly perform semantic saliency prediction, instance-agnostic parsing and instance-aware clustering end-to-end. NAN produces accurate multi-human parsing results through a single forward-pass in a time-efficient manner without tedious pre- or post-processing. We now present each component in details.
Large modality and interaction variations are the main challenge to multi-human parsing and also the key obstacle to learning a well-performing human-centric analysis model. To address this problem, we propose to decompose the original task into three granularities and adaptively impose a prior on the specific process, each with the aid of a GAN-based sub-net. This reduces the training complexity and leads to better empirical performance with limited data.
The first sub-net estimates semantic saliency maps to locate the most noticeable and eye-attracting human regions in images, which serves as a basic prior to facilitate further processing on humans, as illustrated in Fig. 5 left. We formulate semantic saliency prediction as a binary pixel-wise labelling problem to segment out foreground v.s. background. Inspired by the recent success of Fully Convolutional Networks (FCNs)  based methods on image-to-image applications [24, 19], we leverage an FCN backbone (FCN-8s ) as the generator of NAN for semantic saliency prediction, where denotes the network parameters, and , , and denote the image height, width, channel number and semantic category (i.e., foreground plus background) number, repectively.
Formally, let the input RGB image be denoted by and the semantic saliency map be denoted by , then
The key requirements for are that the semantic saliency map should present indistinguishable properities compared with a real one (i.e., ground truth) in appearance while preserving the intrinsic contextually remarkable information.
To this end, we propose to learn by minimizing a combination of two losses:
where is the adversarial loss for refining realism and alleviating artifacts, is the semantic saliency loss for pixel-wise image labelling, are weighting parameters among different losses.
is a pixel-wise cross-entropy loss calculated based on the binary pixel-wise annotations to learn :
is proposed to narrow the gap between the distributions of generated and real results. To facilitate this process, we leverage a Convolutional Neural Network (CNN) backbone as the discriminator to be as simple as possible to avoid typical GAN tricks. We alternatively optimize and to learn and :
where denotes the binary real v.s. fake indicator.
The second sub-net concatenates the information from the original RGB image with semantic saliency prior as input and estimates a fine-grained instance-agnostic parsing map, which further serves as stronger semantic guidance from the global perspective to facilitate instance-aware clustering, as illustrated in Fig. 5 middle. We formulate instance-agnostic parsing as a multi-class dense classification problem to mask semantically consistent regions of body parts and fashion items. Inspired by the leading performance of the skip-net on recognition tasks [40, 20], we modify a skip-net (WS-ResNet ) into an FCN-based architecture as the generator
of NAN to learn a highly non-linear transformation for instance-agnostic parsing, wheredenotes the network parameters for the generator and denotes the semantic category number. The prediction is downsampled by for accuracy v.s. speed trade-off. Contextual information from global and local regions compensates each other and naturally benefits human parsing. The hierarchical features within a skip-net are multi-scale in nature due to the increasing receptive field sizes, which are combined together via skip connections. Such a combined representation comprehensively maintains the contextual information, which is crucial for generating smooth and accurate parsing results.
Formally, let the instance-agnostic parsing map be denoted by , then
Similar to the first sub-net, we propose to learn by minimizing:
where is the global parsing loss for semantic part labelling.
is a standard pixel-wise cross-entropy loss calculated based on the multi-class pixel-wise annotations to learn .
is also slightly finetuned due to the hinged gradient backpropagation route within the nested structure:
is proposed to ensure the correctness and realism of the current phase and also the previous one for information flow consistency. To facilitate this process, we leverage a same CNN backbone with as the discriminator , which are learned separately. We alternatively optimize and to learn , and slightly finetune :
The third sub-net concatenates the information from the original RGB image with semantic saliency and instance-agnostic parsing priors as input and estimates an instance-aware clustering map by associating each semantic parsing mask to one of the person instances in the scene, as illustrated in Fig. 5 right. Inspired by the observation that a human glances at an image and instantly knows how many and where the objects are in the image, we formulate instance-aware clustering by parallelly inferring the instance number and pixel-wise instance location, discarding the requirement of time-consuming region proposal generation. We modify a same backbone architecture to incorporate two sibling branches as the generator of NAN for location-sensitive learning, where denotes the network parameters for the generator and denotes the pre-defined instance location coordinate number. As multi-scale features integrating both global and local contextual information are crucial for increasing location prediction accuracy, we further augment the pixel-wise instance location prediction branch with a Multi-Scale Fusion Unit (MSFU) to fuse shallow-, middle- and deep-level features, while using the feature maps downsampled by concatenated with feature maps from the first branch for instance number regression.
Formally, let the pixel-wise instance location map be denoted by and the instance number be denoted by , then
We propose to learn by minimizing:
where is the pixel-wise instance location loss for pixel-wise instance location regression and is the instance number loss for instance number regression.
is a standard smooth- loss  calculated based on the foreground pixel-wise instance location annotations to learn . Since a person instance can be identified by its top-left corner and bottom-right corner
of the surrounding bounding box, for each pixel belonging to the person instance, the pixel-wise instance location vector is defined as, where and are the width and height of the person instance for normalization, respectively. is a standard loss calculated based on the instance number annotations to learn . and are also slightly finetuned due to the chained schema within the nest:
Given these information, instance-aware clustering maps can be effortlessly generated with little computational overhead, which are denoted by . Similar to , is proposed to ensure the correctness and realism of all phases for the information flow consistency. To facilitate this process, we leverage a same CNN backbone with as the discriminator , which are learned separately. We alternatively optimize and to learn , and slightly finetune and :
The goal of NAN is to use sets of real targets to learn three GAN-like sub-nets that mutually boost and jointly accomplish multi-human parsing. Each separate loss serves as a deep supervision within the nested structure benefitting network convergence. The overall objective function for NAN is
Clearly, the NAN is end-to-end trainable and can be optimized with the proposed nested adversarial learning strategy and BP algorithm.
During testing, we simply feed the input image into NAN to get the instance-agnostic parsing map from , pixel-wise instance location map and instance number from . Then we employ an off-the-shelf clustering method  to obtain the instance-aware clustering map . Example results are visualized in Fig. 6.
We evaluate NAN qualitatively and quantitatively under various settings and granularities for understanding humans in crowded scenes. In particular, we evaluate multi-human parsing performance on the MHP v2.0 dataset proposed in this work, as well as the MHP v1.0  and PASCAL-Person-Part  benchmark datasets. We also evaluate instance-agnostic parsing and instance-aware clustering results on the Buffy  benchmark dataset, which are byproducts of NAN.
Throughout the experiments, the sizes of the RGB image , the semantic saliency prediction , inputs to the discriminator and inputs to the generator are fixed as ; the sizes of the instance-agnostic parsing prediction , instance-aware clustering prediction , inputs to the discriminator , inputs to the generator , inputs to the discriminator and instance location map are fixed as ; the channel number of the pixel-wise instance location map is fixed as , incorporating two corner points of the associated bounding box; the constraint factors are empirically fixed as and , respectively; the generator is initialized with FCN-8s  by replacing the last layer with a new convolutional layer with kernel size , pretrained on PASCAL-VOC-2011  and finetuned on the target dataset; the generator is initialized with WS-ResNet 
by eliminating the spatial pooling layers, increasing the strides of the first convolutional layers up to 2 in B
, eliminating the top-most global pooling layer and the linear classifier, and adding two new convolutional layers with kernel sizesand
, pretrained on ImageNet and PASCAL-VOC-2012 , and finetuned on the target dataset; the generator is initialized with the same backbone architecture and pre-trained weights with (which are learned separately), by further augmenting it with two sibling branches for pixel-wise instance location map prediction and instance number prediction, where the first branch utilizes a MSFU (three convolutional layers with kernal sizes for specific scale adaption) ended with a convolutional layer with kernel size for multi-scale feature aggregation and a final convolutional layer with kernel size for location regression and the second branch utilizes the feature maps downsampled by 8 concatenated with the feature maps from the first branch ended with a global pooling layer, a hidden 512-way fully-connected layer and a final 1-way fully-connected layer for instance number regression; the three discriminators (which are learned separately) are all initialized with a VGG-16  by adding a new convolutional layer at the very begining with kernel size for input adaption, and replacing the last layer with a new 1-way fully-connected layer activated by sigmoid, pre-trained on ImageNet ; we employ an off-the-shelf clustering method  to obtain the instance-aware clustering map ; the dropout ratio is empirically fixed as ; the weight decay and batch size are fixed as and , respectively; We use an initial learning rate of for pre-trained layers, and for newly added layers in all our experiments; we decrease the learning rate to 1] platform, which is trained using Adam () on four NVIDIA GeForce GTX TITAN X GPUs with 12G memory; the same training setting is utilized for all our compared network variants; we evaluate the testing time by averaging the running time for images on the target set on NVIDIA GeForce GTX TITAN X GPU and Intel Core i7-4930K CPU@3.40GHZ; our NAN can rapidly process one image in about 1 second, which compares much favorably to other state-of-the-art approaches, as the current state-of-the-art methods [25, 21, 26] rely on region proposal preprocessing and complex processing steps.
Following , we use the Average Precision based on part () and Percentage of Correctly parsed semantic Parts (PCP) metrics for multi-human parsing evaluation. Different from the Average Precision based on region () used in instance segmentation [27, 18], uses part-level pixel Intersection over Union (IoU) of different semantic part categories within a person instance to determine if one instance is a true positive. We prefer over as we focus on human-centric analysis and we aim to investigate to how well a person instance as a whole is parsed. Additionally, we also report the , which is the mean of the at IoU thresholds ranging from to , in increments of 0.1. As averages the IoU of each semantic part category, it fails to reflect how many semantic parts are correctly parsed. We further incorporate the PCP, originally used in human pose estimation [14, 4], to evaluate the parsing quality within person instances. For each true-positive person instance, we find all the semantic categories (excluding background) with pixel IoU larger than a threshold, which are regarded as correctly parsed. The PCP of one person instance is the ratio between the correctly parsed semantic category number and the total semantic category number of that person. Missed person instances are assigned with PCP. The overall PCP is the average PCP for all person instances. Note that PCP is also a human-centric evaluation metric.
The MHP v2.0 dataset proposed in this paper is the largest and most comprehensive multi-human parsing benchmark to date, which extends MHP v1.0  to push the frontiers of understanding humans in crowded scenes by containing 25,403 elaborately annotated images with 58 fine-grained semantic category labels. Annotation examples are visualized in Fig. 2 (c). The data are randomly organized into 3 splits, consisting of 15,403 training and 5,000 validation images with publicly available annotations, as well as 5,000 testing images with annotations withheld for benchmarking purpose. Evaluation systems report the and PCP over the validation and testing sets.
We first investigate different architectures and loss function combinations of NAN to see their respective roles in multi-human parsing. We compare 16 variants from four aspects, i.e., different baselines (Mask R-CNN 333As existing instance segmentation methods only offer silhouettes of different person instances, for comparison, we combine them with our instance-agnostic parsing prediction to generate the final multi-human parsing results. and MH-Parser ), different network structures (w/o , w/o concatenated input (RGB only), w/o concatenated input (RGB only), w/o , w/o , w/o concatenated input, w/o , w/o concatenated input, w/o MSFU), our proposed NAN, and upperbounds (: use the ground truth semantic saliency maps instead of prediction while keeping other settings the same; : use the ground truth instance-agnostic parsing maps instead of prediction while keeping other settings the same; : use the ground truth instance number instead of prediction while keeping other settings the same; : use the ground truth pixel-wise instance location maps instead of prediction while keeping other settings the same).
|Baseline||Mask R-CNN ||14.50||33.51||25.12|
|w/o concatenated input||21.88||36.79||29.02|
|w/o concatenated input||22.36||35.92||25.48|
|w/o concatenated input||21.55||31.94||24.90|
|w/o concatenated input||21.80||34.54||27.30|
The performance comparison in terms of @IoU=0.5, and PCP@IoU=0.5 on the MHP v2.0 validation set is reported in Tab. 2. By comaring the results from the v.s. panels, we observe that our proposed NAN consistently outperforms the baselines Mask R-CNN  and MH-Parser  by a large margin, i.e., and in terms of , and in terms of , and and in terms of PCP. Mask R-CNN  suffers difficulties to differentiate entangled humans. MH-Parser  involves multiple stages for instance localization, human parsing and result refinement with high complexity, yielding sub-optimal results, whereas NAN parses semantic categories, differentiates different person instances and refines results simultaneously through deep nested adversarial learning in an effective yet time-efficient manner. By comaring the results from the v.s. panels, we observe that NAN consistently outperforms the 9 variants in terms of network structure. In particular, w/o refers to truncating the semantic saliency prediction sub-net from NAN, leading to , and performance drop in terms of all metrics. This verifies the necessity of semantic saliency prediction that locates the most noticeable human regions in images to serve as a basic prior to facilitate further human-centic processing. The superiority of incorporating adaptive prior information to specific process can be verified by comparing w/o concatenated input with NAN, i.e., , and ; , and differences in terms of all metrics. The superiority of incorporating adversarial learning to specific process can be verified by comparing w/o with NAN, i.e., , and ; , and ; , and decrease in terms of all metrics. Nested adversarial learning strategy ensures the correctness and realism of all phases for information flow consistency, the superiority of which is verified by comparing w/o concatenated input with NAN, i.e., , and ; , and decline in terms of all metrics. MSFU dynamically fuses multi-scale features for enhancing instance-aware clustering accuracy, the superiority of which is verified by comparing w/o MSFU with NAN, i.e., , and drop in terms of all metrics. Finally, we also evaluate the limitations of our current algorithm. By comparing with NAN, only , and improvement in term of all metrics are obtained, which shows that the errors from semantic saliency prediction are already small and have only little effect on the final results. A large gap between , and of and , and of NAN shows that a better instance-agnostic parsing network architecture can definitely help improve the performance of multi-human parsing under our NAN framework. By comparing and with NAN, , and ; , and improvement in term of all metrics are obtained, which shows that accurate instance-aware clustering results are critical for superior multi-human parsing.
|Mask R-CNN ||14.90||33.88||25.11||4.77||24.28||12.75||2.23||20.73||8.38||-|
The performance comparison of the proposed NAN with two state-of-the-art methods in terms of @IoU=0.5, and PCP@IoU=0.5 on the MHP v2.0 testing set is reported in Tab. 3. Following , we conduct experiments under three settings: All reports the evaluation over the whole testing set; Inter reports the evaluation over the sub-set containing the images with top 20% interaction intensity444For each testing image, we calculate the pair-wise instance bounding box IoU and use the mean value as the interaction intensity for each image.; Inter reports the evaluation over the sub-set containing the images with top 10% interaction intensity. Our NAN is significantly superior over other state-of-the-arts on setting-1. In particular, NAN improves the -best by , and in terms of all metrics. For the more challenging scenarios with intensive interactions (setting-2, 3), NAN also consistently achieves the best performance. In particular, for Inter and Inter, NAN improves the -best by , and ; , and in terms of all metrics. This verifies the effectiveness of our NAN for multi-human parsing and understanding humans in crowded scenes. Moreover, NAN can rapidly process one 512512 image in about 1 second with acceptable resource consumption, which is attractive to real applications. This compares much favorably to MH-Parser  (14.94 img/s), which relies on separate and complex post-processing (including CRF ) steps.
Fig. 6 visualizes the qualitative comparison of the proposed NAN with two state-of-the-art methods and corresponding ground truths on the MHP v2.0 dataset. Note that Mask R-CNN  only offers silhouettes of different person instances, we only compare our instance-aware clustering results with it while comparing our holistic results with MH-Parser . It can be observed that the proposed NAN performs well in multi-human parsing with a wide range of viewpoints, poses, occlusion, interactions and background complexity. The instance-agnostic parsing and instance-aware clustering predictions of NAN present high consistency with corresponding ground truths, thanks to the novel network structure and effective training strategy. In contrast, Mask R-CNN  suffers difficulties to differentiate entangled humans, while MH-Parser  struggles to generate fine-grained parsing results and clearly segmented instance masks. This further desmonstrates the effectiveness of the proposed NAN. We also show some failure cases of our NAN in Fig. 7. As can be observed, humans in crowded scenes with heavy occlusion, extreme poses and intensive interactions are difficult to identify and segment. Some small-scale semantic categories within person instances are difficult to parse. This confirms that MHP v2.0 aligns with real-world situations and deserves more furture attention and research efforts.
|Mask R-CNN ||52.68||49.81||51.87|
The MHP v1.0555The dataset is available at http://lv-mhp.github.io/ dataset is the first multi-human parsing benchmark, originally proposed by Li et al. , which contains 4,980 images annotated with 18 semantic labels. Annotation examples are visualized in Fig. 2 (b). The data are randomly organized into 3 splits, consisting of 3,000 training, 1,000 validation and 1,000 testing images with publicly available annotations. Evaluation systems report the and PCP over the testing set. Refer to  for more details.
The performance comparison of the proposed NAN with three state-of-the-art methods in terms of @IoU=0.5, and PCP@IoU=0.5 on the MHP v1.0  testing set is reported in Tab. 4. With the nested adversarial learning of semantic saliency prediction, instance-agnostic parsing and instance-aware clustering, our method outperforms the -best by for , for and for PCP. Visual comparison of multi-human parsing results by NAN and three state-of-the-art methods is provided in Fig. 8, which further validates the advantages of our NAN over existing solutions.
|Li et al. ||40.60||30.40||19.10||38.40|
The PASCAL-Person-Part666The dataset is available at http://www.stat.ucla.edu/~xianjie.chen/pascal_part_dataset/pascal_part.html  dataset is a set of additional annotations for PASCAL-VOC-2010 . It goes beyond the original PASCAL object detection task by providing pixel-wise labels for six human body parts, i.e., head, torso, upper-/lower-arms, and upper-/lower-legs. The rest of each image is considered as background. There are 3,535 images in the PASCAL-Person-Part  dataset, which is split into separate training set containing 1,717 images and testing set containing 1,818 images. For fair comparison, we report the over the testing set for multi-human parsing. Refer to [3, 41] for more details.
The performance comparison of the proposed NAN with two state-of-the-art methods in terms of @IoU= and on the PASCAL-Person-Part  testing set is reported in Tab. 5. Our method dramatically surpasses the -best by for and for . Qualitative multi-human parsing results by NAN are visualized in Fig. 9, which possess a high concordance with corresponding ground truths. This again verifies the effectiveness of our method for human-centric analysis.
|Vineet et al. ||-||-||62.40|
|Jiang et al. ||68.22||69.66||68.94|
The Buffy777The dataset is available at https://www.inf.ethz.ch/personal/ladickyl/Buffy.zip  dataset was released in 2011 for human parsing and instance segmentation, which contains 748 images annotated with 12 semantic labels. The data are randomly organized into 2 splits, consisting of 452 training and 296 testing images with publicly available annotations. For fair comparison, we report the Forward (F) and Backward (B) scores  over the episode 4, 5 and 6 for instance segmentation evaluation. Refer to [39, 21] for more details.
The performance comparison of the proposed NAN with three state-of-the-art methods in terms of F and B scores on the Buffy  dataset episode 4, 5 and 6 is reported in Tab. 6. Our NAN consistently achieves the best performance for all metrics. In particualr, NAN significantly improves the -best by
for F score andfor B score, with an average boost of . Qualitative instance-agnostic parsing and instance-aware clustering results by NAN are visualized in Fig. 10, which well shows the promising potential of our method for fine-grained understanding humans in crowded scenes.
In this work, we presented “Multi-Human Parsing (MHP v2.0)”, a large-scale multi-human parsing dataset and a carefully designed benchmark to spark progress in understanding humans in crowded scenes. MHP v2.0 contains 25,403 images, which are richly labelled with 59 semantic categories. We also proposed a novel deep Nested Adversarial Network (NAN) model to address this challenging problem and performed detailed evaluations of the proposed method with current state-of-the-arts on MHP v2.0 and several other datasets. We envision the proposed MHP v2.0 dataset and the baseline method would drive the human parsing research towards real-world application scenario with simultaneous presence of multiple persons and complex interactions among them. In future, we will continue to take efforts to construct a more comprehensive multi-human parsing benchmark dataset with more images and more detailed semantic category annotations to further push the frontiers of multi-human parsing research.
The work of Jian Zhao was partially supported by China Scholarship Council (CSC) grant 201503170248.
The work of Jiashi Feng was partially supported by NUS startup R-263-000-C08-133, MOE Tier-I R-263-000-C21-112, NUS IDS R-263-000-C67-646 and ECRA R-263-000-C87-133.
Tensorflow: A system for large-scale machine learning.
Multi-task recurrent neural network for immediacy prediction.In ICCV, pages 3352–3360, 2015.
The cityscapes dataset for semantic urban scene understanding.In CVPR, pages 3213–3223, 2016.
Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a.In CVPR, pages 1931–1939, 2015.
Human parsing with contextualized convolutional neural network.In ICCV, pages 1386–1394, 2015.
On spectral clustering: Analysis and an algorithm.In NIPS, pages 849–856, 2002.