A comprehensive 2D human body representation should capture both human pose and shape information. Such representation is promising for applications beyond plain keypoint localization, such as graphics and human-computer interaction. However, how to establish such 2D body representation is still an open problem. Current mainstream 2D human body representations are not able to simultaneously capture both information. Skeleton keypoint based representation [andriluka20142d, johnson2010clustered, lin2014microsoft] well captures human poses. However, such representation loses the 2D human shape information which is essential for human body understanding. Pixel-wise human parsing representations [liang2015human, chen2014detect, gong2017look] contain 2D human shape cues. However, such kinds of representations lack accurate keypoint localization information, since all pixels in one part share the same semantic label. Meanwhile, they are inflexible to manipulate and costly to label. This paper aims at discovering a new representation for more comprehensive understanding of the human body. To this end, a novel Triplet Representation for Body (TRB) is introduced. It consists of skeleton and contour keypoint representations, capturing both accurate pose localization and rich semantic human shape information simultaneously, while preserving its flexibility and simplicity.
Since there exists no dataset to quantitatively evaluate TRB estimation, we propose several challenging TRB datasets based on three pose estimation datasets (LSP[johnson2010clustered], MPII [andriluka20142d] and COCO [lin2014microsoft]). We quantitatively evaluate the performance of several state-of-the-art 2D skeleton-based keypoint detectors on the proposed TRB datasets. Our experiments indicate that they are not able to effectively solve the more challenging TRB estimation tasks, which require the approaches to not only understand the concept of human pose and human shape simultaneously, but also exploit the underlying relationship between them.
For effective representation learning, we design a two-branch multi-task framework called TRB-Net, which jointly solves skeleton keypoint estimation and contour keypoint estimation. These two tasks are closely related and will promote each other. Therefore, we design a message passing block to enable information exchange. The message received from the other branch will act as guidance for the current branch to produce finer estimation results. Since feature maps from the two branches have different patterns, spatial feature transformation is necessary for feature alignment and more effective message passing scheme. Therefore, we propose a task-specific directional convolution operator to exploit the inside-out and outside-in spatial relationship between skeleton and contour feature maps. To prevent inconsistent predictions for skeleton and contour branch, we explicitly enforce pairwise mapping constraints. With these techniques, we boost the TRB estimation performance beyond state-of-the-arts. Error is reduced by 13.3% and 15.1% for skeleton and contour keypoint estimation respectively (Sec. LABEL:ablation_part).
Our contributions are three-fold: We propose the novel Triplet Representation for Body (TRB), which embodies both human pose and shape information. We apply TRB to the conditional image generation task, and show its effectiveness in handling pose/shape guided image generation and human shape editing. We introduce a challenging TRB estimation task, establish a benchmark and evaluate various mainstream pose estimation approaches in the context of TRB estimation. We design the TRB-net which jointly learns the skeleton and contour keypoint representation. Three techniques are proposed for effective message passing and feature learning. Extensive experiments show the effectiveness of our proposed methods.