Hierarchical feature extractors such as Convolutional Networks (ConvNets) have achieved impressive performance on a variety of classification tasks using purely feedforward processing. Feedforward architectures can learn rich representations of the input space but do not explicitly model dependencies in the output spaces, that are quite structured for tasks such as articulated human pose estimation or object segmentation. Here we propose a framework that expands the expressive power of hierarchical feature extractors to encompass both input and output spaces, by introducing top-down feedback. Instead of directly predicting the outputs in one go, we use a self-correcting model that progressively changes an initial solution by feeding back error predictions, in a process we call Iterative Error Feedback (IEF). IEF shows excellent performance on the task of articulated pose estimation in the challenging MPII and LSP benchmarks, matching the state-of-the-art without requiring ground truth scale annotation.READ FULL TEXT VIEW PDF
Feature extractors such as Convolutional Networks (ConvNets)  represent images using a multi-layered hierarchy of features and are inspired by the structure and functionality of the visual pathway of the human brain [13, 1]. Feature computation in these models is purely feedforward, however, unlike in the human visual system where feedback connections abound [11, 21, 22]
. Feedback can be used to modulate and specialize feature extraction in early layers in order to model temporal and spatial context (e.g.priming ), to leverage prior knowledge about shape for segmentation and 3D perception, or simply for guiding visual attention to image regions relevant for the task under consideration.
Here we are interested in using feedback to build predictors that can naturally handle complex, structured output spaces. We will use as running example the task of 2D human pose estimation [48, 39, 37, 28], where the goal is to infer the 2D locations of a set of keypoints such as wrists, ankles, etc, from a single RGB image. The space of 2D human poses is highly structured because of body part proportions, left-right symmetries, interpenetration constraints, joint limits (e.g. elbows do not bend back) and physical connectivity (e.g. wrists are rigidly related to elbows), among others. Modeling this structure should make it easier to pinpoint the visible keypoints and make it possible to estimate the occluded ones.
Our main contribution is in providing a generic framework for modeling rich structure in both input and output spaces by learning hierarchical feature extractors over their joint space. We achieve this by incorporating top-down feedback – instead of trying to directly predict the target outputs, as in feedforward processing, we predict what is wrong with their current estimate and correct it iteratively. We call our framework Iterative Error Feedback, or IEF.
In IEF, a feedforward model operates on the augmented input space created by concatenating (denoted by ) the RGB image with a visual representation of the estimated output to predict a “correction” () that brings closer to the ground truth output . The correction signal is applied to the current output to generate and this is converted into a visual representation by , that is stacked with the image to produce new inputs for , and so on iteratively. This procedure is initialized with a guess of the output () and is repeated until a predetermined termination criterion is met. The model is trained to produce bounded corrections at each iteration, e.g. . The motivation for modifying by a bounded amount is that the space of is typically highly non-linear and hence local corrections should be easier to learn. The working of our model can be mathematically described by the following equations:
where functions and have additional learned parameters and , respectively. Although we have used the predicted error to additively modify in equation 2, in general can be a result of an arbitrary non-linear function that operates on .
In the running example of human pose estimation,
is vector of retinotopic positions of all keypoints that are individually mapped byinto heatmaps (i.e. heatmaps for keypoints). The heatmaps are stacked together with the image and passed as input to (see figure 1 for an overview). The “rendering” function
in this particular case is not learnt – it is instead modelled as a 2D Gaussian having a fixed standard deviation and centered on the keypoint location. Intuitively, these heatmaps encode the current belief in keypoint locations in the image plane and thus form a natural representation for learning features over the joint space of body configurations and the RGB image.
The dimensionality of inputs to is , where , represent the height and width of the image and correspond to keypoints and the color channels of the image. We model with a ConvNet with parameters (i.e. ConvNet weights). As the ConvNet takes as inputs, it has the ability to learn features over the joint input-output space.
In order to infer the ground truth output (), our method iteratively refines the current output (). At each iteration, predicts a correction () that locally improves the current output. Note that we train the model to predict bounded corrections, but we do not enforce any such constraints at test time. The parameters () of functions and in our model, are learnt by optimizing equation 4,
where, and are predicted and target bounded corrections, respectively. The function is a measure of distance, such as a quadratic loss. is the number of correction steps taken by the model. can either be chosen to be a constant or, more generally, be a function of (i.e. a termination condition).
We optimize this cost function using stochastic gradient descent (SGD) with every correction step being an independent training example. We grow the training set progressively: we start by learning with the samples corresponding to the first step forepochs, then add the samples corresponding to the second step and train another epochs, and so on, such that early steps get optimized longer – they get consolidated.
As we only assume that the ground truth output () is provided at training time, it is unclear what the intermediate targets () should be. The simplest strategy, which we employ, is to predefine for every iteration using a set of fixed corrections starting from , obtaining (). We call our overall learning procedure Fixed Path Consolidation (FPC) which is formally described by algorithm 1.
The target bounded corrections for every iteration are computed using a function , which can take different forms for different problems. If for instance the output is 1D, then sign, would imply that the target “bounded” error will correct by a maximum amount of in the direction of .
Human pose was represented by a set of 2D keypoint locations where K is the number of keypoints and denotes the keypoint. The predicted location of keypoints at the iteration has been denoted by . The rendering of as heatmaps concatenated with the image was provided as inputs to a ConvNet (see section 1 for details). The ConvNet was trained to predict a sequence of “bounded” corrections for each keypoint () . The corrections were used to iteratively refine the keypoint locations.
Let and the corresponding unit vector be . Then, the target “bounded” correction for the iteration and keypoint was calculated as:
where denotes the maximum displacement for each keypoint location. An interesting property of this function is that it is constant while a keypoint is far from the ground truth and varies only in scale when it is closer than to the ground truth. This simplifies the learning problem: given an image and a fixed initial pose, the model just needs to predict a constant direction in which to move keypoints, and to ”slow down” motion in this direction when the keypoint becomes close to the ground truth. See fig. 2 for an illustration.
The target corrections were calculated independently for each keypoint in each example and we used an regression loss to model in eq. 4. We set to pixels in our experiments. We initialized as the median of ground truth 2D keypoint locations on training images and trained a model for steps, using epochs for each new step. We found the fourth step to have little effect on accuracy and used steps in practice at test time.
We employed a standard ConvNet architecture pre-trained on Imagenet: the very deep googlenet 111The VGG-16 network  produced similar results, but required significantly more memory.. We modified the filters in the first convolution layer (conv-1) to account for
additional channels due to 17 keypoints. In our model, the conv-1 filters operated on 20 channel inputs. The weights of the first three conv-1 channels (i.e. the ones corresponding to the image) were initialized using the weights learnt by pre-training on Imagenet. The weights corresponding to the remaining 17 channels were randomly initialized with Gaussian noise of variance 0.1. We discarded the last layer of 1000 units that predicted the Imagenet classes and replaced it with a layer containingunits, encoding the continuous correction 222Again, we do not bound explicitly the correction at test time, instead the network is taught to predict bounded corrections. expressed in Cartesian coordinates (the 17th ”keypoint” is the location of one point anywhere inside a person, marking her, and which is provided as input both during training and testing, see section 3). We used a fixed ConvNet input size of .
We tested our method on the two most challenging benchmarks for 2D human pose estimation: the MPII Human Pose dataset , which features significant scale variation, occlusion, and multiple people interacting, and Leeds Sports Pose dataset (LSP)  which features complex poses of people in sports. For each person in every image, the goal is to predict the 2D locations of all its annotated keypoints.
MPII – Experimental Details. Human pose is represented as a set of 16 keypoints. An additional marking-point in each person is available both for training and testing, located somewhere inside each person’s boundary. We represent this point as an additional channel and stack it with the other 16 keypoint channels and the 3 RGB channels that we feed as input to a ConvNet. We used the same publicly available train/validation splits of . We evaluated the accuracy of our algorithm on the validation set using the standard PCKh metric , and also submitted results for evaluation on the test set once, to obtain the final score.
We cropped square boxes centered on the marking-point of each person, sampled uniformly over scale, from to of the smallest side of the image and resized them to
pixels. Padding was added as necessary for obtaining these dimensions and the amount of training data was further doubled by also mirroring the images. We used the ground truth height of each person at training time, which is provided on MPII, and select as training examples theboxes for each person having a side closest to the person height in pixels. We then trained googlenet models on random crops of patches, using epochs of consolidation for each of steps. At test time, we predict which one of the boxes is closest to the height of the person in pixels, using a shallower model, the VGG-S ConvNet , trained for that task using an regression loss. We then align our model to the center patch of the selected window. The MatConvnet library  was employed for these experiments.
We train our models using keypoint positions for both visible and occluded keypoints, which MPII provides in many cases whenever they project on to the image (the exception are people truncated by the image border). We zero out the backpropagated gradients for missing keypoint annotations. Note that often keypoints lie outside the cropped image passed to the ConvNet, but this poses no issues to our formulation – keypoints outside the image can be predicted and are still visible to the ConvNet as tails of rendered Gaussians.
Comparison with State-of-the-Art. The standard evaluation procedure in the MPII benchmark assumes ground truth scale information is known and images are normalized using this scale information. The current state-of-the-art is the sliding-window approach of Tompson et al  and IEF roughly matches this performance, as shown in table 1. In the more realistic setting of unknown scale information, the best previous result so far is from Tompson et al.  which was the first work to experiment with this setting and obtained PCKh. IEF significantly improves upon this number to . Note however that the emphasis in Tompson et al’s system was efficiency and they trained and tested their model using original image scales – searching over a multiscale image pyramid or using our automatic rescaling procedure should presumably improve their performance. See the MPII website for more detailed results.
|Yang & Ramanan ||73.2||56.2||41.3||32.1||36.2||33.2||34.5||43.2||44.5|
|Pischulin et al ||74.2||49.0||40.8||34.1||36.5||34.4||35.1||41.3||44.0|
|Tompson et al. ||96.1||91.9||83.9||77.8||80.9||72.3||64.8||84.5||82.0|
|Tompson et al. ||83.4||77.5||67.5||59.8||64.6||55.6||46.1||68.3||66.0|
LSP – Experimental Details. In LSP, differently from MPII, images are usually tight around the person whose pose is being estimated, are resized so people have a fixed size, and have lower resolution. There is also no marking point on the torsos so we initialized the 17th keypoints used in MPII to the center of the image. The same set of keypoints is evaluated as in MPII and we trained a model using the same hyper-parameters on the extended LSP training set. We use the standard LSP evaluation code supplied with the MPII dataset and report person-centric PCP scores in table 2. Our results are competitive with the current state-of-the-art of Chen and Yuille .
|Torso||Upper Leg||Lower Leg||Upper Arm||Forearm||Head||Total|
|Pishchulin et al. ||88.9||64.0||58.1||45.5||35.1||85.1||58.0|
|Tompson et al. ||90.3||70.4||61.1||63.0||51.2||83.7||66.6|
|Fan et al. ||95.4||77.7||69.8||62.8||49.1||86.6||70.1|
|Chen and Yuille ||96.0||77.2||72.2||69.7||58.1||85.6||73.6|
In this section, we perform extensive ablation studies to validate four choices of the IEF model: 1) proceeding iteratively instead of in a single shot, 2) predicting bounded corrections instead of directly predicting the target outputs, 3) curriculum learning of our bounded corrections, and 4) modeling the structure in the full output space (all body joints in this case) over carrying out independent predictions for each label.
|Iterative Error Feedback (IEF)||95.2||91.8||80.8||71.5||82.3||73.7||66.4||81.4||81.0|
|Iterative Direct Prediction||91.9||88.5||73.3||59.9||77.5||61.2||51.8||74.0||73.4|
Iterative v/s Direct Prediction. For evaluating the importance of progressing towards solutions iteratively we trained models to directly predict corrections to the keypoint locations in a single shot (i.e. direct prediction). Table 3 shows that IEF that additively regresses to keypoint locations achieves PCKh-0.5 of 81.0 as compared to PCKh of 74.8 achieved by directly regressing to the keypoints.
Iterative Error Feedback v/s Iterative Direct Prediction. Is iterative prediction of the error important or iterative prediction of the target label directly (as in e.g., [45, 41]) performs comparably? In order to answer this question we trained a model from the pretrained googlenet to iteratively predict the ground truth keypoint locations (as opposed to predicting bounded corrections). For comparing performance, we used the same number of iterations for this baseline model and IEF. Table 3 shows that IEF achieves PCKh-0.5 of 81.0 as compared to PCKh of 73.4 by iterative direct prediction. This can be understood by the fact that the learning problem in IEF is much easier. In IEF, for a given image, the model is trained to predict constant corrections except for the last one which is a scaled version. In iterative direct prediction, because each new pose estimate ends up somewhere around the ground truth, the model must learn to adjust directions and magnitudes in all correction steps.
Importance of Fixed Path Consolidation (FPC). The FPC method (see algorithm 1) for training a IEF model makes corrections is a curriculum learning strategy where in the training stage the model is optimized for performing only the first corrections. Is this curriculum learning strategy necessary or can all the corrections be simultaneously trained? For addressing this question we trained an alternative model that trains for all corrections in all epochs. We trained IEF with and without FPC for the same number of SGD iterations and the performance of both these models is illustrated in figure 4. The figure shows that without FPC, the performance drops by almost 10 PCKh points on the validation set and that there is significant drift when performing several correction steps.
Learning Structured Outputs. One of the major merits of IEF is supposedly that it can jointly learn the structure in input images and target outputs. For human pose estimation, IEF models the space of outputs by augmenting the image with additional input channels having gaussian renderings centered around estimated keypoint locations . If it is the case that IEF learns priors over the appropriate relative locations of the various keypoints, then depriving the model of keypoints other than the one being predicted should decrease performance.
In order to evaluate this hypothesis we trained three different IEF models and tested how well each predicted the location of the “Left Knee” keypoint. The first model had only one input channel corresponding to the left knee, the second model had two channels corresponding to left knee and the left hip. The third model was trained using all keypoints in the standard IEF way. The performance of these three models is reported in table 4. As a baseline, regression gets 64.6, whereas the IEF model with a single additional input channel for the left knee gets PCKh of 69.2 This shows that feeding back the current estimate of the left knee keypoint allows for more accurate localization by itself. Furthermore, the IEF model over both left knee and left hip gets PCKh of 72.8. This suggests that the relationship between neighboring outputs has much of the information, but modeling all joints together with the image still wins, obtaining a PCKh of 73.8.
|Direct Prediction of All Joints||IEF Left Knee||IEF Left Knee + Left Hip||IEF All Joints|
|Left Knee PCKh-0.5||64.6||69.2||72.8||73.8|
) but it is a relatively modern topic in conjunction with feature learning, for computer vision[4, 18, 37, 23].
Here we proposed a feedback-based framework for structured-output learning. Neuroscience models of the human brain suggest that feedforward connections act as information carriers while numerous feedback connections act as modulators or competitive inhibitors to aid feature grouping , figure-ground segregation  and object recognition . In computer vision, feedback has been primarily used so far for learning selective attention ; in  attention is implemented by estimating a bounding box in an image for the algorithm to process next, while in  attention is formed by selecting some convolutional features over others (it does not have a spatial dimension).
Stacked inference methods [31, 44, 45, 41] are another related family of methods. Differently, some of these methods consider each output in isolation , all use different weights or learning models in each stage of inference  or they do not optimize for correcting their current estimates but rather attempt to predict the answer from scratch at each stage [24, 41]. In concurrent work, Oberweger et al  proposed a feedback loop for hand pose estimation from kinect data that is closely related to our approach. The autocontext work of  is also related and iteratively computes label heatmaps by concatenating the image with the heatmaps previously predicted. IEF is inspired by this work and we show how this iterative computation can be carried out effectively with deep Convnet architectures, and with bounded error corrections, rather than aiming for the answer from scratch at each iteration.
Another line of work aims to inject class-specific spatial priors using coarse-to-fine processing, e.g. features arising from different layers of ConvNets were recently used for instance segmentation and keypoint prediction . For pose inference, combining multiple scales [10, 37] aids in capturing subtle long-range dependencies (e.g. distinguishing the left and right sides of the body which depend on whether a person is facing the camera). The system in our human pose estimation example can be seen as closest to approaches employing “pose-indexed features” [12, 8, 17], but leveraging hierarchical feature learning. Graphical models can also encode dependencies between outputs and are still popular in many applications, including human pose estimation .
Classic spatial alignment and warping computer vision models, such as snakes,  and Active Appearance Models (AAMs)  have similar goals as the proposed IEF, but are not learned end-to-end – or learned at all – employ linear shape models and hand designed features and require slower gradient computation which often takes many iterations before convergence. They can get stuck in poor local minimas even for constrained variation (AAMs and small out-of-plane face rotations). IEF, on the other hand, is able to minimize over rich articulated human 3D pose variation, starting from a mean shape. Although extensions that use learning to drive the optimization have been proposed , typically these methods still require manually defined energy functions to measure goodness of fit.
|Mean shape||Step 1||Step 2||Step 4||Ground Truth|
While standard ConvNets offer hierarchical representations that can capture the patterns of images at multiple levels of abstraction, the outputs are typically modeled as flat image or pixel-level 1-of-K labels, or slightly more complicated hand-designed representations. We aimed in this paper to mitigate this asymmetry by introducing Iterative Error Feedback (IEF), which extends hierarchical representation learning to output spaces, while leveraging at heart the same machinery. IEF works by, in broad terms, moving the emphasis from the problem of predicting the state of the external world to one of correcting the expectations about it, which is achieved by introducing a simple feedback connection in standard models.
In our pose estimation working example we opted for feeding pose information only into the first layer of the ConvNet for the sake of simplicity. This information may also be helpful for mid-level layers, so as to modulate not only edge detection, but also processes such as junction detection or contour completion which advanced feature extractors may need to compute. We also have only experimented so far feeding back ”images” made up of Gaussian distributions. There may be more powerful ways to render top-down pose information using parametrized computational blocks (e.g. deconvolution) that can then be learned jointly with the rest of the model parameters using standard backpropagation. This is desirable in order to attack problems with higher-dimensional output spaces such as 3D human pose estimation[32, 33] or segmentation.
This work was supported in part by ONR MURI N00014-14-1-0671 and N00014-10-1-0933. João Carreira was partially supported by the Portuguese Science Foundation, FCT, under grant SFRH/BPD/84194/2012. Pulkit Agrawal was partially supported by a Fulbright Science and Technology Fellowship. We gratefully acknowledge NVIDIA corporation for the donation of Tesla GPUs for this research. We thank Georgia Gkioxari and Carl Doersch for helpful comments.
Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 3686–3693. IEEE, 2014.
Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation.June 2015.