Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.
Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency.However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on three datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance.READ FULL TEXT VIEW PDF
Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.
Faithfully recovering the 3D shapes of human faces from unconstrained 2D images is a challenging task and has numerous applications such as face recognition[6, 51, 59], face media manipulation [5, 50], and face animation [10, 23]. Recently, there is a surge of interest in 3D face reconstruction from a single image using deep Convolutional Neutral Networks (CNN) in lieu of the complex and costly optimization used by traditional methods [37, 13, 38, 51, 25, 45, 49, 48, 53, 46, 14, 18]. Since ground truth 3D face data is scarce, many previous approaches resort to synthetic data or using 3D shapes fitted by traditional methods as surrogate shape labels [37, 57, 45, 31, 14, 18]. However, their accuracy may be jeopardized by the domain gap issue or the imperfect training labels.
To circumvent these issues, methods have been proposed to train networks without shape labels in an unsupervised or weakly-supervised fashion and promising results have been obtained [49, 48, 53, 46, 16]
. The crux of unsupervised learning is a differentiable image formation procedure which renders a face image with the network predictions, and the supervision signal stems from the discrepancy between the input image and the rendered counterpart. For example, Tewariet al.  and Sengupta et al.  use pixel-wise photometric difference as training loss. To improve robustness and expressiveness, Tewari et al.  proposed a two-step reconstruction scheme where the second step produces a shape and texture correction with a neural network. Genova et al. 
proposed to measure face image discrepancy on the perception level by using the distances between deep features extracted from a face recognition network.
Our goal in this paper is to obtain accurate 3D face reconstruction with weakly-supervised learning. We identified that using low-level information of pixel-wise color alone may suffer from local minimum issue where low error can be obtained with unsatisfactory face shapes. On the other hand, using only perceptual loss also lead to sub-optimal results since it ignores the pixel-wise consistency with raw image signal. In light of this, we propose a hybrid-level loss function which integrates both of them, giving rise to accurate results. We also propose a novel skin color based photometric error attention strategy, granting our method further robustness to occlusion and other challenging appearance variations such as beard and heavy make-up. We train an off-the-shelf deep CNN to predict 3D Morphable Model (3DMM)  coefficients, and accurate reconstruction is achieved on multiple datasets [1, 11, 56].
With a strong CNN model for single-image 3D face reconstruction, we take a further step and consider the problem of CNN-based face reconstruction aggregation with a set of images. Given multiple face images of a subject (e.g., from a personal album) captured in the wild under disparate conditions, it is natural to leverage all the images to build a better 3D face shape. To apply the deep neural networks on arbitrary number of images, one solution would be aggregating the single-image reconstruction results, and perhaps the simplest strategy is naively averaging the recovered shapes. However, such a native strategy did not consider the quality of the input images (e.g., if some samples contain severe occlusion). Nor does it take full advantage of pose differences to improve the shape prediction.
In this paper, we propose to learn 3D face aggregation from multiple images, also in an unsupervised fashion. We train a simple auxiliary network to produce “confidence scores” of the regressed identity-bearing 3D model coefficients, and obtain final identity coefficients via confidence-based aggregation. Despite no explicit confidence label is used, our method automatically learns to favor high-quality (especially high-visibility) photos. Moreover, it can exploit pose difference to better fuse the complementary information, learning to more accurate 3D shapes.
To summarize, this paper makes the following two main contributions:
We propose a CNN-based single-image face reconstruction method which exploits hybrid-level image information for weakly-supervised learning. Our loss consists of a robustified image-level loss and a perception-level loss. We demonstrate the benefit of combing them, and show the state-of-the-art accuracy of our method on multiple datasets [1, 11, 56], significantly outperforming previous methods trained in a fully supervised fashion [45, 14, 51]. Moreover, we show that with a low-dimensional 3DMM subspace, we are still able to outperform prior art with “unrestricted” 3D representations [45, 53, 48, 14] by an appreciable margin.
We propose a novel shape confidence learning scheme for multi-image face reconstruction aggregation. Our confidence prediction subnet is also trained in a weakly-supervised fashion without ground-truth label. We show that our method clearly outperforms naive aggregation (e.g
., shape averaging) and some heuristic strategies. To our knowledge, this is the first attempt towards CNN-based 3D face reconstruction and aggregation from an unconstrained image set.
3D face reconstruction has been a longstanding task in computer vision and computer graphics. In the literature, 3D Morphable Models (3DMM)[5, 33, 7] have played a paramount role for 3D face modelling. With a 3DMM, reconstruction can be performed by an analysis-by-synthesis scheme using image intensity  and other features such as edges . More recently, model fitting using facial landmarks has gained much popularity with the growth of face alignment techniques [4, 58, 21, 3]
. However, sparse landmarks cannot well capture the dense facial geometry. Beyond 3DMM, another popular 3D face model is the multilinear tensor model[54, 10, 9, 43]. A few model-free single-image reconstruction methods have been proposed [20, 27, 19]; most require some reference 3D face shapes. For example, [20, 19]estimate image depth by building correspondences between the input image and one or a set of reference 3D faces. In , a shape-from-shading approach is proposed with a reference 3D face as prior.
The aforementioned approaches usually involve costly optimization to recover a quality 3D face. Recently, numerous method are proposed which employ CNNs for efficient face reconstruction. Some of them apply CNNs to regress 3DMM coefficients [37, 13, 2, 49, 51, 16], some use multi-step schemes to add correction or details onto coarse 3DMM predictions [38, 48, 52, 18], while others advocate direct model-free reconstruction [45, 53, 46, 14].
For all these CNN-based methods, one great hurdle is the lack of training data. Many methods resort to synthetic data or using 3D shapes fitted by traditional methods as surrogate labels [37, 57, 45, 31, 14, 18]. Others have attempted unsupervised or weakly-supervised training [49, 48, 53, 46, 16]. Our method is also based on weakly-supervised learning, for which our findings in this paper are threefold: 1) the loss function is important for weakly-supervised learning and both low-level and perception-level information should be leveraged; 2) the results obtained with weak supervision can be significantly better than those trained with synthetic data or pseudo-ground truth shapes, and 3) somewhat surprisingly, the results confined in the low-dimensional 3DMM subspace can still be much better than state-of-the-art results with “unrestricted” representations.
We also studied the problem of reconstruction aggregation from multiple images. One related work is  which investigated the reconstruction quality measurement closest to human ratings and used it to fuse the reconstructions obtained with 3DMM fitting. We however show that their method is deficient in our case. Our method is also related to traditional methods working on unconstrained photo collections [27, 47, 40, 41]. While excellent results have been obtained by these methods, they typically consist of multiple steps such as face frontalization, photometric stereo, and local normal refinement. The whole pipeline is complex and may break down under severe occlusion and extreme pose. Our goal in this paper is not to replace these traditional methods, but to study the shape aggregation problem (similar to ) with a CNN and provide an extremely fast and robust alternative learned end-to-end.
As shown in Fig. 1 (a), we use a CNN to regress coefficients of a 3DMM face model. For unsupervised/weakly-supervised training [49, 48], we also regress the illumination and face pose to enable analytic image regeneration. We detail our models and CNN outputs as follows.
3D Face Model. With a 3DMM, the face shape and the texture can be represented by an affine model:
where and are the average face shape and texture; , , and
are the PCA bases of identity, expression, and texture respectively, which are all scaled with standard deviations;, , and
are the corresponding coefficient vectors for generating a 3D face. We adopt the popular 2009 Basel Face Model for , , , and , and use the expression bases of  which are built from FaceWarehouse . A subset of the bases is selected, resulting in , and . We exclude the ear and neck region, and our final model contains 36K vertices.
Illumination Model. We assume a Lambertian surface for face and approximate the scene illumination with Spherical Harmonics (SH) [35, 36]. The radiosity of a vertex with surface normal and skin texture can then be computed as where are SH basis functions and are the corresponding SH coefficients. We choose bands following [49, 48] and assume monochromatic lights such that .
Camera Model. We use the perspective camera model with an empirically-selected focal length for the 3D-2D projection geometry. The 3D face pose is represented by rotation and translation .
In summary, the unknowns to be predicted can be represented by a vector . In this paper, we use a ResNet-50 network 
to regress these coefficients by modifying the last fully-connected layer to 239 neurons. For brevity, we denote this modified ResNet-50 network for single image reconstruction as R-Net. We present how we train it in the following section.
Given a training RGB image , we use R-Net to regress a coefficient vector , with which a reconstructed image can be analytically generated with some simple, differentiable math derivations (details are deferred to the suppl. material). Some examples of can be found in Fig. 1. Our R-Net is trained without any ground truth labels, but via evaluating a hybrid-level loss on
and backpropagate it.
We first introduce our loss functions on low-level information including per-pixel color and sparse 2D landmarks.
First, it is straightforward to measure the dense photometric discrepancy between the raw image and the reconstructed one [5, 50, 49, 48]. In this paper, we propose a robust, skin-aware photometric loss instead of a naive one, defined as:
where denotes pixel index, is reprojected face region which can be readily obtained, denotes the norm, and is a skin color based attention mask for the training image which is described as follows.
To gain robustness to occlusions and other challenging appearance variations such as beard and heavy make-up, we compute a skin-color probability26] (details can be found in the suppl. material). For each pixel , we set . We find that such a simple skin-aware loss function works remarkably well in practice without the need for a face segmentation method . Figure 2 illustrates the benefit of using our skin attention mask.
We also use landmark locations on the 2D image domain as weak supervision to train the network. We run the state-of-the art 3D face alignment method of  to detect 68 landmarks of the training images. During training, we project the 3D landmark vertices of our reconstructed shape onto the image obtaining , and compute the loss as:
where is the landmark weight which we experimentally set to for inner mouth and nose points and others to .
While using the low-level information to measure image discrepancy can generally yields decent results [5, 50, 49, 48], we find using them alone can lead to local minimum issue for CNN-based 3D face reconstruction. Figure 3 shows that our R-net trained with only image-level losses generates smoother textures and lower photometric errors than the compared opponents, but the resultant 3D shapes are less accurate by visual inspection.
To tackle this issue, we introduce a perception-level loss to further guide the training. Inspired by , we seek for the weak-supervision signal from a pre-trained deep face recognition network. Specifically, we extract the deep features of the images and compute the cosine distance:
where denotes deep feature encoding and vector inner product. In this work, we train a FaceNet  structure using an in-house face recognition dataset with 3M face images of 50K identities crawled from the Internet, and use it as our deep feature extractor.
Figure 3 shows that with the perceptual loss, the textures are sharper and the shapes are more faithful. Quantitative results in the experiment section also show the benefit.
To prevent face shape and texture degeneration, we add a commonly-used loss on the regressed 3DMM coefficients:
which enforces a prior distribution towards the mean face. The balancing weights are empirically set to , and .
Although the face textures in the Basel 2009 3DMM  were obtained with special devices, they still contain some baked-in shading (e.g., ambient occlusion). To favor a constant skin albedo similar to 
, we add a flattening constrain to penalize the texture map variance:
where is a pre-defined skin region covering cheek, noise, and forehead (see suppl. material for more details).
In summary, our loss function for R-Net is composed of two image-level losses, a perceptual loss and two regularization loss. Their weights are set to and respectively in all our experiments.
Given multiple face images of a subject (e.g., a photo album), it is natural to leverage all the images to build a better 3D face shape. Images captured under different conditions should contain information complementary to each other due to change of pose, lighting etc. Moreover, using an image set for reconstruction can gain further robustness to occlusion and bad lighting in some individual images.
Applying deep neural networks on an arbitrary number of orderless images is not straightforward. In this work, we use a network to learn a measurement of confidence or quality of the single-image reconstruction results, and use it to aggregate the individual shapes. Specifically, we seek to generate a vector with positive elements measuring the confidence of the identity-bearing shape coefficients . We do not consider other coefficients such as expression, pose, and lighting as they vary across images and fusion is unnecessary. We also bypass texture as we found the skin color of a subject can vary significantly across in-the-wild images. Let be an image collection of a person, the output coefficient vector from R-Net for each image , and the confidence vector for each , we obtain the final shape via element-wise shape coefficient aggregation:
where and denote Hadamard product and division, respectively.
Next, we present how we train a network, denoted as C-Net, to predict in a weakly-supervised fashion without labels. The structure of C-Net will be presented afterwards.
To train C-Net on image sets, we generate the reconstructed image set of with , where . We define the training loss as
where is our hybrid-level loss function defined in Section 4 evaluated with of .
This way, the error can be backpropagated to thus further to and C-Net weights, since Eq. 7 is differentiable. C-Net will be trained to produce confidences that lead to an aggregated 3D face shape consistent with the face image set as much as possible. The pipeline is illustrated in Fig. 1(c). In the multi-image training stage, the loss weights and are set to , and respectively.
Our aggregation design and training scheme are inspired by the set-based face recognition work of . However,  used a scalar quality score for feature vector aggregation, whereas we produce element-wise scores for 3DMM coefficients. In Section 6.2.1, we show element-wise scores yield superior results and analyze how our network exploits face pose difference for better shape aggregation.
Our C-Net is designed to be light-weight. Since R-Net is able to predict high-level information such as pose and lighting, it is natural to reuse its feature maps for C-Net. In practice, we take both shallow and deep features from R-Net, as illustrated in Fig. 1 (a). The shallow feature can be used to measure image corruptions such as occlusion.
Specifically, we take the features after the first residual block and after global pooling of R-Net as the input to C-Net. We apply three convolution layers
channels and stride, followed by a global pooling layer on to get . We then concatenate and
, and apply two fully-connected layers with 512 and 80 neurons respectively. At last, we apply sigmoid function to make the confidence predictionspositive. Our C-Net has M parameters in total, which is about size of R-Net.
Implementation Details. To train our R-Net, we collected in-the-wild images from multiple sources such as CelebA , 300W-LP , I-JBA , LFW  and LS3D . We balanced the pose and race distributions and get 260K face images as our training set. We use the method of  to detect and align the images. The input image size is 224
224. We take the weights pre-trained in ImageNet as initialization, and train R-Net using Adam optimizer  with batch size of , initial learning rate of , and K total iterations.
To train C-Net, we construct an image corpus using 300W-LP , Multi-PIE  and part of our in-house face recognition dataset. For 300W-LP and Multi-PIE, we choose 5 images with rotation angles evenly distributed for each person. For the face recognition dataset, we randomly select 5 images for each person. The whole training set contains 50K images of 10K identities. We freeze the trained R-Net, and randomly initialize C-Net except for its last fully-connected layer which is initialized to zero (so that we start from average pooling). We train it using Adam  with batch size of , initial learning rate of and K total iterations.
To validate the efficacy of our proposed hybrid-level loss function, we conduct ablation study on two datasets: the MICC Florence 3D Face dataset  and the FaceWarehouse dataset . MICC contains 53 subjects, each associated with a ground truth scan in neutral expression and three video sequences captured in cooperative, indoor, and outdoor scenarios. For FaceWarehouse, we use 9 subjects each with 20 expressions for evaluation.
Table 1 presents the reconstruction errors with various loss combinations. It shows that jointly considering image- and perception-level information gives rise to significantly higher accuracy than using them separately.
|Tran et al. ||1.970.49||2.030.45||1.930.49|
|Genova et al. ||1.780.54||1.780.52||1.760.54|
Comparison on MICC Florence with [51, 16, 14, 25, 57, 31]. We first compare with the methods of Tran et al.  and Genova et al. . For  and ours, we evaluate the error with the average shape from a sequence.  averaged their encoder embeddings from all frames before reconstruction and produce a single shape per sequence. Following , we crop the ground truth mesh to 95mm around the nose tip and run ICP with isotropic scale for alignment. The results of  only contains part of the forehead region, thus we further cut the ground truth meshes accordingly for fair comparison. Table 2 shows that our method significantly outperforms  and  on all three sequences. The qualitative comparison in Fig. 4 and Fig. 5 also demonstrates the superiority of our results. Note that  uses a perceptual loss similar to ours, but they ignores the low-level information such as photometric similarity.
We then compare with PRN , a recent CNN method with supervised learning that predicts unrestricted face shapes. Following , we render face images with 20 poses for each subject using pitch angles of , , and degrees and yaw angles of , , , , degrees. Figure 6 shows the point-to-plane RMSE averaged across subjects and pitch angles. Our method has a much lower error than PRN for all yaw angles. Also note that PRN has a larger model size than ours (160MB vs. 92MB).
We further qualitatively compare with several learning-based methods including VRN , 3DDFA , and Liu et al. . Figure 5 shows that our method can well-recover both identity and expression, whereas the results of other methods have very low shape variance.
Comparison on Facewarehouse with [48, 49, 28, 15]. We compare our results on the 9 Facewarehouse subjects selected by , with three learning-based approaches of Tewari et al. [49, 48], Kim et al.  and an optimization-based approach of Garrido et al. . The evaluation protocol of  is used (see suppl. materal for details).
We evaluate two face regions: a smaller
one same as ’s, and a larger one with more cheek areas included (see Fig. 7). The point-to-point errors are presented in Table 3. Our method achieved the lowest reconstruction error among all learning-based methods. Note that , -C (coarse results), , and our method are all based on 3DMM representation, and we show significant improvement upon theirs. Our method is even better than -F which uses a corrective space to refine the 3DMM shape. Our accuracy gets closer to the optimization-based approach of  while our method can be orders of magnitude faster.
We further compare with  qualitatively in Fig 8. Our recovered shapes are of higher fidelity. Moreover, some artifacts from  can be observed under occlusion while our results are much more pleasing. Also note that our method can handle profile faces (see, e.g., Fig. 9), while the large-pose robustness of the above methods are unclear to us.
|Ours||PRN ||Sela et al. |
Comparison on BU-3DFE with [45, 14]. The BU-3DFE dataset  contains 100 subjects across different races and each with different expressions. Here we use the frontal images of 100 neutral faces for our evaluation. Again, we perform ICP alignment and compute point-to-plane distance. Table 4 shows that our method significantly outperforms  and . The BU-3DFE images are captured under controlled settings. Figure 9 further shows that our method also outperforms  under large pose and challenging appearance variance of in-the-wild images.
To test our multi-image shape aggregation method, we first conduct ablation study on render images of MICC. We render 20 poses for each of the 53 subjects as in Sec. 6.1.2. Table 5 presents the shape error of different aggregation strategies (S1 to S5). For S1, we train a C-Net similar to that described in Sec. 5.1 but modify the final FC layer to output a global confidence score , and we aggregate the identity coefficients via . For S2, we sum all elements in the confidence vector to get a global confidence for aggregation. For S3, we simply choose a single shape with largest confidence vector summation for error computation. For S4, we use our element-wise coefficient aggregation described in Sec. 5.1.
Table 5 shows that all shape aggregation methods including the naive shape averaging have a lower error than the mean of per-frame shape errors. Nevertheless, all our aggregation strategies yield better results than naive shape averaging, demonstrating the efficacy of our learning-based aggregation method. Among them, the element-wise coefficient aggregation (S4) performs best.
|Shape error mean||1.970.70|
|Our S1: Global Aggr. with c||1.710.56|
|Our S2: Global Aggr. with||1.700.55|
|Our S3: Max Conf.||1.710.50|
|Our S4: Elementwise Aggr. with||1.670.54|
Analysis. We further analyze confidence score statistics to see whether they are affected by face pose. We compute the average relative confidence scores for profile and frontal images respectively. As shown in Fig 11 (left), for profile faces the confidence is lower in general, but higher on a few dimensions. Figure 11 (right) further shows that the coefficient entries having larger influence on face depth (-direction in the our 3D face coordinate system) tend to get relatively larger confidence scores on profile faces than on frontal ones. This is consistent with our intuition, and suggests that with element-wise confidences, the network can exploit view difference for better reconstruction. Due to space constraint, more details about this experiment are deferred to the suppl. material.
Figure 12 presents some examples of our confidence prediction on our test set (to ease presentation we show the confidence vector summation for each image). Our C-Net generally favors quality face images with frontal pose, high visibility, natural lighting etc. Occlusions like sun-glasses, hat and hair decrease the confidence.
Multi-image reconstruction error on the MICC dataset. We use same inputs and evaluation metric as in Table2. “-G” and “-S” denote global and segment-based aggregation of our predicted shapes using the strategy of .
To our knowledge, our method is the first one applying neutral networks for face reconstruction confidence prediction and aggregation. So here we compare with a heuristic strategy of Piotraschke and Blanz . Table 6 shows that our method produced better results than shape averaging and  on the MICC dataset (we treat a sequence as an image set). The method of  underperformed. Its results are even slightly worse than shape averaging. We conjecture this is because  rely on the surface normal discrepancy with mean face to eliminate deficient reconstructions, yet our R-Net always produces a smooth, plausible face shape which renders their quality measurement ineffective.
We have proposed a CNN-based single-image face reconstruction method which exploits hybrid-level image information for weakly-supervised learning without ground-truth 3D shapes. Comprehensive experiments have shown that our method outperforms previous methods by a large margin in terms of both accuracy and robustness. We have also proposed a novel multi-image face reconstruction aggregation method using CNNs. Without any explicit label, our method can learn to measure image quality and exploit the complementary information in different images to reconstruct 3D faces more accurately.
3d morphable models as spatial transformer networks.In International Conference on Computer Vision Workshop on Geometry Meets Deep Learning, pages 904–912, 2017.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5543–5552, 2016.
Unrestricted facial geometry reconstruction using image-to-image translation.In IEEE International Conference on Computer Vision (ICCV), pages 1585–1594, 2017.
MoFa: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction.In International Conference on Computer Vision (ICCV), pages 1274–1283, 2017.