Self-supervised learning of a facial attribute embedding from video

08/21/2018 ∙ by Olivia Wiles, et al. ∙ 4

We propose a self-supervised framework for learning facial attributes by simply watching videos of a human face speaking, laughing, and moving over time. To perform this task, we introduce a network, Facial Attributes-Net (FAb-Net), that is trained to embed multiple frames from the same video face-track into a common low-dimensional space. With this approach, we make three contributions: first, we show that the network can leverage information from multiple source frames by predicting confidence/attention masks for each frame; second, we demonstrate that using a curriculum learning regime improves the learned embedding; finally, we demonstrate that the network learns a meaningful face embedding that encodes information about head pose, facial landmarks and facial expression, i.e. facial attributes, without having been supervised with any labelled data. We are comparable or superior to state-of-the-art self-supervised methods on these tasks and approach the performance of supervised methods.



There are no comments yet.


page 2

page 7

page 9

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Babies and children are highly perceptive to the facial expressions of the people they interact with [Gerull and Rapee(2002), Ekman and Oster(1979)]

. The ability to understand and respond to changes in people’s emotional state is similarly important for computer vision systems and affective systems when interacting with a human user. Thus being able to predict head pose and expression is of vital importance.

Recently, leveraging deep learning has led to state-of-the-art results on a variety of tasks such as emotion recognition and facial landmarks detection. Despite these advances, supervised methods require large amounts of labelled data which may be expensive or difficult to obtain in realistic, unconstrained settings, or necessitate assigning data to ill-defined categories. For example, categorising emotions with three human annotators leads to only 46% agreement 

[Barsoum et al.(2016)Barsoum, Zhang, Ferrer, and Zhang], and labelling pose in the wild is notoriously difficult. Moreover, performing each task independently does not leverage the fact that detecting landmarks requires understanding pose and facial features, which in turn correspond to expression.

Figure 1: An overview of FAb-Net. In the single-source case (top), the encoder-decoder architecture takes one source frame and a target frame as inputs and learns to generate the target frame. The -dimensional outputs of the encoders – the source and target attribute embeddings – are concatenated and input to the decoder. The decoder predicts the point-to-point correspondence from the source frame to the target frame, and RGB values for the generated frame are then obtained from the source frame using a bilinear sampler. The network is trained with an loss between the generated frame and the target frame. In the multi-source case (bottom), the decoder also predicts a confidence map, and the confidence maps are used to weight the contributions of the different source frames.

Consequently, we consider the following question: is it possible to learn an embedding of facial attributes that encodes landmarks, pose, emotion, etc. in a self-supervised manner without any hand labelling? The learned embedding can then be used for another task (e.g. landmark, pose, and expression prediction) using a linear layer. To do this, we contribute FAb-Net, a self-supervised framework for learning a low-dimensional face embedding for facial attributes (Section 3). We take advantage of video data which contains a large collection of images of the same person from different viewpoints and with varied expressions. Given only the embeddings corresponding to a source and target frame, the network is tasked to map the source frame to the target frame by predicting a flow field between them. This proxy task forces the network to distil the information required to compute the flow field (e.g. the head pose and expression) into the source and target embeddings.

After explaining the setup for a single source frame in Section 3.1, we introduce our additional contributions: a method for leveraging multiple frames in order to improve the learned embedding in Section 3.2; and how a curriculum strategy for training FAb-Net in Section 4 can be used to improve performance.

The learned embedding is extracted and used for a variety of tasks such as landmark detection, pose regression, and expression classification in Section 5 by simply learning a linear layer. Our results on these tasks are comparable or superior to other self-supervised methods, approaching the performance of supervised methods. These experiments verify the hypothesis that our self-supervised framework learns to encode facial attributes which are useful for a variety of tasks. Finally, the method is tested qualitatively by using the learned embedding to retrieve images with similar facial attributes across different identities.

2 Related Work

Self-supervised learning. Self-supervised methods such as [Pathak et al.(2016)Pathak, Krähenbühl, Donahue, Darrell, and Efros, Zhang et al.(2016a)Zhang, Isola, and Efros, Doersch et al.(2015)Doersch, Gupta, and Efros, Noroozi and Favaro(2016)] require no manual labelling of the images; instead, they directly use image data to provide proxy supervision for learning good feature representations. The benefit of these approaches is that the features learned using large quantities of available data can be transferred to other tasks/domains which have less or even no annotated data.

To provide further supervision from image data, the images themselves can be transformed via a synthetic warp or rotation and the network trained to recognise the rotation [Gidaris et al.(2018)Gidaris, Singh, and Komodakis] or to learn equivariant pixel embeddings [Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi, Thewlis et al.(2017b)Thewlis, Bilen, and Vedaldi, Novotny et al.(2018)Novotny, Albanie, Larlus, and Vedaldi].

Our approach builds in particular on those that use frame synthesis [Xue et al.(2016)Xue, Wu, Bouman, and Freeman, Jia et al.(2016)Jia, De Brabandere, Tuytelaars, and Gool, Pătrăucean et al.(2016)Pătrăucean, Handa, and Cipolla, Denton and Birodkar(2017), Chung et al.(2017)Chung, Jamaludin, and Zisserman, Wiles et al.(2018)Wiles, Koepke, and Zisserman], though for us synthesis is a proxy task rather than the end goal. Note, unlike [Wang and Gupta(2015), Misra et al.(2016)Misra, Zitnick, and Hebert, Fernando et al.(2017)Fernando, Bilen, Gavves, and Gould, Lee et al.(2017)Lee, Huang, Singh, and Yang], we do not make use of the temporal ordering information inherent in a video; nor do we predict future frames conditioned on a number of past frames [Pătrăucean et al.(2016)Pătrăucean, Handa, and Cipolla], or explicitly predict the motion between frames as a convolutional kernel [Xue et al.(2016)Xue, Wu, Bouman, and Freeman, Jia et al.(2016)Jia, De Brabandere, Tuytelaars, and Gool], or condition the generation on another modality (e.g. voice [Chung et al.(2017)Chung, Jamaludin, and Zisserman]). Instead, we treat the frames as an unordered set, and propose a simple formulation: that by embedding the source and target frames in a common space and conditioning the transformation from source to target frame on these embeddings, the learned embeddings must learn to encode the relevant modes of variation necessary for the transformation.

Concurrent to our work, Zhang et al [Zhang et al.(2018)Zhang, Guo, Jin, Luo, He, and Lee] and Jakab et al [Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi] build on [Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi] by using the discovered landmarks to reconstruct the original image. However, unlike these works, we do not place any constraints on the learned representation – such as an explicit representation that encodes landmarks as heatmaps. Supervised learning of face embeddings. Given known (labelled) attribute information, e.g. for pose or expression, the embedding can be learned by training in a supervised manner to directly predict the attribute [Kumar et al.(2011)Kumar, Berg, Belhumeur, and Nayar, Rudd et al.(2016)Rudd, Günther, and Boult, Liu et al.(2015)Liu, Luo, Wang, and Tang], or to generate images (of faces, cars, or other classes) at a new, known pose, expression, etc. [Tran et al.(2017)Tran, Yin, and Liu, Yang et al.(2017)Yang, Ren, Chen, Wen, Li, and Hua, Dosovitskiy et al.(2015)Dosovitskiy, Springenberg, and Brox, Kulkarni et al.(2015)Kulkarni, Whitney, Kohli, and Tenenbaum, Zhou et al.(2016)Zhou, Tulsiani, Sun, Malik, and Efros].

Another way of supervising a face embedding is to explicitly learn the parameters of a 3D morphable model (3DMM) [Blanz et al.(2002)Blanz, Romdhani, and Vetter]. As fitting a 3DMM is relatively expensive, [Bas et al.(2017)Bas, Huber, Smith, Awais, and Kittler, Tewari et al.(2017)Tewari, Zollhöfer, Kim, Garrido, Bernard, Perez, and Theobalt] learn this end-to-end using either landmarks or a photometric error as supervision. However, unlike our method, these methods require either ground truth labels or a morphable model which fixes the modes of variation and the embedding.

An interesting half-way point is weak supervision, where the learned object or face embedding is conditioned for instance on object labels [Novotny et al.(2017)Novotny, Larlus, and Vedaldi] or weather/geo-location information [Li et al.(2015)Li, Wang, Liu, Jiang, Shan, and Chen] respectively. This requires additional meta-data, but results in embeddings that can represent attributes such as age and expression for faces or keypoints for objects.

3 Method

The aim is to train a network to learn an embedding that encodes facial attributes in a self-supervised manner, without any labels. To do this, the network is trained to generate a target frame from one or multiple source frames by learning how to transform the source into the target frame. The source and target frames are taken from the same face-track of a person speaking, i.e. the frames are of the same identity but with different expressions/poses. An overview of the architecture is given in Fig. 1 and further described for a single source frame in Section 3.1 and for multiple source frames in Section 3.2 (additional details are given in the supp. material).

3.1 Single-source frame architecture

The input to the network is a source frame and a target frame from the same face-track. These are passed through encoders with shared weights which learn a mapping from the input frames to a

-dimensional vector embedding (as shown in Fig. 

1). The embeddings corresponding to the target and source frames are and respectively. The source and target embeddings are concatenated to give a -dimensional vector which is upsampled via a decoder. The decoder learns a mapping from the concatenated embeddings to a bilinear grid sampler, which samples from the source frame to create a new, generated frame . Precisely, predicts offsets for each pixel location in the target frame; the generated frame at location is obtained by sampling from the source frame according to these offsets : . The network is trained to minimise the loss between the generated and the target frame: .

This setup enforces that the embeddings and represent facial attributes of the source and target frames respectively since the decoder maps from the source frame to generate the frame (i.e. it uses pixel RGB values from the source frame to create the generated – a similar formulation has been proposed concurrently to this work by [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy]).

As the decoder is a function of the target and source attribute embeddings, and the decoder is the only place in the network where information is shared, the target attribute embedding must encode information about expression and pose in order for the decoder to know where to sample from in the source frame and where to place this information in the generated frame.

3.2 Multi-source frames architecture

While using two frames for training enforces that the network learns a high-quality embedding, additional source frames can be leveraged to improve the learned embedding. This is achieved by also predicting a confidence heatmap – a channel image – for each source frame via an additional decoder. The heatmaps denote how confident the network is of the flow at each pixel location – e.g. if the source frame has a very different pose than the target frame, the confidence heatmap would have low certainty. Moreover, it can express this for sub-parts of the image; if the mouth is closed in the source but open in the target frame, the confidence heatmap can express uncertainty in this region. The confidence heatmaps are combined pixel-wise for each source frame using a soft-max operation. For

source frames, the loss function to be minimised is given as


4 Curriculum Strategy

The training of the network is divided into stages, so that knowledge can be built up over time as the examples given become progressively more difficult, as inspired by [Bengio et al.(2009)Bengio, Louradour, Collobert, and Weston, Kumar et al.(2010)Kumar, Packer, and Koller]. The loss computed by a forward pass is used to rank samples (i.e. source and target frame pairs) in the batch according to their difficulty in a manner similar to [Loshchilov and Hutter(2016), Simo-Serra et al.(2015)Simo-Serra, Trulls, Ferraz, Kokkinos, Fua, and Moreno-Noguer, Shrivastava et al.(2016)Shrivastava, Gupta, and Girshick, Nagrani et al.(2018)Nagrani, Albanie, and Zisserman]. However, these methods use only the most difficult samples, which was found to stop our network from learning. Similarly to [Nagrani et al.(2018)Nagrani, Albanie, and Zisserman], using progressively more difficult samples proved crucial for the strategy’s success.

Given a batch size of randomly chosen samples, i.e. source and target frame pairs, a forward pass is executed and the loss for each sample computed. The samples are ranked and sorted according to this loss. Initially the loss is back-propagated only on the samples in the batch which are in the 50th percentile (i.e. the samples with the lowest loss computed by the forward pass). These are assumed to be easier samples. When the loss on the validation set plateaus, the subset to be back-propagated on is shifted by 10 (e.g. the samples in the 10th-60th percentile range). This is repeated 4 times until the samples being back-propagated on fall into the 40th-90th percentile range. At this point the curriculum strategy is terminated, as it is assumed that the samples in the 90-100th range are too challenging or may be problematic (e.g. there is a large shift in the background which is too challenging to learn).

5 Experiments

In this section, we evaluate the network and the learned embedding. In Section 5.1, the performance of using FAb-Net’s learned representation is compared to that of state-of-the-art self-supervised and supervised methods on a variety of tasks: facial landmark prediction, head pose regression and expression classification.

Section 5.2 discusses the benefit of using additional source frames, and Section 5.3 shows how the learned representation can be used for retrieving images with similar facial attributes.

Training. The model is trained on the VoxCeleb1 and VoxCeleb2 video datasets [Nagrani et al.(2017)Nagrani, Chung, and Zisserman, Chung et al.(2018)Chung, Nagrani, and Zisserman]; we refer to the combined datasets as VoxCeleb+. The VoxCeleb+ dataset consists of videos of interviews containing more than 1 million utterances of around 7,000 speakers. The frames are extracted at 1 fps. The frames are cropped, resized to , and the identities are randomly split into train/val/test (with a split of 75/15/10).

The models are trained in PyTorch 

[Paszke et al.(2017)Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin, Desmaison, Antiga, and Lerer] using SGD, an initial learning rate of , and momentum . When using the curriculum strategy described in Section 4, the batchsize is , else . The learning rate is divided by a factor of 10 when the loss on the validation set plateaus. (If the curriculum strategy is used, the learning rate is updated only when the 40-90th percentile is considered.) This is repeated until the loss converges. Further details about the training can be found in the supp. material.

5.1 Using the embedding for regression and classification

First, we investigate the representation learned in our embedding and evaluate whether it indeed encodes facial attributes by challenging it to predict three different attributes: landmarks, pose, and expression.


Given a network trained on VoxCeleb+, a linear regressor or classifier is trained from the learned embedding to the output task. The linear regressor/classifier consists of two layers: batch-norm 

[Ioffe and Szegedy(2015)] followed by a linear fully connected layer with no bias. The regression tasks are trained using a MSE loss. The classification tasks are trained with a cross-entropy loss. The parameters of the encoder are fixed while the two additional layers are trained on the training set of the target dataset using Adam [Kingma and Ba(2015)], a learning rate of , and .

5.1.1 Baselines


There are prior publications on using self-supervision for landmark prediction on the datasets we evaluate on, but none for predicting emotion on standard datasets. Consequently, we implement an autoencoder and a set of state-of-the-art self-supervised methods 

[Gidaris et al.(2018)Gidaris, Singh, and Komodakis, Zhang et al.(2017)Zhang, Isola, and Efros] for object detection and segmentation. The baselines are trained using the same architecture as FAb-Net but with their associated loss functions and training objectives. For [Zhang et al.(2017)Zhang, Isola, and Efros], the regression loss for both the L and ab channels is used. These models are trained on VoxCeleb+ until convergence, with the same training parameters and data augmentation as FAb-Net. More details are given in the supp. material.

VGG-Face descriptor. We additionally compare to the VGG-Face descriptor which is obtained from the 4096-dimensional FC7 features from a VGG-16 network trained on the VGG-Face dataset [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman]. Contrary to popular belief, it has been recently shown that a network trained for identity does retain information about other facial attributes [Cole et al.(2017)Cole, Belanger, Krishnan, Sarna, Mosseri, and Freeman, Ephrat et al.(2018)Ephrat, Mosseri, Lang, Dekel, Wilson, Hassidim, Freeman, and Rubinstein]

. We use the VGG-Face descriptor to learn a linear regression/classification layer to the desired attribute task. This provides a strong baseline, and the results obtained confirm the finding that a network trained for identity does indeed encode expression and to some extent also pose information. However, note that unlike our method, this face descriptor requires a large dataset of

labelled face images for training.

5.1.2 Results

Facial landmarks.

Method 300-W MAFL Self-supervised Trained on VoxCeleb+ FAb-Net 6.31 3.78 FAb-Net w/ curric. 5.73 3.49 FAb-Net w/ curric., 3 source frames 5.71 3.44 Trained on CelebA Jakab et al. [Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi] (2018) 3.08 Zhang et al. [Zhang et al.(2018)Zhang, Guo, Jin, Luo, He, and Lee] (2018) 3.15 Thewlis et al. [Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi] (2017) 9.30 6.67 Thewlis et al. [Thewlis et al.(2017b)Thewlis, Bilen, and Vedaldi] (2017) 7.97 5.83 Supervised Trained on CelebA MTCNN [Zhang et al.(2014)Zhang, Luo, Loy, and Tang] (2014) 5.39 LBF [Ren et al.(2014)Ren, Cao, Wei, and Sun] (2014) 6.32 CFSS [Zhu et al.(2015)Zhu, Li, Loy, and Tang] (2015) 5.76 cGPRT [Lee et al.(2015)Lee, Park, and Yoo] (2015) 5.71 DDN [Yu et al.(2016)Yu, Zhou, and Chandraker] (2016) 5.65 TCDCN [Zhang et al.(2016b)Zhang, Luo, Loy, and Tang] (2016) 5.54 RAR [Xiao et al.(2016)Xiao, Feng, Xing, Lai, Yan, and Kassim] (2016) 4.94 VGG-Face descriptor [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman] 11.16 5.92
Table 1: Landmark prediction error on 300-W and MAFL datasets. Lower is better.
Method Roll Pitch Yaw MAE Self-supervised FAb-Net 5.54 7.84 12.93 8.77 FAb-Net w/ curric. 5.33 7.21 11.34 7.96 FAb-Net w/ curric., 3 source frames 5.14 7.13 10.70 7.65 Supervised VGG-Face descriptor [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman] 8.24 8.36 18.35 11.65 KEPLER [Kumar et al.(2017)Kumar, Alavi, and Chellappa] (2017) 8.75 5.85 6.45 7.02
Table 2: Pose prediction error on the AFLW test set from [Kumar et al.(2017)Kumar, Alavi, and Chellappa]. Lower is better.

Facial landmark locations are regressed from the learned embedding and compared to state-of-the-art methods on MAFL [Zhang et al.(2016b)Zhang, Luo, Loy, and Tang] and the more challenging -W [Sagonas et al.(2016)Sagonas, Antonakos, Tzimiropoulos, Zafeiriou, and Pantic] datasets. The evaluation is performed as outlined in [Zhang et al.(2016b)Zhang, Luo, Loy, and Tang, Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi], and the errors given in inter-ocular distance. For MAFL, 5 facial landmarks are regressed for 19k/1k train/test images. For -W, landmarks are regressed for train/test images which are obtained (as described in [Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi]) from combining multiple datasets [Zhu and Ramanan(2012), Belhumeur et al.(2013)Belhumeur, Jacobs, Kriegman, and Kumar, Zhou et al.(2013)Zhou, Fan, Cao, Jiang, and Yin].

The results are reported in Table 2 and some qualitative results are visualised in Fig. 2 and Fig. 3. These results first demonstrate that fine-tuning with additional views and our curriculum strategy improve the embedding learned by FAb-Net. Second, these results show that our method performs competitively or better than state-of-the-art unsupervised landmark detection methods, better than the VGG-Face descriptor baseline and competitively with state-of-the-art supervised methods. This is achieved even though the other self-supervised methods [Zhang et al.(2018)Zhang, Guo, Jin, Luo, He, and Lee, Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi, Thewlis et al.(2017b)Thewlis, Bilen, and Vedaldi, Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi] are explicitly engineered to detect landmarks whereas our method is not. In addition to that, our method is able to bridge the domain gap between VoxCeleb+ and CelebA [Liu et al.(2015)Liu, Luo, Wang, and Tang] – the other self-supervised methods that we compare to are pre-trained on CelebA.

Figure 2: Landmark prediction visualisation for FAb-Net on the MAFL dataset. A dot denotes ground truth and the cross FAb-Net’s prediction. A failure case is shown to the right.
(a) Ground truth landmarks
(b) FAb-Net’s predicted landmarks
Figure 3: Landmark prediction visualisation for FAb-Net on the -W dataset.

Pose. The learned embedding is used for pose prediction and compared to a supervised method [Kumar et al.(2017)Kumar, Alavi, and Chellappa] and to using the VGG-Face descriptor. To perform the evaluation, the linear regression is trained from the given embedding to head pose labels using the AFLW dataset [Koestinger et al.(2011)Koestinger, Wohlhart, Roth, and Bischof], but after leaving out the images of the AFLW test set from [Kumar et al.(2017)Kumar, Alavi, and Chellappa]. As can be seen in Table 2, FAb-Net performs better in predicting the roll angle, and the MAE is comparable to [Kumar et al.(2017)Kumar, Alavi, and Chellappa] which is supervised with head pose labels. Furthermore, our embedding outperforms the VGG-Face descriptor which is trained on identities; i.e. our learned embedding encodes more information about head pose.


AUC for different AUs
1 2 4 5 6 9 12 17 20 25 26 avg.
FAb-Net 72.0 68.9 73.2 69.4 88.2 78.6 89.5 71.0 75.9 81.4 72.0 76.4
FAb-Net w/ curriculum 73.4 71.8 75.3 67.8 90.4 78.8 91.9 72.4 74.5 83.7 73.3 77.6
FAb-Net w/ curriculum, 3 source frames 74.1 72.3 75.8 68.8 90.7 81.8 92.5 73.7 77.2 83.6 73.6 78.6
Gidaris et al[Gidaris et al.(2018)Gidaris, Singh, and Komodakis] 68.6 64.0 72.8 70.0 83.9 78.1 83.8 68.4 72.6 73.1 67.2 72.9
SplitBrain [Zhang et al.(2017)Zhang, Isola, and Efros] 65.5 59.8 66.7 60.8 71.8 65.8 73.3 64.5 57.4 68.1 61.1 65.0
Autoencoder 67.2 60.5 70.1 65.1 79.6 70.4 80.1 68.3 66.5 70.5 64.1 69.3
VGG-Face descriptor [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman] 81.8 83.0 83.5 81.8 92.0 90.9 95.7 80.6 85.2 86.5 73.0 84.9
VGG-11 (from scratch) 74.7 77.2 85.8 83.7 93.8 89.7 97.5 78.3 86.9 96.4 81.5 86.0
Table 3: Expression classification results for state-of-the-art self-supervised and supervised methods on EmotioNet [Benitez-Quiroz et al.(2016)Benitez-Quiroz, Srinivasan, and Martinez] for multiple facial action units (AUs). Higher is better for AUC.
Neutral Happy Sad Surprise Fear Disgust Anger Contempt avg.
FAb-Net 70.0 87.6 68.8 75.5 76.5 70.0 73.2 71.2 74.2
FAb-Net w/ curric. 71.5 90.0 70.8 78.2 77.4 72.2 75.7 72.1 76.0
FAb-Net w/ curric., 3 source frames 72.3 90.4 70.9 78.6 77.8 72.5 76.4 72.2 76.4
Gidaris et al[Gidaris et al.(2018)Gidaris, Singh, and Komodakis] 67.8 84.9 69.0 73.9 75.7 69.8 71.5 68.7 72.7
SplitBrain [Zhang et al.(2017)Zhang, Isola, and Efros] 63.9 74.7 64.2 61.3 68.3 58.6 68.2 62.8 65.3
Autoencoder 65.8 80.0 64.7 66.1 70.6 63.4 68.3 65.0 68.0
VGG-Face descriptor [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman] 75.9 92.2 80.5 81.4 82.3 81.4 81.2 77.1 81.5
AlexNet [Mollahosseini et al.(2017)Mollahosseini, Hasani, and Mahoor] 82
Table 4: Expression classification results for state-of-the-art self-supervised and supervised methods on AffectNet [Mollahosseini et al.(2017)Mollahosseini, Hasani, and Mahoor]. Higher is better for AUC.

We evaluate the performance of our learned embedding for expression estimation on two datasets: AffectNet 

[Mollahosseini et al.(2017)Mollahosseini, Hasani, and Mahoor] and EmotioNet [Benitez-Quiroz et al.(2016)Benitez-Quiroz, Srinivasan, and Martinez], which both contain over 900,000 images. These datasets are taken ‘in-the-wild’ as opposed to in a constrained environment. AffectNet contains 8 facial expressions (neutral, happy, sad, surprise, fear, disgust, anger, contempt) and EmotioNet contains 11 action units (AUs) (combinations of AUs correspond to facial expressions).

Both of these datasets were organised for challenges with a held out, unreleased test set. Therefore, the train set is subdivided into two subsets; one is used for training and the other for validation. The validation set of the original dataset is used to test the different models. The linear classifier for EmotioNet is trained with a binary cross-entropy loss for each AU, whereas for AffectNet, a cross-entropy loss is used. Both training datasets are highly imbalanced. As a result, the examples from the under-represented classes are re-weighted inversely proportionally to the class frequencies to penalise the loss more heavily for mis-classifying images of the under-represented classes.

The embedding learned by FAb-Net is compared to a number of self-supervised and supervised methods by measuring the Area Under the ROC curve (AUC). For each class (e.g. emotion or AU), the AUC is computed independently and the result is averaged over all classes. The results are reported in Table 3 and Table 4 for EmotioNet and AffectNet respectively showing that our network performs better than other self-supervised methods over both metrics when given the same training data. This is supposedly due to the fact that the network must learn to transform the source frame in order to generate the target frame. As parts of the face move together (e.g. an eyebrow raise or the lips when the mouth opens), the embedding must learn to encode information about facial features and thereby encode expression. Interestingly, the autoencoder performs well, presumably due to the restricted nature of this domain.

FAb-Net is also not far off supervised methods despite the domain shift; VoxCeleb+ consists only of people being interviewed (and consequently with mostly neutral/smiling faces), so it does not include the range/extremity of expressions found in AffectNet or EmotioNet. Finally, it can be observed that the VGG-Face descriptor trained to predict identities does surprisingly well at predicting emotion.

Discussion. FAb-Net has achieved impressive performance, as most self-supervised methods when transferred to another task have a large gap in comparison to supervised methods. There is no gap or a small gap for the smaller datasets (landmarks/pose) and the model approaches supervised performance for the larger datasets (expression).

Figure 4: Confidence heatmaps learned by FAb-Net. Higher intensity corresponds to higher confidence. The network chooses the frames with most similar poses to draw from and ignores those with less similar poses (see Example (a)). In Example (b), the mouth has higher confidence in the third source frame allowing the network to re-construct the teeth that are present in the target frame. More examples can be found in the supp. material.

5.2 What is the benefit of additional source frames?

The previous sections have shown that using additional source frames improves performance. This is at the expense of performing additional forward passes through the encoder (in this case two). Given enough GPU memory, these forward passes can be done in parallel, affecting only the memory requirements and not the computational speed.

Using multiple source frames is further investigated by visualising confidence heatmaps for a given set of source frames in Fig. 4. The confidence heatmaps allow images with more similar pose to be used for creating the generated frame. Furthermore, the network can focus on one frame for generating a part of the face (e.g. the mouth) and on another one for a different part.

5.3 Image retrieval

This section considers an application of the learned embedding: retrieving images with similar facial attributes (e.g. pose) but across different identities. To perform this task, a subset of 10,000 randomly sampled test images from VoxCeleb+ is obtained. For a given query image, all other images (the gallery) are ranked based on their similarity to the query image using the cosine similarity metric between the corresponding embeddings. For a given query image

, the embedding is extracted by performing a forward pass through the network. Similarly, the embedding is extracted for each image in the gallery. Each image is then ranked according to the cosine similarity between and . If the network does indeed encode salient information about facial attributes, the cosine similarity can be used to identify images with similar poses and facial attributes. For a set of query images, the results are visualised in Fig. 5. From these results it is again affirmed that our embedding encodes information about facial attributes, as the retrieved images have poses and expressions similar to those of the query images. Note, the embedding is largely unaffected by facial decorations (e.g. glasses) and identity, as these do not change within a face-track and so do not need to be learned in order to predict the transformation.

(a) (b) (c) Query Images Nearest Neighbours. Arranged left to right. (d) (e) (f) Query Images Nearest Neighbours. Arranged left to right.
Figure 5: Retrieval results using the embedding learned by FAb-Net. The embedding captures similar visual attributes since gallery images with similar facial attributes are retrieved for a given query image. The retrieved images have similar pose to the query in all cases, and the expression similarity can be seen for example in (b) with the eyes shut, and (a) with the mouth slightly open. Please refer to the supp. material for additional examples.

6 Conclusion

We have introduced FAb-Net: a self-supervised framework for learning facial attributes from videos. Our method learns about pose and expression by watching faces move and change over a large number of videos without any hand labels. The features of our trained network can then be used to predict pose, landmarks, and expression on other datasets (despite the domain shift) by just training a linear layer on top of the learned embedding. The features have been shown to be comparable or superior performance to self-supervised and supervised methods on a variety of tasks. This is impressive as generally the performance of self-supervised methods has been found to be worse than that of supervised methods, yet our method is indeed competitive/superior to supervised methods for pose regression and facial landmark detection, and approaches supervised performance on expression classification.

7 Acknowledgements

The authors would like to thank James Thewlis for helpfully sharing code and datasets. This work was funded by an EPSRC studentship and EPSRC Programme Grant Seebibyte EP/M013774/1.


  • [Agrawal et al.(2015)Agrawal, Carreira, and Malik] P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In Proc. ICCV, 2015.
  • [Barsoum et al.(2016)Barsoum, Zhang, Ferrer, and Zhang] E. Barsoum, C. Zhang, C. C. Ferrer, and Z. Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 279–283. ACM, 2016.
  • [Bas et al.(2017)Bas, Huber, Smith, Awais, and Kittler] A. Bas, P. Huber, W. A. P. Smith, M. Awais, and J. Kittler.

    3D morphable models as spatial transformer networks.

    In Proc. ICCV Workshop on Geometry Meets Deep Learning, 2017.
  • [Belhumeur et al.(2013)Belhumeur, Jacobs, Kriegman, and Kumar] P. N. Belhumeur, D. W. Jacobs, D. J. Kriegman, and N. Kumar. Localizing parts of faces using a consensus of exemplars. IEEE PAMI, 2013.
  • [Bengio et al.(2009)Bengio, Louradour, Collobert, and Weston] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proc. ICML, 2009.
  • [Benitez-Quiroz et al.(2016)Benitez-Quiroz, Srinivasan, and Martinez] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez. EmotioNet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proc. CVPR, 2016.
  • [Blanz et al.(2002)Blanz, Romdhani, and Vetter] V. Blanz, S. Romdhani, and T. Vetter. Face identification across different poses and illumination with a 3D morphable model. In Proc. AFGR, 2002.
  • [Chung et al.(2017)Chung, Jamaludin, and Zisserman] J. S. Chung, A. Jamaludin, and A. Zisserman. You said that? In Proc. BMVC., 2017.
  • [Chung et al.(2018)Chung, Nagrani, and Zisserman] J. S. Chung, A. Nagrani, and A. Zisserman. VoxCeleb2: Deep speaker recognition. In INTERSPEECH, 2018.
  • [Cole et al.(2017)Cole, Belanger, Krishnan, Sarna, Mosseri, and Freeman] F. Cole, D. Belanger, D. Krishnan, A. Sarna, I. Mosseri, and W. T. Freeman. Synthesizing normalized faces from facial identity features. In Proc. CVPR, 2017.
  • [Denton and Birodkar(2017)] E. L. Denton and V. Birodkar. Unsupervised learning of disentangled representations from video. In NIPS, 2017.
  • [Doersch et al.(2015)Doersch, Gupta, and Efros] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proc. ICCV, 2015.
  • [Dosovitskiy et al.(2015)Dosovitskiy, Springenberg, and Brox] A. Dosovitskiy, J. T. Springenberg, and T. Brox.

    Learning to generate chairs with convolutional neural networks.

    In Proc. CVPR, 2015.
  • [Ekman and Oster(1979)] P. Ekman and H. Oster. Facial expressions of emotion. Annual review of psychology, 30(1):527–554, 1979.
  • [Ephrat et al.(2018)Ephrat, Mosseri, Lang, Dekel, Wilson, Hassidim, Freeman, and Rubinstein] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. Proc. ACM SIGGRAPH, 2018.
  • [Fernando et al.(2017)Fernando, Bilen, Gavves, and Gould] B. Fernando, H. Bilen, E. Gavves, and S. Gould.

    Self-supervised video representation learning with odd-one-out networks.

    In Proc. CVPR, 2017.
  • [Gan et al.(2018)Gan, Gong, Liu, Su, and Guibas] C. Gan, B. Gong, K. Liu, H. Su, and L. J. Guibas. Geometry guided convolutional neural networks for self-supervised video representation learning. In Proc. CVPR, 2018.
  • [Gerull and Rapee(2002)] F. C. Gerull and R. M. Rapee. Mother knows best: effects of maternal modelling on the acquisition of fear and avoidance behaviour in toddlers. Behaviour research and therapy, 40(3):279–287, 2002.
  • [Gidaris et al.(2018)Gidaris, Singh, and Komodakis] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. ICLR, 2018.
  • [Ioffe and Szegedy(2015)] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, 2015.
  • [Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi. Conditional image generation for learning the structure of visual objects. arXiv preprint arXiv:1806.07823, 2018.
  • [Jia et al.(2016)Jia, De Brabandere, Tuytelaars, and Gool] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. In NIPS, 2016.
  • [Kingma and Ba(2015)] D. Kingma and J. Ba. Adam: A method for stochastic optimization. Proc. ICLR, 2015.
  • [Koestinger et al.(2011)Koestinger, Wohlhart, Roth, and Bischof] M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof. Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization. In Proc. First IEEE International Workshop on Benchmarking Facial Image Analysis Technologies, 2011.
  • [Kulkarni et al.(2015)Kulkarni, Whitney, Kohli, and Tenenbaum] T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. In NIPS, 2015.
  • [Kumar et al.(2017)Kumar, Alavi, and Chellappa] A. Kumar, A. Alavi, and R. Chellappa. KEPLER: keypoint and pose estimation of unconstrained faces by learning efficient H-CNN regressors. In Proc. Int. Conf. Autom. Face and Gesture Recog., 2017.
  • [Kumar et al.(2010)Kumar, Packer, and Koller] M. P. Kumar, B. Packer, and D. Koller. Self-paced learning for latent variable models. In NIPS. 2010.
  • [Kumar et al.(2011)Kumar, Berg, Belhumeur, and Nayar] N. Kumar, A. Berg, P. Belhumeur, and S. Nayar. Describable visual attributes for face verification and image search. IEEE PAMI, 33(10):1962–1977, 2011.
  • [Lee et al.(2015)Lee, Park, and Yoo] D. Lee, H. Park, and C. D. Yoo. Face alignment using cascade gaussian process regression trees. In Proc. CVPR, 2015.
  • [Lee et al.(2017)Lee, Huang, Singh, and Yang] H.-Y. Lee, J.-B. Huang, M. Singh, and M.-H. Yang. Unsupervised representation learning by sorting sequences. In Proc. ICCV, 2017.
  • [Li et al.(2015)Li, Wang, Liu, Jiang, Shan, and Chen] Y. Li, R. Wang, H. Liu, H. Jiang, S. Shan, and X. Chen.

    Two birds, one stone: Jointly learning binary code for large-scale face image retrieval and attributes prediction.

    In Proc. ICCV, 2015.
  • [Liu et al.(2015)Liu, Luo, Wang, and Tang] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proc. ICCV, 2015.
  • [Loshchilov and Hutter(2016)] I. Loshchilov and F. Hutter. Online batch selection for faster training of neural networks. ICLR Workshops, 2016.
  • [Misra et al.(2016)Misra, Zitnick, and Hebert] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Proc. ECCV, 2016.
  • [Mollahosseini et al.(2017)Mollahosseini, Hasani, and Mahoor] A. Mollahosseini, B. Hasani, and M. H. Mahoor. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 2017.
  • [Nagrani et al.(2017)Nagrani, Chung, and Zisserman] A. Nagrani, J. S. Chung, and A. Zisserman. VoxCeleb: a large-scale speaker identification dataset. In INTERSPEECH, 2017.
  • [Nagrani et al.(2018)Nagrani, Albanie, and Zisserman] A. Nagrani, S. Albanie, and A. Zisserman. Learnable pins: Cross-modal embeddings for person identity. In Proc. ECCV, 2018.
  • [Noroozi and Favaro(2016)] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. ECCV, 2016.
  • [Novotny et al.(2017)Novotny, Larlus, and Vedaldi] D. Novotny, D. Larlus, and A. Vedaldi. Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching. In Proc. CVPR, 2017.
  • [Novotny et al.(2018)Novotny, Albanie, Larlus, and Vedaldi] D. Novotny, S. Albanie, D. Larlus, and A. Vedaldi. Self-supervised learning of geometrically stable features through probabilistic introspection. In Proc. CVPR, 2018.
  • [Parkhi et al.(2015)Parkhi, Vedaldi, and Zisserman] O. M. Parkhi, A. Vedaldi, and A. Zisserman.

    Deep face recognition.

    In Proc. BMVC., 2015.
  • [Paszke et al.(2017)Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin, Desmaison, Antiga, and Lerer] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in PyTorch. 2017.
  • [Pathak et al.(2016)Pathak, Krähenbühl, Donahue, Darrell, and Efros] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proc. CVPR, 2016.
  • [Pătrăucean et al.(2016)Pătrăucean, Handa, and Cipolla] V. Pătrăucean, A. Handa, and R. Cipolla. Spatio-temporal video autoencoder with differentiable memory. In NIPS, 2016.
  • [Ren et al.(2014)Ren, Cao, Wei, and Sun] S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment at 3000 fps via regressing local binary features. In Proc. CVPR, 2014.
  • [Rudd et al.(2016)Rudd, Günther, and Boult] E. M. Rudd, M. Günther, and T. E. Boult. Moon: A mixed objective optimization network for the recognition of facial attributes. In Proc. ECCV, 2016.
  • [Sagonas et al.(2016)Sagonas, Antonakos, Tzimiropoulos, Zafeiriou, and Pantic] C. Sagonas, E. Antonakos, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 300 faces in-the-wild challenge: Database and results. Image and Vision Computing, 47:3–18, 2016.
  • [Shrivastava et al.(2016)Shrivastava, Gupta, and Girshick] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In Proc. CVPR, 2016.
  • [Simo-Serra et al.(2015)Simo-Serra, Trulls, Ferraz, Kokkinos, Fua, and Moreno-Noguer] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors. In Proc. ICCV, 2015.
  • [Tewari et al.(2017)Tewari, Zollhöfer, Kim, Garrido, Bernard, Perez, and Theobalt] A. Tewari, M. Zollhöfer, H. Kim, P. Garrido, F. Bernard, P. Perez, and C. Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proc. ICCV, 2017.
  • [Thewlis et al.(2017a)Thewlis, Bilen, and Vedaldi] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In Proc. ICCV, 2017a.
  • [Thewlis et al.(2017b)Thewlis, Bilen, and Vedaldi] J. Thewlis, H. Bilen, and A. Vedaldi. Unsupervised learning of object frames by dense equivariant image labelling. In NIPS, 2017b.
  • [Tran et al.(2017)Tran, Yin, and Liu] L. Tran, X. Yin, and X. Liu. Disentangled representation learning gan for pose-invariant face recognition. In Proc. CVPR, 2017.
  • [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy. Tracking emerges by colorizing videos. In Proc. ECCV, 2018.
  • [Wang and Gupta(2015)] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proc. ICCV, 2015.
  • [Wiles et al.(2018)Wiles, Koepke, and Zisserman] O. Wiles, A. S. Koepke, and A. Zisserman. X2Face: A network for controlling face generation using images, audio, and pose codes. In Proc. ECCV, 2018.
  • [Xiao et al.(2016)Xiao, Feng, Xing, Lai, Yan, and Kassim] S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, and A. Kassim. Robust facial landmark detection via recurrent attentive-refinement networks. In Proc. ECCV, 2016.
  • [Xue et al.(2016)Xue, Wu, Bouman, and Freeman] T. Xue, J. Wu, K. Bouman, and B. Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In NIPS, 2016.
  • [Yang et al.(2017)Yang, Ren, Chen, Wen, Li, and Hua] J. Yang, P. Ren, D. Chen, F. Wen, H. Li, and G. Hua. Neural aggregation network for video face recognition. In Proc. CVPR, 2017.
  • [Yu et al.(2016)Yu, Zhou, and Chandraker] X. Yu, F. Zhou, and M. Chandraker. Deep deformation network for object landmark localization. In Proc. ECCV, 2016.
  • [Zamir et al.(2016)Zamir, Wekel, Agrawal, Wei, Malik, and Savarese] A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and S. Savarese. Generic 3d representation via pose estimation and matching. In Proc. ECCV, 2016.
  • [Zhang et al.(2016a)Zhang, Isola, and Efros] R. Zhang, P. Isola, and A. A. Efros.

    Colorful image colorization.

    In Proc. ECCV, 2016a.
  • [Zhang et al.(2017)Zhang, Isola, and Efros] R. Zhang, P. Isola, and A. A. Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proc. CVPR, 2017.
  • [Zhang et al.(2018)Zhang, Guo, Jin, Luo, He, and Lee] Y. Zhang, Y. Guo, Y. Jin, Y. Luo, Z. He, and H. Lee. Unsupervised discovery of object landmarks as structural representations. In Proc. CVPR, 2018.
  • [Zhang et al.(2014)Zhang, Luo, Loy, and Tang] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial landmark detection by deep multi-task learning. In Proc. ECCV, 2014.
  • [Zhang et al.(2016b)Zhang, Luo, Loy, and Tang] Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Learning deep representation for face alignment with auxiliary attributes. IEEE PAMI, 2016b.
  • [Zhou et al.(2013)Zhou, Fan, Cao, Jiang, and Yin] E. Zhou, H. Fan, Z. Cao, Y. Jiang, and Q. Yin. Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In ICCVW, 2013.
  • [Zhou et al.(2016)Zhou, Tulsiani, Sun, Malik, and Efros] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. View synthesis by appearance flow. In Proc. ECCV, 2016.
  • [Zhu et al.(2015)Zhu, Li, Loy, and Tang] S. Zhu, C. Li, C. C. Loy, and X. Tang. Face alignment by coarse-to-fine shape searching. In Proc. CVPR, 2015.
  • [Zhu and Ramanan(2012)] X. Zhu and D. Ramanan. Face detection, pose estimation and landmark localization in the wild. Proc. CVPR, 2012.