Leveraging Mid-Level Deep Representations For Predicting Face Attributes in the Wild

by   Yang Zhong, et al.
KTH Royal Institute of Technology

Predicting facial attributes from faces in the wild is very challenging due to pose and lighting variations in the real world. The key to this problem is to build proper feature representations to cope with these unfavourable conditions. Given the success of Convolutional Neural Network (CNN) in image classification, the high-level CNN feature, as an intuitive and reasonable choice, has been widely utilized for this problem. In this paper, however, we consider the mid-level CNN features as an alternative to the high-level ones for attribute prediction. This is based on the observation that face attributes are different: some of them are locally oriented while others are globally defined. Our investigations reveal that the mid-level deep representations outperform the prediction accuracy achieved by the (fine-tuned) high-level abstractions. We empirically demonstrate that the midlevel representations achieve state-of-the-art prediction performance on CelebA and LFWA datasets. Our investigations also show that by utilizing the mid-level representations one can employ a single deep network to achieve both face recognition and attribute prediction.



There are no comments yet.


page 1

page 2

page 3

page 4


Face Attribute Prediction Using Off-the-Shelf CNN Features

Predicting attributes from face images in the wild is a challenging comp...

Deep Learning Face Attributes in the Wild

Predicting face attributes in the wild is challenging due to complex fac...

A Novel Multi-Task Tensor Correlation Neural Network for Facial Attribute Prediction

Face multi-attribute prediction benefits substantially from multi-task l...

Multi-label Learning Based Deep Transfer Neural Network for Facial Attribute Classification

Deep Neural Network (DNN) has recently achieved outstanding performance ...

Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network

The 3D shapes of faces are well known to be discriminative. Yet despite ...

Slim-CNN: A Light-Weight CNN for Face Attribute Prediction

We introduce a computationally-efficient CNN micro-architecture Slim Mod...

Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data

The way people look in terms of facial attributes (ethnicity, hair color...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Telling attributes from face images in the wild is known to be very useful in large scale face search, image understanding and face recognition. However, the problem indeed is very challenging since the faces captured in the real world are often affected by many unfavourable influences, such as illumination, pose and expression. To build usable attribute prediction, it is important to preserve the essential face traits in the representations and at the same time, make them insensitive to interference.

The representations describing faces in prior literature generally form two groups: the hand-crafted representations and learned representations. Exemplified as in [7, 2], local low-level features were constructed from detected face regions. The use of local features were mostly based on the consideration that they are less likely to be influenced than the holistic features by pose changes and facial expressions. In recent work [8], multi-scale Gabor features were used as a holistic face representation, which is then converted by a learned hashing process for efficient face retrieval and attribute prediction.

Driven by the great improvements brought by the CNN in image classification [6, 14, 12] and face recognition [13, 9, 15, 11]

, features extracted from deep architectures became a natural and reasonable choice to represent faces for attribute prediction. In

[17], local semantic image patches were first detected and fed into deep networks to construct concatenated, pose normalized representation. In [18], through intensive training, two concatenating CNNs were built to locate and predict attributes from arbitrary size of faces in the wild. The first CNN stage was pre-trained by image categories and fine-tuned by face attribute tags to locate face from complex background; the second stage was pre-trained by identity labels and fine-tuned by face attribute data to achieve an effective fusion of the discrimination of inter-person representations and the variability of non-identity related factors in the face representations. As a common character, high-level hidden (Fully Connected, FC) layers were especially trained for representing attributes in these work.

The best representation for retrieval tasks has been shown to come from the FC layer [1, 10]. Naturally, features from FC layers are commonly used for attribute prediction. But considering that different levels of ConvNet encode different levels of abstraction, one can expect that such representations may not be optimal to describe the physical facial characteristics, especially the local attributes, such as “open mouth” and “wearing glasses”. It is therefore rational to consider CNN features from earlier layers, which better preserve both discriminating power and spatial information, for predicting face attributes.

As indicated in [19]

, the earlier-layer CNN features are likely to better describe the facial appearance than the high-level features. In this work, our focus is on identifying the best face representations for recognizing face attributes. We employ publicly available data, architecture and a deep learning framework to train a face classification CNN. Hierarchical representations are then constructed and evaluated in a face attribute prediction context. Through intensive investigations, we empirically show the effectiveness of the hierarchical deep representations and demonstrate the advantages of the mid-level representations for tackling the face attribute prediction problem. The major contributions of this work are:

  • Our investigations reveal the fact of the diverse utilities of deep hierarchical representations for face attribute prediction; intermediate representations are shown very effective for predicting the human describable face attributes (Section 2.2).

  • By jointly leveraging the mid-level hierarchical representations, we further improve the state-of-the-art on two large scale datasets (Section 2.3).

  • One can construct a single deep network for both face recognition and attribute prediction.

2 Investigation

Our motivation in this work is to study the effectiveness of hierarchical CNN representations for attribute prediction of faces captured in the wild. Our procedure is to first extract hierarchical representations from aligned faces using our trained CNN and then we construct and evaluate linear attribute classifiers to identify the most effective representation of each attribute. In the following sections, we first describe the experimental details and then present our investigations and results on the CelebA dataset and the annotated LFW (LFWA) dataset.

2.1 Implementation details

CNN Training: The face representations studied in this work were extracted from a face classification CNN. The architecture of the CNN with configuration details is shown in Table 1. We adopted the structure of the convolutional filters of “FaceNet NN.1 [11]” , with slight modifications, into our CNN and concatenated two FC layers and trained it in a N-way classification manner. The network was initialized from random and started with a learning rate of , which was then decreased two times when the classification accuracy stopped increasing on the validation set. PReLU [3] rectification was used after all the convolutional and FC layers. Dropout layers were inserted between FC layers with dropout rate of to prevent overfitting. We used around identities with face images from the publicly accessible dataset WebFace [16] for training. The training instances were augmented with random flipping, jittering, and slight rotation. The trained CNN had a verification accuracy of on LFW dataset [4], which is comparable to that of the DeepID [13] structure used by [18].

Layer Kernel Output
Conv 1 conv3-64
Pool 1 max 2,2 56*56*64
RNorm 1 5
Conv 2a conv1-64
Conv 2 conv3-192 28*28*192 3* 3* 192
RNorm 2 5
Pool 2 max 2,2
Conv 3a conv1-192
Conv 3 conv3-384 14*14*384 3* 3* 384
Pool 3 max 2,2
Conv 4a conv1-384
Conv 4 conv3-256 14*14*384 3* 3* 384
Conv 5a conv1-256
Conv 5 conv3-256 14*14*384 3* 3* 384
Conv 6a conv1-256
Conv 6 conv3-256 7*7*256 3* 3* 256
Pool 6
FC 1 512 1*1*512
FC 2 512 1*1*512
Table 1: CNN architecture used in our experiments. The perception size is 112 112 (except experiments in Section 2.4). The dimensions of the investigated deep representations are shown in the last column.

Feature Representations: In our experiments faces were aligned based on the detected feature points [5] (or provided feature points). The aligned image, approximately covering from the top of head to neck, had a size of . The center patch, with a size of , and its horizontal flipped patch, were fed in to the CNN. We extracted the average representations from , , , , , and

layers, and additionally applied multi-scale spatial pooling to improve the invariance of CNN activations without degrading their discriminating power. The first average pooling halved the dimension and the second overlapping max. pooling selected the most locally activated neuron

111For example, for layer, the first stage is average pooling (window = 4

4, stride = 4) and the second one is max. pooling (window = 3

3, stride = 2).
. That is, all the resulting representations from the convolutional layers had the same feature map size of 3 3 as shown in the last column of Table 1. The resulting features are correspondingly denoted by , … , , and in the following.

Evaluation Dataset: In this work, we used two datasets222http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html. Released in Oct 2015., CelebA and LFWA, to conduct benchmark evaluations of attribute prediction accuracy. The CelebA contains around images of identities and LFWA has images of identities. Images on both datasets are labeled with 40 binary codes to represent the presence of facial attributes. These attributes ranges from demographic information to physical characteristics such as nose size, eyebrow shape and expressions.

2.2 Exploration

Figure 1: Prediction effectiveness discrepancy of hierarchical representations over 40 attributes on the training set of CelebA. In each grid, y-scale stands for the prediction accuracy in %. The mean prediction accuracy values of each representation (from - ) are , , , , , and .

The attribute prediction power of the hierarchical representations was first investigated on the training set of the CelebA dataset. We used the training set defined in [18] to construct linear SVM attribute classifiers for each representation. The prediction accuracy on all the 40 attributes for each type of representation is demonstrated in Figure 1. It can be observed that features from the intermediate layers ( to ) demonstrate an obvious advantage over the final FC layer on average, especially for attributes describing motions of the mouth area where the gap is almost . Feature had the highest prediction accuracy on average, which was more than higher than the last FC layer. One can also observe that the mid-level convolutional representations also featured slightly different prediction power on different attributes, e.g.  outperformed on “Rosy Cheeks” but not on hair related attributes. Through this exploration, we identified the best representation for each attribute.

2.3 Benchmark evaluations

We then compared the identified best representations of each attribute with a baseline approach and the current state-of-the-art [18], on CelebA and LFWA datasets respectively. The results are compared in Figure 2 and the detailed recognition rates of our approach are given in Table 2. The baseline approach (denoted by [17]+ ANetin in [18]) used CNN trained by identities to extract features from aligned faces; the pipeline of constructing deep representations is the same as ours. The current best, denoted by “LNet+ANet” in [18], employed concatenated CNNs to locate faces and extract features in order to construct attribute classifiers.

One can observe that: by jointly leveraging the best representations, superior performance has been achieved over the equivalent baseline approach and even the current state-of-the-art. For some attributes with relatively lower (baseline) accuracy, e.g. “Blurry” “Oval Face”, “Wearing Necktie” and “Rosy Cheeks”, the advantage of our approach is especially significant. Among the evaluated attributes, the mid-level convolutional features dominated the best representations (only 4 best representations came from FC layers on CelebA and 1 on LFWA).

In addition, we also comprehensively compared the overall performance in Table 3 and layer wise prediction accuracy in Table 4. It can be found that:

  1. utilizing the best representations demonstrates outperforming effectiveness over the equivalent baseline method and state-of-the-art approach on both large scale datasets;

  2. the prediction power increases along the convolutional layers of the CNN;

  3. even the very early layer astonishingly achieved entirely comparable performance as the state-of-the-art.

Indeed, even if we utilize convolutional features on the attributes which had FC as the best representations (e.g. “Bald”), the effect on prediction performance is not detectable (which also reflects in Figure 1).

Figure 2: Comparative prediction accuracy on CelebA (upper) and LFWA (lower) with best performing representation given in parenthesis after each attribute (refer our numeric accuracy values to Table 2).
5 o’Clock Shadow 93.34 76.90 Male 98.09 93.93
Arched Eyebrows 82.50 83.37 Mouth S. Open 92.56 82.13
Attractive 80.77 81.47 Mustache 96.56 94.33
Bags Under Eyes 82.24 83.63 Narrow Eyes 86.92 81.90
Bald 97.75 91.33 No Beard 95.38 80.17
Bangs 95.58 90.93 Oval Face 70.63 77.17
Big Lips 69.90 78.50 Pale Skin 96.69 89.07
Big Nose 82.64 83.53 Pointy Nose 76.17 82.57
Black Hair 86.04 90.90 Receding Hairline 92.14 85.47
Blond Hair 94.89 97.00 Rosy Cheeks 94.29 86.50
Blurry 96.15 88.43 Sideburns 97.44 82.37
Brown Hair 84.15 77.53 Smiling 92.11 91.80
Bushy Eyebrows 91.89 83.67 Straight Hair 80.00 79.93
Chubby 94.87 75.13 Wavy Hair 77.35 80.60
Double Chin 96.19 80.80 Wearing Earrings 86.74 93.67
Eyeglasses 99.48 91.50 Wearing Hat 98.78 90.80
Goatee 97.07 83.23 Wearing Lipstick 92.35 95.30
Gray Hair 97.77 87.67 Wearing Necklace 85.78 89.87
Heavy Makeup 90.14 95.27 Wearing Necktie 94.42 82.50
High Cheekbones 86.06 88.17 Young 87.48 86.47
Table 2: Our attribute recognition accuracy on the CelebA and LFWA in %

Moreover, it is worth nothing that using features at FC layers is computationally expensive due to the bottle neck between convolutional and FC layer. Thus, leveraging the mid-level representations features two inherent advantages:

  1. very efficient inference with competitive performance;

  2. breaking the bounds of the interconnections between convolutional and FC layers so that the CNN accepts arbitrary receptive size (as demonstrated in Section 2.4).

Recall that the training effort of a face classification CNN is almost minimal compared to state-of-the-art two-stage CNN approach and other potential end-to-end learning for attribute prediction. However, the deep hierarchical representations from the trained face classification CNN still allow superior performance. This indicates that these mid-level CNN representations have contained rich spatial information which can be utilized more in building effective (local) appearances description, attribute prediction and retrieval applications.

Baseline LNet+ANet Ours
CelebA 83% 87% 89.8%
LFWA 76% 84% 85.9%
Table 3: Overall comparisons of the baseline approach, current state-of-the-art (LNet+ANet) and our approach.
C2 C3 C4 C5 C6 F1 F2
CelebA 89% 90% 90% 90% 90% 87% 88%
LFWA 83% 86% 86% 86% 85% 82% 81%
Table 4: Layer wise average prediction accuracy over 40 attributes on the test set of CelebA and LFWA of our approach.

2.4 Discussion

It is easy to expect that a number of factors, e.g. image resolution, could affect the face attribute prediction performance. A small receptive size would affect the prediction relating to small scale face traits. We found that by doubling the input size of our CNN the prediction accuracy for “Bag under eyes” increased from to with as the best representation.

Similarly, attributes that are related to the face components with deterministic distribution can have condensed representations. For instance, “arched eyebrow” always appears in the upper part of face images and “wearing necktie” appears in the lower part. We studied the condensed feature () as the representation for locally related attributes and found that the average prediction accuracy remained almost the same (89.3%). This means that we can further reduce the memory footprint of the representations and the classifiers for the attributes defined by deterministic facial components.

3 Conclusions

In this paper, we have proposed to leverage the mid-level representations from deep convolutional neural network to tackle the attribute prediction problem for faces in the wild. We used an off-the-shelf architecture and a publicly available dataset to train a plain classification network, and conduct investigations on the utility of the deep representations from various levels of the network. Although the trained network is not optimized either towards face attribute prediction or recognition, it still allows accurate attribute prediction surpassing the state-of-the-art with a noticeable margin.

Our investigations indicate that CNNs trained for face classification have implicitly learned many semantic concepts and human describable attributes have been embedded in the mid-level deep representations, which can be separated by simple classifiers. They also reveal the potential utility of the intermediate deep representations for other face related tasks. These findings promise to increase the face attribute prediction performance and to achieve multiple intelligent functions, such as face recognition, retrieval and attribute prediction, in one single deep architecture.


  • [1] Hossein Azizpour, Ali Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. From generic to specific deep representations for visual recognition. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

    , pages 36–45, 2015.
  • [2] Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Describing people: A poselet-based approach to attribute classification. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 1543–1550. IEEE, 2011.
  • [3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852, 2015.
  • [4] Gary B. Huang, Marwan Mattar, Tamara Berg, and Erik Learned-miller. E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, 2007.
  • [5] Davis E. King.

    Dlib-ml: A machine learning toolkit.

    Journal of Machine Learning Research, 10:1755–1758, 2009.
  • [6] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
  • [7] Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In Computer Vision, 2009 IEEE 12th International Conference on, pages 365–372. IEEE, 2009.
  • [8] Yan Li, Ruiping Wang, Haomiao Liu, Huajie Jiang, Shiguang Shan, and Xilin Chen.

    Two birds, one stone: Jointly learning binary code for large-scale face image retrieval and attributes prediction.

    In Proceedings of the IEEE International Conference on Computer Vision, pages 3819–3827, 2015.
  • [9] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. Proceedings of the British Machine Vision, 1(3):6, 2015.
  • [10] Ali S Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 512–519. IEEE, 2014.
  • [11] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823, 2015.
  • [12] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  • [13] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1891–1898, 2013.
  • [14] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.
  • [15] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1708, 2013.
  • [16] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014.
  • [17] Ning Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev. Panda: Pose aligned networks for deep attribute modeling. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1637–1644, June 2014.
  • [18] Ziwei Liu, Ping Luo, Xiaogang Wang and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
  • [19] Yang Zhong, Josephine Sullivan, and Haibo Li, “Face Attribute Prediction Using Off-the-Shelf CNN Features,” In Proceedings of International Conference on Biometrics (ICB), 2016.