Telling attributes from face images in the wild is known to be very useful in large scale face search, image understanding and face recognition. However, the problem indeed is very challenging since the faces captured in the real world are often affected by many unfavourable influences, such as illumination, pose and expression. To build usable attribute prediction, it is important to preserve the essential face traits in the representations and at the same time, make them insensitive to interference.
The representations describing faces in prior literature generally form two groups: the hand-crafted representations and learned representations. Exemplified as in [7, 2], local low-level features were constructed from detected face regions. The use of local features were mostly based on the consideration that they are less likely to be influenced than the holistic features by pose changes and facial expressions. In recent work , multi-scale Gabor features were used as a holistic face representation, which is then converted by a learned hashing process for efficient face retrieval and attribute prediction.
, features extracted from deep architectures became a natural and reasonable choice to represent faces for attribute prediction. In, local semantic image patches were first detected and fed into deep networks to construct concatenated, pose normalized representation. In , through intensive training, two concatenating CNNs were built to locate and predict attributes from arbitrary size of faces in the wild. The first CNN stage was pre-trained by image categories and fine-tuned by face attribute tags to locate face from complex background; the second stage was pre-trained by identity labels and fine-tuned by face attribute data to achieve an effective fusion of the discrimination of inter-person representations and the variability of non-identity related factors in the face representations. As a common character, high-level hidden (Fully Connected, FC) layers were especially trained for representing attributes in these work.
The best representation for retrieval tasks has been shown to come from the FC layer [1, 10]. Naturally, features from FC layers are commonly used for attribute prediction. But considering that different levels of ConvNet encode different levels of abstraction, one can expect that such representations may not be optimal to describe the physical facial characteristics, especially the local attributes, such as “open mouth” and “wearing glasses”. It is therefore rational to consider CNN features from earlier layers, which better preserve both discriminating power and spatial information, for predicting face attributes.
As indicated in 
, the earlier-layer CNN features are likely to better describe the facial appearance than the high-level features. In this work, our focus is on identifying the best face representations for recognizing face attributes. We employ publicly available data, architecture and a deep learning framework to train a face classification CNN. Hierarchical representations are then constructed and evaluated in a face attribute prediction context. Through intensive investigations, we empirically show the effectiveness of the hierarchical deep representations and demonstrate the advantages of the mid-level representations for tackling the face attribute prediction problem. The major contributions of this work are:
Our investigations reveal the fact of the diverse utilities of deep hierarchical representations for face attribute prediction; intermediate representations are shown very effective for predicting the human describable face attributes (Section 2.2).
By jointly leveraging the mid-level hierarchical representations, we further improve the state-of-the-art on two large scale datasets (Section 2.3).
One can construct a single deep network for both face recognition and attribute prediction.
Our motivation in this work is to study the effectiveness of hierarchical CNN representations for attribute prediction of faces captured in the wild. Our procedure is to first extract hierarchical representations from aligned faces using our trained CNN and then we construct and evaluate linear attribute classifiers to identify the most effective representation of each attribute. In the following sections, we first describe the experimental details and then present our investigations and results on the CelebA dataset and the annotated LFW (LFWA) dataset.
2.1 Implementation details
CNN Training: The face representations studied in this work were extracted from a face classification CNN. The architecture of the CNN with configuration details is shown in Table 1. We adopted the structure of the convolutional filters of “FaceNet NN.1 ” , with slight modifications, into our CNN and concatenated two FC layers and trained it in a N-way classification manner. The network was initialized from random and started with a learning rate of , which was then decreased two times when the classification accuracy stopped increasing on the validation set. PReLU  rectification was used after all the convolutional and FC layers. Dropout layers were inserted between FC layers with dropout rate of to prevent overfitting. We used around identities with face images from the publicly accessible dataset WebFace  for training. The training instances were augmented with random flipping, jittering, and slight rotation. The trained CNN had a verification accuracy of on LFW dataset , which is comparable to that of the DeepID  structure used by .
|Pool 1||max 2,2||56*56*64|
|Conv 2||conv3-192||28*28*192||3* 3* 192|
|Pool 2||max 2,2|
|Conv 3||conv3-384||14*14*384||3* 3* 384|
|Pool 3||max 2,2|
|Conv 4||conv3-256||14*14*384||3* 3* 384|
|Conv 5||conv3-256||14*14*384||3* 3* 384|
|Conv 6||conv3-256||7*7*256||3* 3* 256|
Feature Representations: In our experiments faces were aligned based on the detected feature points  (or provided feature points). The aligned image, approximately covering from the top of head to neck, had a size of . The center patch, with a size of , and its horizontal flipped patch, were fed in to the CNN. We extracted the average representations from , , , , , and
layers, and additionally applied multi-scale spatial pooling to improve the invariance of CNN activations without degrading their discriminating power. The first average pooling halved the dimension and the second overlapping max. pooling selected the most locally activated neuron111For example, for layer, the first stage is average pooling (window = 4
4, stride = 4) and the second one is max. pooling (window = 33, stride = 2).. That is, all the resulting representations from the convolutional layers had the same feature map size of 3 3 as shown in the last column of Table 1. The resulting features are correspondingly denoted by , … , , and in the following.
Evaluation Dataset: In this work, we used two datasets222http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html. Released in Oct 2015., CelebA and LFWA, to conduct benchmark evaluations of attribute prediction accuracy. The CelebA contains around images of identities and LFWA has images of identities. Images on both datasets are labeled with 40 binary codes to represent the presence of facial attributes. These attributes ranges from demographic information to physical characteristics such as nose size, eyebrow shape and expressions.
The attribute prediction power of the hierarchical representations was first investigated on the training set of the CelebA dataset. We used the training set defined in  to construct linear SVM attribute classifiers for each representation. The prediction accuracy on all the 40 attributes for each type of representation is demonstrated in Figure 1. It can be observed that features from the intermediate layers ( to ) demonstrate an obvious advantage over the final FC layer on average, especially for attributes describing motions of the mouth area where the gap is almost . Feature had the highest prediction accuracy on average, which was more than higher than the last FC layer. One can also observe that the mid-level convolutional representations also featured slightly different prediction power on different attributes, e.g. outperformed on “Rosy Cheeks” but not on hair related attributes. Through this exploration, we identified the best representation for each attribute.
2.3 Benchmark evaluations
We then compared the identified best representations of each attribute with a baseline approach and the current state-of-the-art , on CelebA and LFWA datasets respectively. The results are compared in Figure 2 and the detailed recognition rates of our approach are given in Table 2. The baseline approach (denoted by + ANetin in ) used CNN trained by identities to extract features from aligned faces; the pipeline of constructing deep representations is the same as ours. The current best, denoted by “LNet+ANet” in , employed concatenated CNNs to locate faces and extract features in order to construct attribute classifiers.
One can observe that: by jointly leveraging the best representations, superior performance has been achieved over the equivalent baseline approach and even the current state-of-the-art. For some attributes with relatively lower (baseline) accuracy, e.g. “Blurry” “Oval Face”, “Wearing Necktie” and “Rosy Cheeks”, the advantage of our approach is especially significant. Among the evaluated attributes, the mid-level convolutional features dominated the best representations (only 4 best representations came from FC layers on CelebA and 1 on LFWA).
utilizing the best representations demonstrates outperforming effectiveness over the equivalent baseline method and state-of-the-art approach on both large scale datasets;
the prediction power increases along the convolutional layers of the CNN;
even the very early layer astonishingly achieved entirely comparable performance as the state-of-the-art.
Indeed, even if we utilize convolutional features on the attributes which had FC as the best representations (e.g. “Bald”), the effect on prediction performance is not detectable (which also reflects in Figure 1).
|5 o’Clock Shadow||93.34||76.90||Male||98.09||93.93|
|Arched Eyebrows||82.50||83.37||Mouth S. Open||92.56||82.13|
|Bags Under Eyes||82.24||83.63||Narrow Eyes||86.92||81.90|
|Big Lips||69.90||78.50||Pale Skin||96.69||89.07|
|Big Nose||82.64||83.53||Pointy Nose||76.17||82.57|
|Black Hair||86.04||90.90||Receding Hairline||92.14||85.47|
|Blond Hair||94.89||97.00||Rosy Cheeks||94.29||86.50|
|Bushy Eyebrows||91.89||83.67||Straight Hair||80.00||79.93|
|Double Chin||96.19||80.80||Wearing Earrings||86.74||93.67|
|Gray Hair||97.77||87.67||Wearing Necklace||85.78||89.87|
|Heavy Makeup||90.14||95.27||Wearing Necktie||94.42||82.50|
Moreover, it is worth nothing that using features at FC layers is computationally expensive due to the bottle neck between convolutional and FC layer. Thus, leveraging the mid-level representations features two inherent advantages:
very efficient inference with competitive performance;
breaking the bounds of the interconnections between convolutional and FC layers so that the CNN accepts arbitrary receptive size (as demonstrated in Section 2.4).
Recall that the training effort of a face classification CNN is almost minimal compared to state-of-the-art two-stage CNN approach and other potential end-to-end learning for attribute prediction. However, the deep hierarchical representations from the trained face classification CNN still allow superior performance. This indicates that these mid-level CNN representations have contained rich spatial information which can be utilized more in building effective (local) appearances description, attribute prediction and retrieval applications.
It is easy to expect that a number of factors, e.g. image resolution, could affect the face attribute prediction performance. A small receptive size would affect the prediction relating to small scale face traits. We found that by doubling the input size of our CNN the prediction accuracy for “Bag under eyes” increased from to with as the best representation.
Similarly, attributes that are related to the face components with deterministic distribution can have condensed representations. For instance, “arched eyebrow” always appears in the upper part of face images and “wearing necktie” appears in the lower part. We studied the condensed feature () as the representation for locally related attributes and found that the average prediction accuracy remained almost the same (89.3%). This means that we can further reduce the memory footprint of the representations and the classifiers for the attributes defined by deterministic facial components.
In this paper, we have proposed to leverage the mid-level representations from deep convolutional neural network to tackle the attribute prediction problem for faces in the wild. We used an off-the-shelf architecture and a publicly available dataset to train a plain classification network, and conduct investigations on the utility of the deep representations from various levels of the network. Although the trained network is not optimized either towards face attribute prediction or recognition, it still allows accurate attribute prediction surpassing the state-of-the-art with a noticeable margin.
Our investigations indicate that CNNs trained for face classification have implicitly learned many semantic concepts and human describable attributes have been embedded in the mid-level deep representations, which can be separated by simple classifiers. They also reveal the potential utility of the intermediate deep representations for other face related tasks. These findings promise to increase the face attribute prediction performance and to achieve multiple intelligent functions, such as face recognition, retrieval and attribute prediction, in one single deep architecture.
-  Hossein Azizpour, Ali Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. From generic to specific deep representations for visual recognition. In , pages 36–45, 2015.
-  Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Describing people: A poselet-based approach to attribute classification. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 1543–1550. IEEE, 2011.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv:1502.01852, 2015.
-  Gary B. Huang, Marwan Mattar, Tamara Berg, and Erik Learned-miller. E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, 2007.
Davis E. King.
Dlib-ml: A machine learning toolkit.Journal of Machine Learning Research, 10:1755–1758, 2009.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
-  Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In Computer Vision, 2009 IEEE 12th International Conference on, pages 365–372. IEEE, 2009.
Yan Li, Ruiping Wang, Haomiao Liu, Huajie Jiang, Shiguang Shan, and Xilin Chen.
Two birds, one stone: Jointly learning binary code for large-scale face image retrieval and attributes prediction.In Proceedings of the IEEE International Conference on Computer Vision, pages 3819–3827, 2015.
-  Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. Proceedings of the British Machine Vision, 1(3):6, 2015.
-  Ali S Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 512–519. IEEE, 2014.
-  Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823, 2015.
-  Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
-  Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1891–1898, 2013.
-  Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014.
-  Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1708, 2013.
-  Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014.
-  Ning Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev. Panda: Pose aligned networks for deep attribute modeling. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1637–1644, June 2014.
-  Ziwei Liu, Ping Luo, Xiaogang Wang and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
-  Yang Zhong, Josephine Sullivan, and Haibo Li, “Face Attribute Prediction Using Off-the-Shelf CNN Features,” In Proceedings of International Conference on Biometrics (ICB), 2016.