Face verification with mxnet on android
Convolution neural network (CNN) has significantly pushed forward the development of face recognition and analysis techniques. Current CNN models tend to be deeper and larger to better fit large amounts of training data. When training data are from internet, their labels are often ambiguous and inaccurate. This paper presents a light CNN framework to learn a compact embedding on the large-scale face data with massive noisy labels. First, we introduce the concept of maxout activation into each convolutional layer of CNN, which results in a Max-Feature-Map (MFM). Different from Rectified Linear Unit that suppresses a neuron by a threshold (or bias), MFM suppresses a neuron by a competitive relationship. MFM can not only separate noisy signals and informative signals but also plays a role of feature selection. Second, a network of five convolution layers and four Network in Network (NIN) layers are implemented to reduce the number of parameters and improve performance. Lastly, a semantic bootstrapping method is accordingly designed to make the prediction of the models be better consistent with noisy labels. Experimental results show that the proposed framework can utilize large-scale noisy data to learn a light model in terms of both computational cost and storage space. The learnt single model with a 256-D representation achieves state-of-the-art results on five face benchmarks without fine-tuning. The light CNN model is released on https://github.com/AlfredXiangWu/face_verification_experiment.READ FULL TEXT VIEW PDF
With the development of convolution neural network, more and more resear...
Benefit from large-scale training datasets, deep Convolutional Neural
To improve the discriminative and generalization ability of lightweight
Probabilistic Face Embeddings (PFE) can improve face recognition perform...
The 3D shapes of faces are well known to be discriminative. Yet despite ...
Face recognition has attracted increasing attention due to its wide rang...
We introduce the Single Stage Headless (SSH) face detector. Unlike two s...
Face verification with mxnet on android
In the last decade, convolution neural network (CNN) has become one of the most popular techniques for computer vision. Numerous vision tasks, such as image classification, object detection , face recognition [19, 24, 27], have benefited from the robust and discriminative representation learnt via CNN models. Their performances have obtained great progress and some algorithms have been applied to commercial systems. For example, the accuracy on the challenging LFW benchmark has been improved from  to [12, 16, 17, 19] by using CNN based model. Face verification is one of the most successful applications of CNN. It identifies whether two facial images are from the same person (one-to-one matching).
to extract face feature vectors and then processes these vectors by classifiers or multi-patch ensemble models. Some methods in this group are often based on strong assumptions of data distributions that may not make effect on various situations, such as Joint Bayesian and Gaussian Processing . Although multi-patch ensemble enhances robustness of features from multi-class classification, it is time consuming. The second group aims to directly optimize the verification loss for matching and non-matching pairs [3, 7, 17]. These methods potentially overcome the bottleneck of multi-class classification based networks, which may not be generalized for a new identity that does not exist in the training set. However, one limitation of these methods is that it is difficult to select training dataset for negative pairs and the threshold in the verification loss is manually determined. The last group employs a joint identification and verification constraint to optimize deep face models [12, 16, 19, 27]. Multi-task learning provides an efficient way to enhance the generalization ability of face representation. However, convergence is still challenge for multi-task based CNNs. The trade-off between identification and verification is manually determined and depends on the training set.
have achieved ultimate accuracy on the LFW dataset, computational costs are still an ongoing issue, which has been a bottleneck for face recognition systems on embedding devices or smart phones. First, the very deep CNN leads to a large model size and long computation time for feature extraction on CPU or GPU. Particularly, the usage of multiple facial patches requires much time and introduces uncertainty incurred by the automatic location of facial landmarks. Second, most of the state-of-the-art CNN models are based on the ReLU activation function that makes learnt features usually high dimensional and sparse. To obtain a low-dimensional and compact representation, one often utilizes Joint Bayesian or metric learning  to reduce the learnt high-dimensional sparse features in another independent stage. Hence, it is important to directly seek a CNN model with a small size, fast speed of feature extraction and low-dimensional representation.
This paper studies a lightened CNN framework (as shown in Fig. 1) to learn a deep face representation as feature extractors. We define a Max-Feature-Map (MFM) activation function for compact representation and feature selection as an alternative of ReLU. MFM in convolution layers is a variation of the maxout operator  in fully connected layers. The CNN model is trained on CASIA-WebFace dataset111http://www.cbsr.ia.ac.cn/english/CASIA-WebFace-Database.html and evaluated on LFW and YTF datasets. Our proposed single net model achieves 98.13% and 91.6% on LFW and YTF respectively. Moreover, the CPU time is improved from 581ms222The CPU time is tested by VGG model which is released on http://www.robots.ox.ac.uk/ vgg/software/vgg_face/ nearly 67ms to extract one face image representation on a single core i7-4790. The contributions are summarized as follows:
A new Max-Feature-Map (MFM) activation function is introduced to the convolution layers of CNN. Its learnt features are compact whereas ReLU learnt features are sparse and often high-dimensional. It is an approach of aggregate statistics to obtain more notable and discriminative nodes in both convolution and fully connected layers.
Two lightened convolution neural networks are designed for extracting face representation of face images. One contains 4 convolution layers, 4 max-pooling layers and 2 fully connected layers and totally contains about 4M parameters, the other reduces the kernel size of convolution layers and employs Network in Network (NIN) between convolution layers. These configurations not only ensure the generalization but also make great improvements on speed and storage space.
The proposed lightened CNN models obtain comparable performance on the LFW and YTF datasets. The size of model file is about 20-30MB and the CPU time of extracting face feature vector based on CNN is nearly 67ms, which has significant potential to be deployed on real-time applications.
The paper is organized as follows. In Section 2, we briefly review the CNN based face representation methods. Section 3 describes the proposed lighten CNN framework. Finally, we present our experimental results in Section 4 and conclude in Section 5.
Current methods for face recognition are often designed by CNN to obtain a robust face feature extractor. To the best of our knowledge, DeepFace  is the first one to train CNN by 4.4M face images as a feature extractor for face verification tasks. It employs a 3D alignment for data pre-processing and its network contains convolution and local connected layers. It achieves 97.35% accuracy on LFW with 4096-d feature vectors. As the extension of DeepFace, Web-Scale  applied a semantic bootstrapping method to select an efficient training set from a large dataset. It certificates that high dimensional feature vectors are not necessary for face recognition problems because the low dimensional features of Web-Scale can outperform DeepFace. And  also discuss stabler protocol  of the LFW benchmark, which can indicate the robustness of face features more representatively.
To further improve accuracy, Sun et al. [19, 21] resorts to a multi-patch ensemble model. An ensemble of 25 CNN models is trained on different local patches and Joint Bayesian is applied to obtain a robust embedding space. Verification loss and classification loss are further combined to increase the interclass distance and decrease the intra-class distance. Compared with DeepFace, they did not use 3D face alignment and trained multiple CNN models on 0.2M face images. Their final performance obtains 99.47%  on LFW.
Recently, triplet loss is introduced into CNN, which leads to a new method named FaceNet . FaceNet is trained on totally about 100-200M face images with 8M face identities. Since the selection of triplet pairs is important for accuracy, FaceNet presents a novel online triplet mining method for training triplet-based CNN and achieves good performance (99.63%). Then Parkhi et al.  combined the very deep convolution neural network  and the triplet embedding. The input of their model is
as imagenet task. They trained the CNN model on 2622 identities of 2.6M images collected from Internet and then fine-tuned the model via triplet-based metric learning method like FaceNet. The classification-based net obtains 97.27% and the deep embedding model achieves 98.95% on LFW.
Besides, by taking advantages of multiple CNNs and triple-based method, Liu et al.  proposed a two-stage algorithm that combined multi-patch deep CNN and deep metric learning. They cropped 7 local patches from face images according to the facial landmarks and trained CNN models, contained 7 convolution layers, separately. Then a deep metric learning algorithm on triplets is used to reduce the dimension of features and enhance discrimination. The ensemble model is targeting the accuracy of 99.77%.
Although these CNN based methods have achieved ultimate accuracy on the LFW dataset, their computational costs are still high due to deep architectures and multiple local facial patches. Hence, it is important to seek a CNN model with a small size, fast speed of feature extraction and low dimensional representation for real-time applications.
In this section, we first define the compact Max-Feature-Map activation function for CNN and then introduce the framework of our deep face representation models.
Sigmoid or Tanh activation function is a nonlinear activation for neural networks and often leads to robust optimization during DNN training . But it may suffer from a vanishing gradient when lower layers have gradients of nearly 0 because higher layer units are nearly saturate at -1 or 1. The vanishing gradient may lead to a slow convergence or a poor local optima for convolution neural networks.
To overcome vanishing gradient, the Rectified linear unit(ReLU)  offers a sparse representation. However, ReLU is at a potential disadvantage during optimization because the value is 0 if the unit is not active. It might lead to the loss of some information especially for the first several convolution layers because these layers are similar to Gabor filter which both positive and negative responses are respected.
To alleviate this problem, the leaky rectified linear (Leaky ReLU) , parametric rectified linear (PReLU)  and randomized rectified linear (RReLU)  are proposed. Fig. 2 shows the difference among them. However, these activation functions are simple piece-wise linear activation functions, therefore, it can not represent the features efficiently in some cases.
In order to make the representation compact instead of sparsity in ReLU, we propose the Max-Feature-Map(MFM) activation function which is inspired by maxout networks . Given an input convolution layer , as is shown in Fig. 3, the Max-Feature-Map activation function can be written as
where the channel of the input convolution layer is , , . As is shown in Eq.(1), the output via MFM activation function belongs to .
According to the Eq.(1), the gradient of this activation function can be shown as
As is shown in Eq.(3), the
The MFM activation function is not a normal single-input-single-output function such as sigmoid or ReLU, while it is the maximum between two convolution feature map candidate nodes. The MFM activation function utilizes an approach of aggregate statistics and can not only obtain a compact representation, but also obtain sparse gradients which achieve variable selection and dimension reduction. Moreover, the MFM activation function can also be treated as the sparse connection between two convolution layers, which encodes the information sparsely into a feature space.
For our proposed lighten CNN-based framework shown in Fig. 1, we discuss two different architectures. The deep model A is constructed by 4 convolution layers, Max-Feature-Map activation functions, 4 max-pooling layers and 2 fully connected layers which is inspired from AlexNet . The detail parameter setting is presented in Table 1.
The deep model B contains 5 convolution layers, 4 Network in Network (NIN) layers , Max-Feature-Map activation functions, 4 max-pooling layers and 2 fully connected layers which the detail is presented in Table 1. The model B with NIN and small convolution kernel size are from  because NIN can do feature selection between convolution layers and small convolution kernel can reduce the number of parameters for the model.
The input image is gray-scale face image from CASIA-WebFace dataset. We crop each input image randomly into patch as the input of the first convolution layer for training. Each convolution layer of network A and B is combined with two independent convolution parts calculated from the input. The Max-Feature-Map activation function and max pooling layer are used later. The fc1 layer is a 256-dimensional face representation. And the fc2 layer is used as the input of the softmax cost function and is set to the number of CASIA-WebFace dataset identities(10,575). Besides, the proposed network A has 3,961K parameters and network B has 3,244K parameters which are both smaller than DeepFace, WebFace and VGG models.
In this section, we evaluate our lightened CNN models on the LFW and YTF datasets. Besides, we analyze the performance of MFM activation and the speed of our proposed models compared with other CNN methods. The network A model is released on my github333https://github.com/AlfredXiangWu/face_verification_experiment and the network B model will be released soon.
The CASIA-WebFace dataset is used to train our lightened convolution neural network. It contains 493,456 face images of 10,575 identities and all the face images are converted to gray-scale and normalized to via landmarks as shown in Fig. 4. The normalized face image is shown in Fig. 4. According to the 5 facial points extracted by  and manually adjusted, we rotate two eye points horizontally which can overcome the pose variations in roll angle. The distance between the midpoint of eyes and the midpoint of mouth, as well as the y axis of midpoint of eyes, will be used for facial image normalization, because the distance between the midpoint of eyes and the midpoint of mouth is relative invariant to pose variations in yaw angle.
The evaluation is performed on the LFW dataset444http://vis-www.cs.umass.edu/lfw/ and the YTF dataset555http://www.cs.tau.ac.il/ wolf/ytfaces/ in detail. LFW contains 13,233 images of 5,749 people for face verification and YTF contains 3,425 videos of 1,595 different people. And all the images in the LFW and YTF datasets are processed by the same pipeline as the training dataset and normalized to . The details of face image normalization setting is shown in Table 2
|WebFace+Joint Bayes ||1||97.30%||-||unsupervised||-||-|
|WebFace+Joint Bayes ||1||97.73%||80.26%||unrestricted||-||-|
|Our model A||1||97.77%||84.37%||unsupervised||84.79%||63.09%|
|Our model B||1||98.13%||87.13%||unsupervised||89.21%||69.46%|
To train the convolution network A and B, we randomly select one face image from each identity as the validation set and the other images as the training set. The open source deep learning frameworkCaffe  is used for training the model.
The input for the lightened CNN models is the gray-scale face image and we crop the input image into randomly and mirror it. These data augmentation methods can improve the generalization of the convolution neural network and overcome the overfitting . Dropout is also used for fully connected layers and the ratio is set to 0.7.
Moreover, the momentum is set to 0.9, and the weight decay is set to 5e-4 for convolution layer and fully connected layer except the fc2 layer. It is obvious that the fc1 fully connected layer is the face representation used for face verification tasks. However, the parameters from fc1 layer to fc2 layer is very large and is not used for feature extractor. Therefore, it might lead to overfitting for learning the large fully-connected layer parameters. To overcome it, we set the weight decay of fc2 layer to 5e-3.
The learning rate is set to 1e-3 initially and reduced to 5e-5 gradually. The parameter initialization for convolution is Xavier and Gaussian is used for fully-connected layers. Moreover, the deep model is trained on GTX980 for 2 weeks.
For the verification protocol, face images are divided in 10 folds which contain different identities and 600 face pairs. There are two evaluation settings about LFW training and testing: restricted and unrestricted. In restricted setting, the pre-defined image pairs are fixed by the author  (each fold contains the 5400 pairs for training and the 600 pairs for testing). And in unrestricted setting, the identities within each fold for training is allowed to be much larger.
For the probe-gallery identification testing, there are two new protocols called the close set task and the open set task on LFW. (i) For the close set identification task, the gallery contains 4,249 identities, each with only a single face image, and the probe set contains 3,143 face images belonging to the same set of identities. The performance is measured by Rank-1 identification accuracy. (ii) For the open set identification task, the gallery set includes 3,143 images of 596 identities. The probe set includes 10,090 images which is constructed by 596 genuine probes and 9,494 impostor ones. The accuracy is evaluated by the Rank-1 Detection and Identification Rate (DIR), which is genuine probes matched in Rank-1 at a 1% False Alarm Rate (FAR) of impostor ones that are not rejected.
We test the performance of our model A and B with cosine similarity. As shown in Table3, the model A achieves 97.77% and model B obtains 98.13% on the LFW dataset with unsupervised setting666The unsupervised setting means that the model is not trained on LFW in supervised way.. The results of our model A and B on LFW verification protocol outperform those of DeepFace , DeepID2 , WebFace  and VGG  for single net. For the probe-gallery identification protocol, our model performance also outperforms the results of DeepFace and VGG777The VGG performance on the probe-gallery identification protocol are evaluated based on VGG released model. with 89.21% and 69.46%, respectively.
Besides, our dataset is inferior to DeepFace, Google and VGG. Their training sets contain millions of images while CASIA-WebFace only includes 0.5M images. Considering the limitation of the GPU computation resources and the speed on practical systems, we don’t train several networks and perform model ensemble to improve face verification performance.
To evaluate the generalization of our lightened CNN models, we also test them on the Youtube Face dataset. Due to low resolution and motion blur, the quality of images in the YTF dataset is worse than LFW. For the evaluation protocol, YTF dataset is divided into 10 splits. Each split includes 250 positive pairs and 250 negative ones. We randomly select 100 samples from each video and compute the average similarities. As shown in Table 4, we obtain 91.6% on YTF dataset with unsupervised setting for a single model which outperform the results of DeepFace and WebFace.
|Our model A||1||90.72%||unsupervised|
|Our model B||1||91.60%||unsupervised|
The Max-Feature-Map (MFM) activation function is proposed to alleviate the disadvantage of the simple piece-wise functions such as ReLU and its variations. Fig. 5 presents the example of MFM activation functions over our lightened CNN models. It is obvious that the network can obtain a compact representation from the MFM activation function as shown in Fig. 5, while the gradient of MFM is sparse.
Due to the sparse gradient, on the one hand, when doing backpropagation for training CNN, the processing of stochastic gradient descent (SGD) can only make effects on the neuron of response variables. On the other hand, when extracting features for testing, the MFM can obtain more competitive nodes from previous convolution layers by activating the maximum of two feature maps. These appearances result in the properties that MFM can realize feature selection and sparse connection in our lightened CNN models.
Compared with ReLU and Max-Feature-Map, we observe in Fig. 5 that the speed of convergence for Max-Feature-Map network is slower than that for ReLU due to the complexity of the activation and the randomness of initial parameters. However, with the progress of training, the validation accuracy for Max-Feature-Map outperforms ReLU finally. Table 5 presents the performance of ReLU and MFM on LFW. Although there are little difference between ReLU and MFM on verification accuracy, the identification performance of MFM has been improved about 4%-5%, compared with ReLU.
Since most of CNN based face representation extractors are based on either the very deep convolution neural networks or multi-patch ensemble, they may be time-consuming for practical systems. In this section, we compare our lightened CNN models with the VGG released model.
As shown in Table 6, the size of our lightened CNN model is 20 times smaller than that of the VGG model, while the CPU time is about 9 times faster. The results indicate that our lightened models are potentially suitable and practical on embedding devices and smart phones for real-time applications than other complex models.
In this paper, we have developed a lightened convolution neural network framework to learn a robust face representation. We have proposed a Max-Feature-Map activation function to obtain a compact low-dimensional face representation and achieve the accuracy 98.13% on LFW by only using a single model. One advantage of our framework is that it is faster and smaller than other CNN methods. Its time to extract one face image representation is about 67ms on a single core i7-4790, and its storage only occupies 31.2MB on hard disk. Experimental results on LFW and YTF datasets show that MFM is effective for learning compact features and the proposed lightened CNN framework has practical value for real-time face recognition systems. Future work will focus on better performance of lightened CNN models.
Proceedings of the 30th International Conference on Machine Learning, pages 1319–1327, 2013.
Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 3811–3819, 2015.
Rectified linear units improve restricted boltzmann machines.In Proceedings of the 27th International Conference on Machine Learning, pages 807–814, 2010.