Pairwise Relational Networks using Local Appearance Features for Face Recognition

11/15/2018
by   Bong-Nam Kang, et al.
POSTECH
0

We propose a new face recognition method, called a pairwise relational network (PRN), which takes local appearance features around landmark points on the feature map, and captures unique pairwise relations with the same identity and discriminative pairwise relations between different identities. The PRN aims to determine facial part-relational structure from local appearance feature pairs. Because meaningful pairwise relations should be identity dependent, we add a face identity state feature, which obtains from the long short-term memory (LSTM) units network with the sequential local appearance features. To further improve accuracy, we combined the global appearance features with the pairwise relational feature. Experimental results on the LFW show that the PRN achieved 99.76 state-of-the-art accuracy (96.3 the state-of-the-art for both face verification and face identification tasks on the IJB-A and IJB-B. This work is already published on ECCV 2018.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 7

08/15/2018

Pairwise Relational Networks for Face Recognition

Existing face recognition using deep neural networks is difficult to kno...
08/17/2019

Attentional Feature-Pair Relation Networks for Accurate Face Recognition

Human face recognition is one of the most important research areas in bi...
05/15/2019

Deep Kinship Verification via Appearance-shape Joint Prediction and Adaptation-based Approach

Kinship verification aims to identify the kin relation between two given...
11/29/2017

Interpretable Facial Relational Network Using Relational Importance

Human face analysis is an important task in computer vision. According t...
11/19/2018

Contextual Face Recognition with a Nested-Hierarchical Nonparametric Identity Model

Current face recognition systems typically operate via classification in...
07/07/2021

IntraLoss: Further Margin via Gradient-Enhancing Term for Deep Face Recognition

Existing classification-based face recognition methods have achieved rem...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Face recognition in unconstrained environments is a very challenging problem in computer vision. Faces of the same identity can look very different when presented in different illuminations, facial poses, facial expressions, and occlusions. Such variations within the same identity could overwhelm the variations due to identity differences and make face recognition challenging. To solve these problems, many deep learning-based approaches have been proposed and achieved high accuracies of face recognition such as DeepFace

DeepFace , DeepID series DeepID ; DeepID2 ; DeepID2+ ; DeepID3 , FaceNet FaceNet , PIMNet PIMNet_CVPRW2017 , SphereFace SphereFace , and ArcFace ArcFace

. In face recognition tasks in unconstrained environments, the deeply learned and embedded features need to be not only separable but also discriminative. However, these features are learned implicitly for separable and distinct representations to classify between different identities without what part of the features is used, what part of the feature is meaningful, and what part of the features is separable and discriminative. Therefore, it is difficult to know what kinds of features are used to discriminate the identities of face images clearly. To overcome this limitation, we propose a novel face recognition method, called a pairwise relational network (PRN) to capture unique relations within same identity and discriminative relations between different identities. To capture relations, the PRN takes local appearance features as input by ROI projection around landmark points on the feature map. With these local appearance features, the PRN is trained to capture unique pairwise relations between pairs of local appearance features to determine facial part-relational structures and properties in face images. Because the existence and meaning of pairwise relations should be identity dependent, the PRN could condition its processing on the facial identity state feature. The facial identity state feature is learned from the long short-term memory (LSTM) units network with the sequential local appearance features on the feature maps. To more improve accuracy of face recognition, we combined the global appearance features with the relation features. We present extensive experiments on the public available datasets such as Labeled Faces in the Wild (LFW)

LFW , YouTube Faces (YTF) YTF_CVPR2011 , IARPA Janus Benchmark A (IJB-A) IJB-A_CVPR2015 , and IARPA Janus Benchmark B (IJB-B) IJB-B_CVPRW2017 and show that the proposed PRN is very useful to enhance the accuracy of both face verification and face identification.

2 Pairwise relational network

The pairwise relational network (RRN) takes a set of local appearance features on the feature map as its input and outputs a single vector as its relational representation feature for the face recognition task. The PRN captures unique and discriminative pairwise relations between different identities. In other words, the PRN captures the core unique and common properties of faces within the same identity, whereas captures the separable and discriminative properties of faces between different identities. Therefore, the PRN aims to determine pairwise-relational structures from pairs of local appearance features in face images. The relational feature

represents a latent relation of a pair of two local appearance features, and can be written as follows:

(1)

where

is a multi-layer perceptron (MLP) and its parameters

are learnable weights. is a pair of two local appearance features, and , which are -th and -th local appearance features corresponding to each facial landmark point, respectively. Each is extracted by the RoI projection which projects a region around -th landmark point in the input facial image space to a region on the feature maps space. The same MLP operates on all possible parings of local appearance features.

Figure 1: Pairwise Relational Network. The PRN captures unique and discriminative pairwise relations dependent on facial identity.

However, the permutation order of local appearance features is a critical for the PRN, since without this invariance, the PRN would have to learn to operate on all possible permuted pairs of local appearance features without explicit knowledge of the permutation invariance structure in the data. To incorporate this permutation invariance, we constrain the PRN with an aggregation function (Figure 1):

(2)

where is the aggregated relational feature, and is the aggregation function which is summation of all pairwise relations among all possible pairing of the local appearance features. Finally, a prediction of the PRN can be performed with where is a function with parameters , and are implemented by the MLP. Therefore, the final form of the PRN is a composite function as follows:

(3)

where is a set of all possible pairs of local appearance features where denotes the number of local appearance features on the feature maps.

To capture unique and discriminative pairwise relations among different identities, a pairwise relation should be identity dependent. So, we modify the PRN such that could condition its processing on the identity information. To condition the identity information, we embed a face identity state feature as the identity information in the :

(4)

To get this

, we use the final state of a recurrent neural network composed of LSTM layers and two fully connected layers that process a sequence of local appearance features:

where is a neural network module which composed of LSTM layers and two fully connected layers with learnable parameters . We train with softmaxloss function. The detailed configuration of used in our proposed method is in Appendix A.5.

Loss function

To learn the PRN, we use jointly the triplet ratio loss , pairwise loss , and softmax loss to minimize distances between faces that have the same identity and to maximize distances between faces that are of different identity. is defined to maximize the ratio of distances between the positive pairs and the negative pairs in the triplets of faces . To maximize , the Euclidean distances of positive pairs should be minimized and those of negative pairs should be maximized. Let , where is the input facial image, denote the output of a network, the is defined as follows:

(5)

where is the output for an anchor face , is the output for a positive face image , and is the output for a negative face in , respectively. is a margin that defines a minimum ratio in Euclidean space. From recent work by Kang et al. PIMNet_CVPRW2017 , they reported that although the ratio of the distances is bounded in a certain range of values, the range of the absolute distances is not. To solve this problem, they constrained by adding the pairwise loss function . is defined to minimize the sum of the squared Euclidean distances between for the anchor face and for the positive face. These pairs of and are in the triplets of faces .

(6)

The joint training with and minimizes the absolute Euclidean distance between face images of a given pair in the triplets of facs . We also use these loss functions with softmax loss jointly.

3 Experiments

We evaluated the proposed face recognition method on the public available benchmark datasets such as the LFW, YTF, IJB-A, and IJB-B. For fair comparison in terms of the effects of each network module, we train three kinds of models (model A (base model, just use the global appearance feature ), model B ( + in Eq, (3)), and model C ( + in Eq. (4)) under the supervision of cross-entropy loss with softmax PIMNet_CVPRW2017 . More detailed configuration of them is presented in Appendix A.6.

Effects of the PRN and the face identity state feature

To investigate the effectiveness of the PRN model with , we performed experiments in terms of the accuracy of classification on the validation set during training. For these experiments, we trained (Eq. (3)) and (Eq. (4)) with the face identity state feature . We achieved and accuracies of classification for and , respectively. From this evaluation, when using , we observed that the face identity state feature represents the identity property, and the pairwise relations should be dependent on an identity property of a face image. Therefore, this evaluation validated the effectiveness of using the PRN network model and the importance of the face identity state feature.

Experiments on the Labeled Faces in the Wild (LFW)

From the experimental results on the LFW (See Table 2 in Appendix B.1), we have the following observation. First, model C (jointly combined with ) beats the baseline model model A (the base CNN model, just uses ) by significantly margin, improving the accuracy from to . This shows that combination of and can notably increase the discriminative power of deeply learned features, and the effectiveness of the pairwise relations between facial local appearance features. Second, compared to model B (jointly combined with ), model C achieved better accuracy of verification ( vs. ). This shows the importance of the face identity state feature to capture unique and discriminative pairwise relations in the designed PRN model. Last, compared to the state-of-the-art methods on the LFW, the proposed model C is among the top-ranked approaches, outperforming most of the existing results (Table 2 in Appendix B.1). This shows the importance and advantage of the proposed method.

Experiments on the YouTube Face Dataset (YTF)

From the experimental results on the YTF (See Table 3 in Appendix B.2), we have the following observations. First, model C beats the baseline model model A by a significantly margin, improving the accuracy from to . This shows that combination of and can notably increase the discriminative power of deeply learned features, and the effectiveness of the pairwise relations between local appearance features. Second, compared to model B, model C achieved better accuracy of verification ( vs. ). This shows the importance of the face identity state feature to capture unique and discriminative pairwise relations in the designed PRN model. Last, compared to the state-of-the-art methods on the YTF, the proposed method model C is the state-of-the-art ( accuracy), outperforming the existing results (Table 3 in Appendix B.2). This shows the importance and advantage of the proposed method.

Experiments on the IARPA Janus Benchmark A (IJB-A)

From the experimental results (See Table 4 in Appendix B.3), we have the following observations. First, compared to model A, model C achieves a consistently superior accuracy (TAR and TPIR) on both 1:1 face verification and 1:N face identification Second, compared to model B, model C achieved also a consistently better accuracy (TAR and TPIR) on both 1:1 face verification and 1:N face identification Last, more importantly, model C is trained from scratch and achieves comparable results to the state-of-the-art (VGGFace2 VGG2Face ) which is first pre-trained on the MS-Celeb-1M dataset MS-Celeb-1M , which contains roughly 10M face images, and then is fine-tuned on the VGGFace2 dataset. It shows that our proposed method can be further improved by training on the MS-Celeb-1M and fine-tuning our training dataset.

Experiments on the IARPA Janus Benchmark B (IJB-B)

From the experimental results (See Table 5 in Appendix B.4), we have the following observations. First, compared to model A, model C (jointly combined with as the local appearance representation) achieved a consistently superior accuracy (TAR and TPIR) on both 1:1 face verification and 1:N face identification. Second, compared to model B (jointly combined with the ), model C achieved also a consistently better accuracy (TAR and TPIR) on both 1:1 face verification and 1:N face identification. Last, more importantly, model C achieved consistent improvement of TAR and TPIR on both 1:1 face verification and 1:N face identification, and achieved the state-of-the-art results on the IJB-B.

4 Conclusion

We proposed a new face recognition method using the pairwise relational network (PRN) which takes local appearance feature around landmark points on the feature maps from the backbone network, and captures unique and discriminative pairwise relations between a pair of local appearance features. To capture unique and discriminative relations for face recognition, pairwise relations should be identity dependent. Therefore, the PRN conditioned its processing on the face identity state feature embedded by LSTM networks using a sequential local appearance features. To more improve accuracy of face recognition, we combined the global appearance feature with the PRN. Experiments verified the effectiveness and importance of our proposed PRN with the face identity state feature, which achieved accuracy on the LFW, the state-of-the-art accuracy () on the YTF, and comparable results to the state-of-the-art for both face verification and identification tasks on the IJB-A and IJB-B.

Acknoledgement

This research was supported by the MSIT(Ministry of Science, ICT), Korea, under the SW Starlab support program (IITP-2017-0-00897), the “ICT Consilience Creative program” (IITP-2018-2011-1-00783), and “Development of Open Informal Dataset and Dynamic Object Recognition Technology Affecting Autonomous Driving” (IITP-2018-0-01290) supervised by the IITP.

References

  • (1) Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in

    2014 IEEE Conference on Computer Vision and Pattern Recognition

    , June 2014, pp. 1701–1708.
  • (2) Y. Sun, X. Wang, and X. Tang, “Deep learning face representation from predicting 10,000 classes,” in Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, ser. CVPR ’14, 2014, pp. 1891–1898.
  • (3) Y. Sun, X. Wang, and X. Tang, “Deep Learning Face Representation by Joint Identification-Verification,” ArXiv e-prints, Jun. 2014.
  • (4) Y. Sun, X. Wang, and X. Tang, “Deeply learned face representations are sparse, selective, and robust,” ArXiv e-prints, Dec. 2014.
  • (5) X. W. Yi Sun, Ding Liang and X. Tang, “Deepid3: Face recognition with very deep neural networks,” CoRR, vol. abs/1502.00873, 2015.
  • (6) F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 815–823.
  • (7)

    B.-N. Kang, Y. Kim, and D. Kim, “Deep convolutional neural network using triplets of faces, deep ensemble, and score-level fusion for face recognition,” in

    2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), July 2017, pp. 611–618.
  • (8) W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, pp. 6738–6746.
  • (9) J. Deng, J. Guo, and S. Zafeiriou, “ArcFace: Additive Angular Margin Loss for Deep Face Recognition,” ArXiv e-prints, Jan. 2018.
  • (10) G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech. Rep. 07-49, October 2007.
  • (11) L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2011, pp. 529–534.
  • (12) B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, M. Burge, and A. K. Jain, “Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 1931–1939.
  • (13) C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, J. Cheney, and P. Grother, “Iarpa janus benchmark-b face dataset,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), July 2017, pp. 592–600.
  • (14) Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” CoRR, vol. abs/1710.08092, 2017.
  • (15) Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in Computer Vision – ECCV 2016, 2016, pp. 87–102.
  • (16) J. Yoon and D. Kim, “An accurate and real-time multi-view face detector using orfs and doubly domain-partitioning classifier,” Journal of Real-Time Image Processing, Feb 2018.
  • (17) M. Kowalski, J. Naruniec, and T. Trzcinski, “Deep alignment network: A convolutional neural network for robust face alignment,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), July 2017, pp. 2034–2043.
  • (18) K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 770–778.
  • (19) F. Chollet et al.

    , “Keras,”

    https://github.com/fchollet/keras, 2015.
  • (20) M. Abadi et al.

    , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available:

    https://www.tensorflow.org/
  • (21)

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in

    Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, 2015, pp. 448–456.
  • (22)

    V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in

    Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, 2010, pp. 807–814.
  • (23) G. B. H. E. Learned-Miller, “Labeled faces in the wild: Updates and new reporting procedures,” University of Massachusetts, Amherst, Tech. Rep. UM-CS-2014-003, May 2014.
  • (24) D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” CoRR, vol. abs/1411.7923, 2014.
  • (25) Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in Computer Vision – ECCV 2016, 2016, pp. 499–515.
  • (26) J. Yang, P. Ren, D. Zhang, D. Chen, F. Wen, H. Li, and G. Hua, “Neural aggregation network for video face recognition,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, pp. 5216–5225.
  • (27) A. R. Chowdhury, T. Y. Lin, S. Maji, and E. Learned-Miller, “One-to-many face recognition with bilinear cnns,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2016, pp. 1–9.
  • (28) D. Wang, C. Otto, and A. K. Jain, “Face search at scale,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1122–1136, June 2017.
  • (29) J. C. Chen, R. Ranjan, A. Kumar, C. H. Chen, V. M. Patel, and R. Chellappa, “An end-to-end system for unconstrained face verification with deep convolutional neural networks,” in 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Dec 2015, pp. 360–368.
  • (30) S. Sankaranarayanan, A. Alavi, C. D. Castillo, and R. Chellappa, “Triplet probabilistic embedding for face verification and clustering,” in 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), Sept 2016, pp. 1–8.
  • (31) I. Masi, S. Rawls, G. Medioni, and P. Natarajan, “Pose-aware face recognition in the wild,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 4838–4846.
  • (32) W. AbdAlmageed, Y. Wu, S. Rawls, S. Harel, T. Hassner, I. Masi, J. Choi, J. Lekust, J. Kim, P. Natarajan, R. Nevatia, and G. Medioni, “Face recognition using deep multi-pose representations,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2016, pp. 1–9.
  • (33) J. C. Chen, V. M. Patel, and R. Chellappa, “Unconstrained face verification using deep cnn features,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2016, pp. 1–9.
  • (34) O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in British Machine Vision Conference, 2015.
  • (35) N. Crosswhite, J. Byrne, C. Stauffer, O. Parkhi, Q. Cao, and A. Zisserman, “Template adaptation for face verification and identification,” in 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), May 2017, pp. 1–8.
  • (36) F. J. Chang, A. T. Tran, T. Hassner, I. Masi, R. Nevatia, and G. Medioni, “Faceposenet: Making a case for landmark-free face alignment,” in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Oct 2017, pp. 1599–1608.

Appendix A Implementation details

a.1 Training data

We used the web-collected face dataset (VGGFace2 VGG2Face ). All of the faces in the VGGFace2 dataset and their landmark points are detected by the recently proposed face detector FD_YOON2018 and facial landmark point detector DAN_CVPRW2017 . We used landmark points for the face alignment and extraction of local appearance features. When the detection of faces or facial landmark points is failed, we simply discard the image. Thus, we discarded face images from subjects. After removing these images without landmark points, it roughly goes to M images of unique persons. We generated a validation set by selecting randomly about from each subject in refined dataset, and the remains are used as the training set. Therefore, the training set roughly has M face images and the validation set has face images, respectively.

a.2 Data preprocessing

We employ a new face alignment to align training face images into the predefined template. The alignment procedures are as follows: 1) Use the DAN implementation of Kowalski et al. by using multi-stage neural network DAN_CVPRW2017 to detect facial landmarks (Figure 2b); 2) rotate the face in the image to make it upright based on the eye positions; 3) find a center on the face by taking the mid-point between the leftmost and rightmost landmark points (the red point in Figure 2d); 4) the centers of the eyes and mouth (blue points in Figure 2d) are found by averaging all the landmark points in the eye and mouth regions; 5) center the faces in the -axis, based on the center (red point); 6) fix the position along the -axis by placing the eye center at from the top of the image and the mouth center at from the bottom of the image; 7) resize the image to a resolution of . Each pixel which value is in a range of in the RGB color space is normalized by dividing to be in a range of .

Figure 2: A face image alignment example. The original image is shown in (a); (b) shows the landmark points detected by the method in DAN_CVPRW2017 , (c) shows the landmark points aligned into the aligned image plane; and (d) is the final aligned face image, where the red circle was used to center the face image along -axis, and the blue circles denote the two points used for face cropping.

a.3 Base CNN model

The base CNN model is the backbone neural network which accepts the RGB values of the aligned face image with resolution as its input, and has

convolution filters with a stride of

in the first layer. After max pooling with a stride of , it has several 3-layer residual bottleneck blocks similar to the ResNet-101 ResNet . In the last layer, we use the global average pooling with filter in each channel and use the fully connected layer. The output of the fully connected layer are fed into softmax loss layer (Table 1).

Layer name Output size 101-layer
conv1 ,
conv2_x max pool, stride
conv3_x
conv4_x
conv5_x
average pool, 8630-d fc, softmax
Table 1: A backbone conovlutional neural network architecture.

a.4 Detailed settings in the PRN

For pairwise relations between facial parts, we first extracted a set of local appearance feature from each local region (nearly size of regions) around landmark points by ROI projection on the feature maps (conv5_3 in Table 1) in the backbone CNN model. Using this , we make () possible pairs of local appearance features. Then, we used three-layered MLP consisting of

units per layer with BN and ReLU non-linear activation functions for

, and three-layered MLP consisting of units per layer with BN and ReLU non-linear activation functions for . To aggregate all of relations from , we used summation as an aggregation function. The PRN is optimized jointly with triplet ratio loss, pairwise loss, and softmax

loss over the ground-truth identity labels using stochastic gradient descent (SGD) optimization method with learning rate

. We used mini-batches size on four NVIDIA Titan X GPUs. During training the PRN, we froze the backbone CNN model to only update weights of the PRN model.

a.5 Face identity state feature

Pairwise relations should be identity dependent to capture unique pairwise relations within same identity and discriminative pairwise relations between different identities. Based on the feature maps in the CNN, the face is divided into local regions by ROI projection around landmark points. In these local regions, we extract the local appearance features to encode the facial identity state feature . Let denote the local appearance feature of -th local region. To encode , an LSTM-based network has been devised on top of a set of local appearance features as followings:

(7)

where is a neural network module which composed of LSTM layers and two fully connected layers with learnable parameters . We train with softmax loss function (Figure 3). To capture unique and discriminative pairwise relations dependent on identity, the PRN should condition its processing on the face identity state feature . For , we use the LSTM-based recurrent network over a sequence of the local appearance features which is a set ordered by landmark points order from . In other words, there were a sequence of length per face. In , it consist of LSTM layers and two-layer MLP. Each of the LSTM layer has memory cells. The MLP consists of and units per layer, respectively. The cross-entropy loss with softmax was used for training the (Figure 3).

Figure 3: Face identity state feature. A face on the feature maps is divided into regions by ROI projection around landmark points. A sequence of local appearance features in these regions are used to encode the face identity state feature from LSTM networks.

a.6 Detailed settings in the model

We implemented the base CNN and the PRN model using the Keras framework Keras_LIB with TensorFlow Tensorflow backend. For fair comparison in terms of the effects of each network module, we train three kinds of models (model A, model B, and model C) under the supervision of cross-entropy loss with softmax PIMNet_CVPRW2017 : model A is the baseline model which is the base CNN (Table 1). model B combining two different networks, one of which is the base CNN model (model A) and the other is the (Eq. (3)), concatenates the output feature of the global average pooling layer in model A as the global appearance feature and the output of the MLP in the . is the feature of size from each face image. The output of the MLP in the is the feature of size . These two output features are concatenated into a single feature vector with size, then this feature vector is fed into the fully connected layer with units. model C is the combined model with the output of model A and the the output of the (Eq. (4)). The output of model A in model C is the same of the output in model B. The size of the output in the is same as compared with the , but output values are different. All of convolution layers and fully connected layers use batch normalization (BN) BatchNormal and rectified linear units (ReLU) ReLU as nonlinear activation functions except for LSTM laeyrs in .

Appendix B Detailed Results

b.1 Experiments on the LFW

We evaluated the proposed method on the LFW dataset LFW , which reveals the state-of-the-art of face verification in unconstrained environments. LFW dataset is excellent benchmark dataset for face verification in image and contains web crawling images with large variations in illuminations, occlusions, facial poses, and facial expressions, from different identities. Our models such as model A, model B, and model C were trained on the roughly M outside training set, with no people overlapping with subjects in the LFW. Following the test protocol of unrestricted with labeled outside data LFWTechUpdate , we test on face pairs by using a squared distance threshold to determine classification of same and different and report the results in comparison with the state-of-the-art methods (Table 2).

Method   Images   Networks   Dimension   Accuracy (%)
Human - - -
DeepFace DeepFace M
DeepID DeepID
DeepID2+ DeepID2
DeepID3 DeepID3
FaceNet FaceNet M
Learning from Scratch LFR
Center Face CenterLoss M
PIMNet PIMNet_CVPRW2017
PIMNet PIMNet_CVPRW2017
SphereFace SphereFace
ArcFace ArcFace M
model A (baseline) M
model B M
model C M
Table 2: Comparison of the number of images, the number of networks, the dimensionality of feature, and the accuracy of the proposed method with the state-of-the-art methods on the LFW.

b.2 Experiments on the YTF

We evaluated the proposed method on the YTF datasetYTF_CVPR2011 , which reveals the state-of-the-art of face verification in unconstrained environments. YTF dataset is excellent benchmark dataset for face verification in video and contains videos with large variations in illuminations, facial pose, and facial expressions, from different identities, with an average of videos per person. The length of video clip varies from to frames and average of frames. We follow the test protocol of unrestricted with labeled outside data. We test on video pairs by using a squared distance threshold to determine to classification of same and different and report the results in comparison with the state-of-the-art methods (Table 3).

Method   Images   Networks   Dimension   Accuracy (%)
DeepFace DeepFace M
DeepID2+ DeepID2
FaceNet FaceNet M
Learning from Scratch LFR
Center Face CenterLoss M
SphereFace SphereFace
NAN NAN_CVPR2017 M
model A (baseline) M
model B M
model C M
Table 3: Comparison of the number of CNNs, the number of images, the dimensionality of feature, and the accuracy of the proposed method with the state-of-the-art methods on the YTF.

b.3 Experiments on the IJB-A

We evaluated the proposed method on the IJB-A dataset IJB-A_CVPR2015 which contains face images and videos captured from unconstrained environments. It features full pose variation and wide variations in imaging conditions thus is very challenging. It contains subjects with images and videos in total, and images and videos per subject on average. In this dataset, each training and testing instance is called a ’template’, which comprises to mixed still images and video frames. IJB-A dataset provides split evaluations with two protocols (1:1 face verification and 1:N face identification). For face verification, we report the test results by using true accept rate (TAR) vs. false accept rate (FAR) (Table 4). For face identification, we report the results by using the true positive identification (TPIR) vs. false positive identification rate (FPIR) and Rank-N (Table 4). All measurements are based on a squared distance threshold.

Method 1:1 Verification TAR 1:N Identification TPIR
FAR=0.001 FAR=0.01 FAR=0.1 FPIR=0.01 FPIR=0.1 Rank-1 Rank-5 Rank-10
B-CNN B-CNN_WACV2016 - - - -
LSFS LSFS_PAMI2017 -
DCNN+metric DCNN_Metric_ICCVW2015 - - -
Triplet Similarity TripletSimilarity_BTAS2016
Pose-Aware Models POSE_AWARE_CVPR2016 - - -
Deep Multi-Pose Deep_Multi_Pose_WACV2016 -
DCNN DCNN_Fusion_WACV2016 -
Triplet Embedding TripletSimilarity_BTAS2016 -
VGG-Face VGGFACE_BMVC2015 - - -
Template Adaptation Template_ADAPT_FGR2017
NAN NAN_CVPR2017
VGGFace2 VGG2Face
model A (baseline)
model B
model C
Table 4: Comparison of performances of the proposed PRN method with the state-of-the-art on the IJB-A dataset. For verification, the true accept rates (TAR) vs. false accept rates (FAR) are reported. For identification, the true positive identification rate (TPIR) vs. false positive identification rate (FPIR) and the Rank-N accuracies are presented.

b.4 Experiments on the IJB-B

We evaluated the proposed method on the IJB-B dataset IJB-B_CVPRW2017 which contains face images and videos captured from unconstrained environments. The IJB-B dataset is an extension of the IJB-A, having subjects with K still images (including face and non-face) and K frames from videos, an average of images per subject. Because images in this dataset are labeled with ground truth bounding boxes, we only detect landmark points using DAN DAN_CVPRW2017 , and then align face images with our face alignment method. Unlike the IJB-A, it does not contain any training splits. In particular, we use the 1:1 Baseline Verification protocol and 1:N Mixed Media Identification protocol for the IJB-B. For face verification, we report the test results by using TAR vs. FAR (Table 5). For face identification, we report the results by using TPIR vs. FPIR and Rank-N (Table 5). We compare our proposed methods with VGGFace2 VGG2Face and FacePoseNet (FPN) FPN_Align_ICCVW2017 . All measurements are based on a squared distance threshold.

Method 1:1 Verification TAR 1:N Identification TPIR
FAR=0.00001 FAR=0.0001 FAR=0.001 FAR=0.01 FPIR=0.01 FPIR=0.1 Rank-1 Rank-5 Rank-10
VGGFace2 VGG2Face
VGGFace2_ft VGG2Face
FPN FPN_Align_ICCVW2017 - - -
model A (baseline, only )
model B ( + )
model C ( + )
Table 5: Comparison of performances of the proposed PRN method with the state-of-the-art on the IJB-B dataset. For verification, TAR vs. FAR are reported. For identification, TPIR vs. FPIR and the Rank-N accuracies are presented