Despite the recent noticeable advances, the security of face recognition systems is still vulnerable to Presentation Attacks (PA) with printed photos or replayed videos. To counteract PA, face anti-spoofing [25, 19] is developed and serves as a pre-step prior to face recognition.
Earlier face anti-spoofing approaches mainly adopt handcrafted features, like LBP , HoG  and SURF , to find the differences between live and spoofing faces. In , CNN was used for face anti-spoofing for the first time, with remarkable performance achieved in intra-database tests. Following their work, a number of CNN-based methods have been proposed, almost all treating face anti-spoofing as a binary (live vs. spoofing) classification problem. However, given the enormous solution space of CNN, these methods tend to suffer overfitting and poor generalizability to new PA patterns and environments. In this work, we attempt to enable an anti-spoofing system to be deployed in various environments, i.e. with good generalizability.
For CNN-based methods, an important clue to differentiate live vs. spoofing faces is the spoof pattern, including color distortion, moir pattern, shape deformation, spoofing artifacts (e.g., reflection), etc. During CNN model training, strong patterns make more contributions, and the resultant model is more discriminative for them. However, if these patterns are absent in the testing data, the performance would severely drop. The CNN-based methods tend to overfit to some strong spoof patterns and thus suffer poor generalizability . Apart from overfitting, domain shift  is also an important reason for the poor generalizability of face anti-spoofing methods. A domain here refers to a certain environment where an image is acquisited, consisting of various factors such as illumination, background, facial appearance, camera type, etc. Considering the huge diversity of real world environments, it is very common that different samples have different domains. For example, the domains of two paper attacks may be quite different even in case of the same face if reproduced with different pieces of paper (e.g. glossy vs.
rough paper). Such domain variance may lead to distribution dissimilarity of different samples in the feature space and cause the models to fail on new domains.
Based on the above observations, we propose a new Total Pairwise Confusion (TPC) loss to balance the contributions of all involved spoof patterns, and also employ a Fast Domain Adaptation (FDA) model  to narrow the distribution discrepancy of samples from different domains in the feature space. We then obtain a Generalizable Face Authentication CNN model, shorted as GFA-CNN. Different from prior methods that take face anti-spoofing as a pre-step of face authentication, our GFA-CNN works in a multi-task manner, performing simultaneously face anti-spoofing and face recognition, as shown in Fig. 1. Since the CNN layers of the two tasks share the same parameters, our model works with high efficiency.
Extensive experiments on five popular benchmarks for face anti-spoofing demonstrate the superiority of our method over the state-of-the-arts. Our code and trained models will be available upon acceptance. Our contributions are summarized as follows:
We propose a Total Pairwise Confusion (TPC) loss to effectively relieve the overfitting problems of CNN-based face anti-spoofing models to dataset-specific spoof patterns, which improves generalizability of face anti-spoofing methods.
We incorporate the Fast Domain Adaptation (FDA) model to learn more robust Presentation Attack (PA) representations, which reduces domain shift in the feature space.
We develop a multi-task CNN model for face authentication. Our GFA-CNN performs jointly face anti-spoofing and face recognition.
2 Related Work
, which are subsequently fed to a supervised classifier (e.g., SVM, LDA) for binary classification. However, such handcrafted features are very sensitive to different illumination conditions, camera devices, specific identities, etc. Though noticeable performance achieved under the intra-dataset protocol, the sample from a different environment may fail the model. In order to obtain features with better generalizability, some approaches leverage temporal information, e.g. making use of the spontaneous motions of the live faces, such as eye-blinking  and lip motion . Though these methods are effective against photo attacks, they become vulnerable when attackers simulate these motions through a paper with eye/mouth positions cut.
Recently, deep learning based methods[27, 17] have been proposed to address face anti-spoofing. They use CNNs to learn highly discriminative representations by taking face anti-spoofing as a binary classification problem. However, most of them easily suffer overfitting. Current publicly available face anti-spoofing datasets are too limited to cover various potential spoofing types. A very recent work  by Liu et al. leverages the depth map and rPPG signal as auxiliary supervision to train CNN instead of treating face anti-spoofing as a simple binary classification problem in order to avoid overfitting. Another critical issue for face anti-spoofing is domain shift. To bridge the gap between training and testing domains,  generalizes CNN to unknown conditions by minimizing the feature distribution dissimilarity across domains, i.e. minimizing the Maximum Mean Discrepancy distance among representations.
To our best knowledge, almost all previous works take face anti-spoofing as a pre-step prior to face recognition and address it as a binary classification problem. Compared with previous literature, we solve face anti-spoofing and face recognition at one shot. A most related work to ours is , which proposed a two-tier framework to ensure the authenticity of the user to the recognition system, namely, monitoring whether the user has passed the biometric system as a live or spoofing one. It performs authentication based on fingerprint, palm vein print, face, etc., with two separated tiers: the anti-spoofing is powered by CNN learned representations while the recognition is based on pre-defined handcrafted features like ORB points.
Different with , we build our GFA-CNN in a multi-task manner, our framework can recognize the identity of a given face, and meanwhile judge whether the face is a live or spoofing one. It is worth mentioning that for face recognition, our method achieves single-model accuracy up to 97.1% on the LFW database , which is even comparable to state-of-the-arts.
3 Generalizable Face Authentication CNN
3.1 Multi-Task Network Architecture
The proposed Generalizable Face Authentication CNN (GFA-CNN) is able to jointly address face recognition and face anti-spoofing in a mutual boosting way. The network has two branches: the face anti-spoofing branch and the face recognition branch. Each branch consists of 5 blocks of CNN layers and 3 fully connected (FC) layers, and each block contains 3 CNN layers. The parameters are shared between these two branches. The face anti-spoofing branch is trained by minimizing TPC loss and face anti-spoofing loss (Anti-loss), while the face recognition branch is trained by optimizing face recognition loss (Recg-loss). The anti-spoofing branch takes as input raw face images with background, while the recognition branch takes cropped faces as input. Before fed to the face anti-spoofing branch, the training images are transferred to a target domain by a given target-domain image. In testing phase, each query image is transferred to the target domain and then propagated forward the network.
The CNN blocks are structured the same with the convolution part of VGG16. Before training, the CNN blocks are first trained on the VGG-face dataset to obtain fundamental weights for face recognition. The FC layers of face anti-spoofing and face recognition branches have the same structure except for the output dimension of the last FC layer. The face anti-spoofing branch takes 2 dimensions for the last FC layer, while the dimensions of the last FC layer in the face recognition branch depend on the number of subjects involved in training. The overall objective function is
where and are the cross entropy losses for face anti-spoofing and face recognition respectively, is the Total Pairwise Confusion (TPC) loss, and and are the weighting parameters among different losses.
3.2 Total Pairwise Confusion Loss
In order to learn Presentation Attack (PA) representations that are adaptable to varying environment conditions, we propose a novel Total Pairwise Confusion (TPC) loss. Our inspiration comes from the pairwise confusion (PC) loss  that tackles the overfitting issue in fine-grained visual classification by intentionally introducing confusion in the feature activations. We modify their confusion implementation to make it applicable to the face anti-spoofing task. Our TPC loss is defined as
where and are two randomly selected images (sample pair), is the total number of sample pairs involved in training and denotes the representations of the second fully connected layer of the face anti-spoofing branch.
differs from the original PC loss in two-fold: 1) TPC loss minimizes the distribution distance of a random sample pair from the training set, rather than the sample pair from two different categories, to force CNN to learn slightly less discriminative features. 2) We minimize the Euclidean distance in the feature space while the original PC loss minimizes the distance in the probability space (output of softmax) to make samples in the same pair have a similar conditional probability distribution.
Our modifications are based on below considerations: 1) With face anti-spoofing taken as a binary classification issue, confusion across categories would not excessively affect the discriminability of the PA feature on differentiating live vs. spoofing samples. 2) Face samples related to the same subject would usually cluster in the feature space, and implementing confusion on all samples could compact and homogenize the whole feature distribution (see Fig. 3), thus benefiting generalization performance. 3) As a binary classification problem of simpler structure, regularizing the model within the feature space would be more useful than imposing regularization within the output probabilistic space.
Our can effectively improve the generalizability of PA representations. This can be understood as follows. Suppose there are components in the PA representations, each corresponding to one spoof pattern, which is called a Spoof-pattern Specific Feature (SSF) in this work. As shown in Fig. 4, different SSFs contribute differently to the final decision. If we define the feature for a live and a spoofing sample as and , respectively, where is the SSF of the live sample and is the SSF of the spoofing sample. The SSFs are ranked based on their importance to the classification of live vs. spoofing. On one hand, aims to enlarge the distance between and for better discrimination. On the other hand, attempts to narrow the difference between and . As contributes the most to the differentiation of live and spoofing samples, it will be impaired the most by . However, the contributions of less important SSFs, such as and , will be enhanced by to offset the impaired discriminative ability. In this trade-off game, the contributions of all SSFs tend to be equalized, meaning more spoof patterns are involved in the decision rather than just a couple of strong spoof patterns specific to the training set. This could effectively alleviate overfitting risks. If some spoof patterns disappear in testing, a fair decision can still be achieved by other patterns, ensuring CNN would not overfit to some specific features.
3.3 Fast Domain Adaptation
Besides the proposed TPC loss that balances the contribution of each spoof pattern, we also apply FDA to reduce domain shift in the feature space to further improve the generalizability of our framework.
Generally, an image contains two components: content and appearance . The appearance information (e.g., colors, localised structures) makes up the style of images from a certain domain and is mostly represented by features in the bottom layers of CNN 
. For face anti-spoofing, the domain variance among face samples may introduce the distribution dissimilarity in the feature space and hurt anti-spoofing performance. Here, we employ the FDA to alleviate negative effects brought by domain changes. The FDA consists of an image transformation networkthat generates a synthetic image from a given image :
, and a loss networkthat computes content reconstruction loss and domain reconstruction loss .
Let be the layer of with the shape of . The content reconstruction loss penalizes the output image when it deviates in content from the input . We thus minimize the Euclidean distance between the feature representations of and :
The domain reconstruction loss enables the output image to have the same domain with the target-domain image . We then minimize the squared Frobenius norm of the difference between the Gram matrices of and :
The Gram matrix is computed by reshaping into a matrix , . Then the optimal image is generated by solving the following objective function:
where is the optimal parameters of network , is the content image, , is the target-domain image, and and are scalars. By solving Eqn. (5), is transferred to , preserving the content of with the domain of .
Fig. 5 shows some of our domain transferred samples. The target-domain image is sampled from the training data. Detailed analysis on the feature diversity between domains w/ and w/o FDA is provided in Sec. 4.2.
4.1 Experimental Setup
. CASIA-FASD and MSU-MFSD are small datasets, containing 50 and 35 subjects, respectively. Oulu-NPU and SiW are high-resolution databases published very recently. Oulu-NPU contains 4 testing protocols: Protocol 1 evaluates the environment condition variations; Protocol 2 examines the influences of different spoofing mediums; Protocol 3 estimates the effects of different input cameras; Protocol 4 considers all the challenges above. We conduct intra-database tests on MSU-MFSD and Oulu-NPU, respectively. Cross-database tests are performed between CASIA-FASDvs. Replay-Attack and MSU-MFSD vs. Replay-Attack, respectively. The face recognition performance is evaluated on SiW, which contains 165 subjects with large variations in poses, illumination, expressions (PIE), and different distances from subject to camera. The LFW, the most widely used benchmark for face recognition, is also used to evaluate the face recognition performance.
The proposed GFA-CNN is implemented with TensorFlow. We use Adam optimizer with a learning rate beginning at 0.0003 and decaying half after every 2,000 steps. The batch size is set as 32. and in Eqn. (1) are set as 0.1 and , respectively. All experiments are performed according to the protocols provided in the datasets. The CNN layers are pre-trained on the VGG-face dataset . For data balance, we triple the live samples in the training set of CASIA-FASD, MSU-MFSD and Replay-Attack with horizontal and vertical flipping, while doubling the live samples in the training set of SiW by just flipping horizontally.
We have two evaluation protocols, intra-test and cross-test, which test samples from and not from the domain of the training set, respectively. We report our results with the following metrics. Intra-test evaluation: Equal Error Rate (EER), Attack Presentation Classification Error Rate (APCER), Bona Fide Presentation Classification Error Rate (BPCER) and, ACER=(APCER+BPCER)/2. Cross-test evaluation: HTER.
|MFSD||Replay||MFSD Replay||Replay MFSD|
4.2 Ablation Study
We perform ablation analysis to reveal the role of TPC loss and FDA in our framework. We retrain the proposed network by adding/ablating TPC and FDA. As shown in Tab. 1, if TPC is removed, the HTER of intra-test on MFSD drops by 2.9% (w/ FDA) and 4.1% (w/o FDA), respectively. Since Replay-Attack is usually free of severe overfitting, it is reasonable to see the improved performance is not significant when using FDA, 0.3% (w/ FDA) and 0.6% (w/o FDA) on HTER.
For cross-test, if TPC is ablated, the HTER dramatically decreases by over 10% for MFSD Replay111The acronym means training on database ” and testing on database ”., and over 8% for Replay MFSD, no matter FDA is used or not. The best cross-test result is achieved by using both TPC and FDA, indicating FDA can further improve the generalizability of the proposed method.
To evaluate the feature diversity between domains w/ and w/o FDA, we calculate the feature divergence via symmetric KL divergence. Similar to , we denote the mean value of a channel from the feature embedding of CNN as
. Given a Gaussian distribution of, with mean and variance , the symmetric KL divergence of this channel between domain A and B is
Denote as the symmetric KL divergence of the channel. Then the average feature divergence of the layer is defined as
where C is the channel number of this layer. This metric measures the distance between the feature distributions of domain A and B. We calculate the feature divergence of each layer in a CNN model for comparison. In particular, we randomly select 5,000 face samples from MSU-MFSD and Replay-Attack, respectively. Each dataset is considered as one domain. These samples are then fed to a pre-trained VGG16  model to calculate the KL divergence at each layer following Eqn. (8). The comparison results are shown in Fig. 6. As can be seen, with the FDA, the feature divergence between MSU-MFSD and Replay-Attack is significantly reduced.
|LBP + SVM baseline||14.7|
|DoG + LBP + SVM baseline||23.1|
|IDA + SVM ||8.58|
|Color LBP ||10.8|
|Color texture ||4.9|
|Color SURF ||2.2|
|Prot.||Methods||APCER(%)||BPCER (%)||ACER (%)|
|Color LBP ||37.9||35.4||44.8||33.0||37.8|
|Color Tex. ||30.3||37.7||33.9||34.1||34.0|
|Color SURF ||26.9||23.2||29.7||31.8||27.9|
4.3 Face Anti-spoofing Evaluation
We perform intra-test on MSU-MFSD and Oulu-NPU. Tab. 2 shows the comparisons of our method with other state-of-the-art methods on MSU-MFSD. For Oulu-NPU, we refer to the face anti-spoofing competition results in  and use the best two for each protocol for comparison. All results are reported in Tab. 3.
As shown in Tab. 2, GFA-CNN achieves the EER of 7.5%, ranking the among all the compared methods. This result is satisfactory considering GFA-CNN is not designed blindly to pursue high performance in the intra-test setting. In our experiments, we find the proposed TPC loss may slightly decrease the intra-test performance, mainly because TPC loss impairs the contributions of several strongest SSFs w.r.t the training datasets. The weakening of these dataset-specific features may in turn affect the intra-test performance (however, they may improve the performance in cross-test). According to Tab. 3, our method achieves the lowest ACER in 3 out of 4 protocols. For the most challenging protocol 4, we achieve the ACER of 8.9%, which is 1.1% lower than the best performer.
To demonstrate the strong generalizability of GFA-CNN, we perform cross-test on CASIA-FASD, Replay-Attack, and MSU-MFSD by comparing with other state-of-the-arts. We adopt the most widely used cross-test settings: CASIA-FASD vs. Replay-Attack and MSU-MFSD vs. Replay-Attack, and report comparison results in Tab. 4. As can be seen, GFA-CNN achieves the lowest HTERs in cross-test: CASIA Replay, MFSD Replay and Replay MFSD. Especially for Replay MFSD, GFA-CNN reduces the cross-testing HTER by 8.3% compared with the best state-of-the-art.
However, we also observe GFA-CNN has a relatively worse HTER compared with the best method on Replay Attack CASIA-FASD. This is probably due to the “quality degradation” by FDA when the resolution of a source-domain image to be transferred is much higher than that of the target-domain image. During the cross-testing on Replay-Attack CASIA-FASD, the target-domain image is selected from Replay-Attack with a low-resolution of . However, CASIA-FASD contains quite a number of images with high-resolution of . Such a “resolution gap” leads to a “quality degradation” of FDA, as shown in the rightmost image in Fig. 7.
4.4 Face Recognition Evaluation
We further evaluate the face recognition performance of our GFA-CNN on SiW and LFW. Since our method is not targeted specifically at face recognition, we only adopt VGG-16 as the baseline. On LFW, we follow the provided protocol to perform testing. On SiW we use 90 subjects for training and the other 75 subjects for testing, which is its default data splitting. This dataset also provides a frontal legacy face image corresponding to each subject. At the testing phase, we select the legacy image w.r.t each subject of the testing set as the gallery faces, and use all images in the testing set (including both live and spoofing) as the probe faces.
The ROC curves of face verification are shown in Fig. 8. As can be observed, GFA-CNN achieves competitive results to VGG16 on LFW, 97.1% and 97.6%, respectively. However, when testing on SiW, the declined accuracy of GFA-CNN is much lower than that of VGG16: the accuracy of GFA-CNN reduces by 4.5%, while VGG16 drops by 14%. The degraded performance is mainly due to face reproduction by spoofing mediums, in which some of the finer facial details might be lost. However, GFA-CNN still achieves satisfactory performance compared with VGG16. This is mainly because the face anti-spoofing and face recognition tasks mutually enhance each other, making the representations learned for face recognition less sensitive to spoof patterns.
4.5 Discussions on Multi-task Setting
In this subsection, we investigate how the multi-task learning affects model performance for face anti-spoofing. We retrain our model without the face recognition branch, keep hyper-parameters unchanged and evaluate with the same protocol as the GFA-CNN. From the experiments, we observe the multi-task training slightly decreases the intra-test performance of face anti-spoofing (dropping 2.5% and 0.3% on MSU-MFSD and Replay-Attack, respectively). This is reasonable, since the single model learns to perform two different tasks. However, two advantages are achieved compared with the single task training. Firstly, the training process becomes more stable with the Anti-loss decreasing gradually, rather than dropping sharply after some steps by single task training, suggesting multi-task setting can help overcome overfitting. Secondly, as shown in Fig. 8, multi-task training helps learn face representations less sensitive to spoof patterns for face recognition. This mainly benefits from sharing parameters in the convolutional layers, giving more generic fusion features.
This paper presents a novel CNN model to jointly address face recognition and face anti-spoofing in a mutual boosting way. In order to learn more generalizable Presentation Attack (PA) representations for face anti-spoofing, we propose a novel Total Pairwise Confusion (TPC) loss to balance the contribution of each spoof pattern, preventing the PA representations from overfitting to dataset-specific spoof patterns. The Fast Domain Adaptation (FDA) is also incorporated into our framework to reduce distribution dissimilarity of face samples from different domains, further enhancing the robustness of PA representations. Extensive experiments on both face anti-spoofing and face recognition datasets show that our GFA-CNN achieves not only superior performance for face anti-spoofing on cross-tests, but also high accuracy for face recognition.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,
S. Ghemawat, G. Irving, M. Isard, et al.
Tensorflow: a system for large-scale machine learning.In OSDI, volume 16, pages 265–283, 2016.
-  Z. Boulkenafet, J. Komulainen, Z. Akhtar, A. Benlamoudi, D. Samai, S. E. Bekhouche, A. Ouafi, F. Dornaika, A. Taleb-Ahmed, L. Qin, et al. A competition on generalized software-based face presentation attack detection in mobile scenarios. In IJCB, pages 688–696, 2017.
-  Z. Boulkenafet, J. Komulainen, and A. Hadid. Face anti-spoofing based on color texture analysis. In ICIP, pages 2636–2640, 2015.
-  Z. Boulkenafet, J. Komulainen, and A. Hadid. Face spoofing detection using colour texture analysis. T-IFS, 11(8):1818–1830, 2016.
Z. Boulkenafet, J. Komulainen, and A. Hadid.
Face antispoofing using speeded-up robust features and fisher vector encoding.IEEE Signal Processing Letters, 24(2):141–145, 2017.
-  Z. Boulkenafet, J. Komulainen, and A. Hadid. On the generalization of color texture-based face anti-spoofing. Image and Vision Computing, 77:1–9, 2018.
-  Z. Boulkenafet, J. Komulainen, L. Li, X. Feng, and A. Hadid. Oulu-npu: A mobile face presentation attack database with real-world variations. In FG, pages 612–618, 2017.
-  I. Chingovska, A. Anjos, and S. Marcel. On the effectiveness of local binary patterns in face anti-spoofing. In BIOSIG, 2012.
-  T. de Freitas Pereira, A. Anjos, J. M. De Martino, and S. Marcel. Can face anti-spoofing countermeasures work in a real world scenario? In ICB, pages 1–8, 2013.
-  A. Dubey, O. Gupta, P. Guo, R. Raskar, R. Farrell, and N. Naik. Pairwise confusion for fine-grained visual classification. In ECCV, pages 70–86, 2018.
-  L. Engstrom. Fast style transfer. https://github.com/lengstrom/fast-style-transfer/, 2016.
-  G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In ECCVW, 2008.
J. Johnson, A. Alahi, and L. Fei-Fei.
Perceptual losses for real-time style transfer and super-resolution.In ECCV, pages 694–711, 2016.
-  A. Jourabloo, Y. Liu, and X. Liu. Face de-spoofing: Anti-spoofing via noise modeling. arXiv preprint arXiv:1807.09968, 2018.
-  K. Kollreider, H. Fronthaler, M. I. Faraj, and J. Bigun. Real-time face detection and motion analysis with application in liveness assessment. T-IFS, 2(3):548–558, 2007.
-  J. Komulainen, A. Hadid, and M. Pietikainen. Context based face anti-spoofing. In BTAS, pages 1–8, 2013.
H. Li, P. He, S. Wang, A. Rocha, X. Jiang, and A. C. Kot.
Learning generalized deep feature representation for face anti-spoofing.T-IFS, 13(10):2639–2652, 2018.
-  H. Li, W. Li, H. Cao, S. Wang, F. Huang, and A. C. Kot. Unsupervised domain adaptation for face anti-spoofing. T-IFS, 13(7):1794–1809, 2018.
-  Y. Liu, A. Jourabloo, and X. Liu. Learning deep models for face anti-spoofing: Binary or auxiliary supervision. In CVPR, pages 389–398, 2018.
-  G. Pan, L. Sun, Z. Wu, and S. Lao. Eyeblink-based anti-spoofing in face recognition from a generic webcamera. 2007.
-  X. Pan, P. Luo, J. Shi, and X. Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. arXiv preprint arXiv:1807.09441, 2018.
-  O. M. Parkhi, A. Vedaldi, A. Zisserman, et al. Deep face recognition. In BMVC, volume 1, page 6, 2015.
-  M. Sajjad, S. Khan, T. Hussain, K. Muhammad, A. K. Sangaiah, A. Castiglione, C. Esposito, and S. W. Baik. Cnn-based anti-spoofing two-tier multi-factor authentication system. Pattern Recognition Letters, 2018.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
-  X. Tan, Y. Li, J. Liu, and L. Jiang. Face liveness detection from a single image with sparse low rank bilinear discriminative model. In ECCV, pages 504–517, 2010.
-  D. Wen, H. Han, and A. K. Jain. Face spoof detection with image distortion analysis. T-IFS, 10(4):746–761, 2015.
-  J. Yang, Z. Lei, and S. Z. Li. Learn convolutional neural network for face anti-spoofing. arXiv preprint arXiv:1408.5601, 2014.
-  Z. Zhang, J. Yan, S. Liu, Z. Lei, D. Yi, and S. Z. Li. A face antispoofing database with diverse attacks. In ICB, pages 26–31, 2012.