Automatic fingerprint recognition systems have been widely adopted to perform reliable and highly accurate biometric identification. Compared to other biometric traits, such as iris, the fingerprint has a unique superiority of being collected indirectly from crime scenes from latent friction ridge impressions. Fingerprint samples can be categorized into three main groups based on the acquisition techniques such as: inked, live-scan, or latent samples. The inked and live-scan samples are considered as clean samples for which users leave impressions intentionally in access control or authentication scenarios. In addition, an agent, or the acquisition process of the system itself, can monitor quality of the samples, and guide users to leave appropriate fingerprints. In past decades, algorithms for preprocessing and matching clean fingerprints have advanced rapidly, resulting in the development of numerous and varied commercial fingerprint recognition systems.
In contrast, latent fingerprints are the marks of fingers unintentionally left on the surface of an object in a crime scene. Typically, a latent fingerprint is a ‘noisy’ image with a notable missing area, therefore containing a lower amount of ridge information (i.e. minutiae) compared to inked or live-scan fingerprints. Processing latent fingerprints is a complex and challenging problem due to the under-determined properties of the problem, and presence of many disturbing factors introduced by background objects or patterns on the substrate or surface, the force and torque involved in depositing the latent print, etc. Commercial and state-of-the-art methods for recognizing the inked or live-scan fingerprints often fail to process latent samples, even in the preprocessing stage . Therefore, various approaches have been proposed in the literature to tackle the problem of latent [15, 34, 33, 5, 1] and distorted [3, 26, 25] fingerprints.
Enhancing latent fingerprints often leads to optimizing a cost function that measures the quality of reconstruction by comparing reconstructed information and their ground truths. Based on the type of the reconstructed information, latent fingerprint reconstruction methods can be divided into two main categories: ridge-base and minutiae-based methods.
, algorithms try to predict the ridge information, which can be the orientation map or the ridge pattern itself, and minimize the similarity between the generated information and their ground truths. Then minutiae information can be extracted from the predicted ridge information. These methods are optimized to predict the ridge information without estimating the local quality of the input samples. For parts of the input latent fingerprint that there exists some information, typically these algorithms produce useful results. However, for the parts in which there is a severe distortion, not only these algorithms can not predict the missing information, but also, they can destroy the ID information by generating erroneous minutiae. Figure1(c) shows a reconstructed ridge map for a latent fingerprint with severe distortion. As the reconstructed ridge map indicates, algorithm  generates meaningful ridge information for parts that contain some ridge information in the input latent sample, but for other parts it produces random ridge patterns which drastically decreases the matching score.
directly predict the type and location of minutiae of the latent fingerprints without reconstructing the ridge pattern. Minutiae-based reconstruction methods are more robust against severe distortion or missing areas of the input latent fingerprint, since they predict the probability of a minutia by analyzing a small area around each candidate ridge point, and they reject large missing areas. Hence, this rejection drastically decreases the number of founded minutiae. Figure1(b) shows some minutiae that were detected using local processing of a latent fingerprint.
In case of severely distorted latent fingerprints, both ridge-based and minutiae-based reconstruction methods fail. Ridge-based reconstruction methods fail because they fill missing areas with incorrect ridge patterns, therefore they introduce minutiae which change the ID of reconstructed samples. Minutiae-based prediction methods fail because they reject most of the missing areas, and at the end, they often predict fewer minutiae, which are not enough to identify samples.
In this study we developed a deep convolutional neural network (DCNN) model to reconstruct the ridge information of latent fingerprints. The core network in the model is a conditional generative adversarial network (cGAN) that reconstructs the obscured ridge information of the latent samples. To overcome the limitation of the previous ridge-based reconstruction methods, our model predicts three extra maps in addition to the ridge map: the orientation, frequency and segmentation maps. Generating the orientation and frequency maps ensure that the model is considering the orientation and frequency information of the input latent fingerprints. Generating a segmentation map prevents the model from filling large missing areas in the input latent samples; thus, it optimizes the amount of ridge information that can be reconstructed. In addition, to force the generator to preserve the ID information (type and location of minutiae), we developed an auxiliary deep model to extract the perceptual ID information (PIDI) of the generated sample and fuse it into the cGAN model to enhance the reconstruction process.
2.1 Conditional Generative Adversarial Networks (cGANs)
GANs  are one of the most popular groups of generative networks. Generative networks map a sample from a random distribution to a target domain of desired samples , through training parameters of the network. GANs are different from conventional generative models because they prosper from a discriminator network. The discriminator network compares the generated samples (fake samples) with the real samples from the target domain, and tries to distinguish between them. Simultaneously, the generator (typically an auto-encoder) tries to fool the discriminator by generating more realistic samples. In each iteration, the generator produces better samples in an attempt to fool the discriminator, and the discriminator improves by comparing the real samples with the generated samples. In other words, the discriminator and the generator play a two-player minimax game with the following objective function:
where generator tries to minimize Eq. 1 and discriminator tries to maximize it. An additional or distance loss is added to the objective function in the literature to force the network to generate samples which are closer to the target ground truth. The final generator model trains as follows:
where is the coefficient of distance and is:
where is the ground truth for the output of the generator.
In the real-world application of restoring a latent fingerprint image, both the source and target domains are available for training. Therefore, Isola et al.  proposed the conditional GAN model which has two modifications compared to conventional GANs. First, instead of using noise as the input to the network, real training samples from the source domain are fed to the network . Second, they put a condition on the discriminator by concatenating the input sample with the generated sample. Therefore, the discriminator judges a generated sample based (conditioned) on the original sample which was fed to the network. The new objective function for the cGAN is:
2.2 cGAN for Latent Fingerprint Reconstruction
The formulation and the network architecture of cGAN proposed by Isola et al. 
is a universal setup for image-to-image translation. However, processing a biometric image such as latent fingerprints is often more complicated than other image types due to the identification information that is embedded in the pattern of the image which should be preserved during the reconstruction process. For this purpose, we perform two modifications to the formulation and the network architecture to emphasize the ID of the input sample.
First, to increase the quality of reconstruction, we force the network to generate four fingerprint maps for each latent fingerprint input as follows:
where, is the new output of the generator, and consists of the ridge map (), the orientation map (), the frequency map () and the ridge segmentation map () of the input latent fingerprint. These four maps are concatenated depth-wise to form . Figure 4(b) demonstrates the new output of the generator.
Eq. 3 computes the direct reconstruction error. For the new outputs of the generator, the same formula is extended as:
where, , , and are the ground truth values for the ridge, frequency, orientation and segmentation map respectively, and , , and are weights for scaling loss of reconstruction of each generated map. Since all maps except the ridge map have low-pass characteristic and may prevent the generator from predicting the ridge map with sufficient detail, we set to 1.0 and all other coefficients (, and ) to 0.1.
Second, the generator should preserve the identification information embedded in minutiae and ridge patterns. To extract and preserve the identification information we developed a method that is inspired by perceptual loss [4, 7, 12, 17] and multi-level feature abstraction [27, 28, 13, 10]. For this purpose, we separately trained a deep Siamese CNN as a fingerprint verifier. The trained model is used to extract the perceptual ID information (PIDI) of the generated maps. Extracted PIDI are output feature maps of the first four convolutional layers of the verifier module, and are concatenated to the corresponding layers of the discriminator to emphasize the ID information on the discriminator’s decision. Figure 4(d) shows how the output feature maps of the verifier contributes to the discriminator’s decision making. Figure 3 shows the training criteria of the discriminator in more detail.
|L #||Type||K. Size||Input Size||Output Size|
|1||C, B, R|
|2||C, B, R|
|3||C, B, R|
|4||C, B, R|
|5||C, B, R|
|6||C, B, R|
Architecture of the PIDI extractor. All layers except the last one comprise Convolution (C), Batch Normalization (B).
2.3 Network Architecture
The proposed model consists of three networks: a fingerprint PIDI extractor, a generator, and a discriminator. The fingerprint PIDI extractor is one tower from a deep Siamese  fingerprint verifier that is trained using a contrastive loss . We fix all weights of the CNN tower, and use it to extract the PIDI of samples generated by the generator. Figure 4(d) shows how the output of the first four layers of the PIDI extractor are fused into the discriminator to provide the identification information for the discriminator. Table 1 details the architecture of the PIDI extractor.
The generator is a ‘U-net’ [19, 11] auto-encoder CNN. In the ‘U-net’ architecture, some layers from the encoder are concatenated to layers of the decoder to keep the high frequency details of the input samples, and increase the quality of reconstruction in the decoder. Figure 4(a) shows a diagram of the generator, and Table 3 details the structure of the generator.
The discriminator is a deep CNN which maps the conditioned output of the generator with a size of to a discrimination matrix of size . To force the discriminator to consider the ID information of the input samples, outputs of the first four layers of the PIDI extractor network are concatenated to the output of the corresponding layers of the discriminator. Table 2 details the architecture of the discriminator network. The discriminator judges the output of the generator in a patch-wise manner. Each element of the discriminator’s output map represents the discriminator’s decision about a corresponding patch in the generator’s output.
The whole generative model was trained over 400 epochs, each epoch consisting of 7,812 iterations with a batch size = 64. Adam optimization method is used as the optimizer due to its fast convergence with beta = 0.5 and learning rate .
|Layer||Type||St.||Kernel Size||Input Size||Output Size||Concat. with||Final Output Size|
|1||C, B, R||2||L 1|
|2||C, B, R||2||L 2|
|3||C, B, R||2||L 3|
|4||C, B, R||2||L 4|
|5||C, B, S||1||-|
|#L||T||S||#K||Input Size||Output Size||#L||T||S||#K||Input Size||Output Size||Cn|
2.4 Synthetic Dataset of Latent Fingerprints
Training a generative model requires a large number of training samples. We synthetically generated a comprehensive database of latent fingerprints by distorting 5,000 clean fingerprints from the BioCOP 2013  database. For each clean fingerprint we generated 100 different latent samples. Our final database of latent fingerprints consists of 500,000 latent fingerprints and their corresponding clean samples. Examples of generated latent samples are shown in Figure 2.
To evaluate the performance of the proposed latent fingerprint reconstruction technique, we conducted three different experiments on publicly available datasets of latent fingerprints. Unfortunately, NIST-SD27  is no longer available, so the IIIT-Delhi Latent fingerprint  and IIIT-Delhi Multi Sensor Latent Fingerprint (MOLF)  databases were used to evaluate the proposed method. In all experiments, we used VeriFinger 7.0 SDK  and the NIST Biometrics Image Software (NBIS)  to match the reconstructed samples. In addition, to evaluate the role of the PIDI fusion which is developed to force the generator to preserve the ID information, we developed a second model which is exactly the same as our complete model but without the PIDI extractor module. Results for the complete model are named ‘cGAN+PIDI’, and results for the second model are named ‘cGAN’.
3.1 Latent-to-sensor matching
For the first experiment, we used the IIIT-Delhi MOLF database. This database contains 19,200 fingerprint samples from 1000 classes (10 fingers of 100 individuals). For each ID, a set of latent fingerprint samples and the corresponding clean samples captured from three different commercial fingerprint scanners (Crossmatch, Secugen, Lumidigm) are available. As in the testing protocol established by Sankaran et al. , the first and second fingerprint samples of each user captured by a sensor are selected as the gallery. The entire latent fingerprint database consisting of 4,400 samples used as the probe set. CMC curves for this experiment are shown in Figure 6 and Figure 7. Table 4 and Table 5 show rank-25 and rank-50 accuracy for the latent-to-sensor matching experiment.
3.2 Latent-to-latent matching
For the second experiment, we evaluated the proposed method on the IIIT-Delhi latent fingerprint dataset which contains 1046 samples from all ten fingerprints recorded from 15 subjects. The experimental setup is defined the same as  by randomly choosing 395 images as gallery and 520 samples as probes. The CMC curves for this experiment is shown in Figure 9. Rank-1, rank-10 and rank-25 results are also shown in Table 6.
3.3 Quality of the reconstructed fingerprints
Using the NFIQ utility from NBIS, the quality of reconstructed samples is measured to directly assess the performance of the reconstruction model. NFIQ assigns each fingerprint a numerical score from 1 (high quality) to 5 (low quality). Quality scores are computed for the reconstructed samples by our method and compared to score of both the raw latent fingerprints and those enhanced by the generative model developed by Svoboda et al. . Figure 8 shows the quality scores of the reconstructed samples, and Fig. 5 shows three examples of the reconstructed fingerprints with different amount of distortion in the input sample.
|Svoboda et al.||16.71||23.03|
|Svoboda et al.||12.33||20.47|
|Svoboda et al.||14.39||22.73|
|Svoboda et al.||19.51||26.24|
|Svoboda et al.||15.23||21.81|
|Svoboda et al.||18.34||24.78|
Recognizing latent fingerprint samples is a challenging problem for identification systems since a latent fingerprint image can be ‘noisy’ with a large portions of the fingerprint missing, leading to a lower amount of ridge information compared to normal fingerprints. Following the successful outcomes of exploiting deep generative models for the traditional image processing problems, such as denoising, inpainting, and image to image translations, we propose a deep latent fingerprint reconstruction model based on conditional generative adversarial networks. We applied two modifications to the cGAN formulation and network architecture to adapt it for the task of latent fingerprint reconstruction. Generated ridge maps using GAN models often contain random ridge patterns for severely distorted areas of the input fingerprint. Main generator of our model is forced to generate three extra fingerprint maps. One of these maps is the ridge segmentation map which shows the reliability of the corresponding ridge map.
Opposed to the previous works in the literature, the proposed network directly translates the input latent fingerprints to the clean binary ridge maps by predicting the missing ridge information. Incorporating a discriminator network which measures both quality and PIDI of the reconstructed samples simultaneously, along with the generation process in the training phase, increases the quality of the generated samples directly without a need to define multiple complex loss functions for minimizing the similarity between the generated ridge patterns and the ground truth maps.
The proposed method successfully reconstructed latent fingerprints from the IIIT-Delhi latent and IIIT-Delhi MOLF databases in different experimental setups of latent-to-sensor and latent-to-latent matching. We achieved rank-50 accuracy of 70.89% for the latent-to-sensor matching on the IIIT-Delhi MOLF database. For the latent-to-latent matching we achieved rank-10 accuracy of 88.02%. Although the best results in both experiments were obtained when VeriFinger was used as the matcher, NBIS matching algorithm also resulted in high matching accuracy.
In addition, measuring the quality of reconstructed fingerprints using NFIQ shows that the generated fingerprints are significantly enhanced compared to the raw latent samples.
For future work it is desired to directly extract minutiae from the latent input fingerprints. On the other hand, increasing the size of the synthetic latent database by introducing more complex distortions is another future direction for this work.
This work is based upon a work supported by the Center for Identification Technology Research and the National Science Foundation under Grant #1650474.
-  K. Cao and A. K. Jain. Latent orientation field estimation via convolutional neural network. In Biometrics (ICB), 2015 International Conference on, pages 349–356. IEEE, 2015.
-  S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 539–546. IEEE, 2005.
-  A. Dabouei, H. Kazemi, S. M. Iranmanesh, J. Dawson, and N. M. Nasrabadi. Fingerprint distortion rectification using deep convolutional neural networks. arXiv preprint arXiv:1801.01198, 2018.
-  A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, pages 658–666, 2016.
-  J. Feng, J. Zhou, and A. K. Jain. Orientation field estimation for latent fingerprint enhancement. IEEE transactions on pattern analysis and machine intelligence, 35(4):925–940, 2013.
-  M. D. Garris and R. M. McCabe. Nist special database 27: Fingerprint minutiae from latent and matching tenprint images. NIST Technical Report NISTIR, 6534, 2000.
-  L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, pages 2414–2423. IEEE, 2016.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
S. Ioffe and C. Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
International Conference on Machine Learning, pages 448–456, 2015.
-  S. M. Iranmanesh, A. Dabouei, H. Kazemi, and N. M. Nasrabadi. Deep cross polarimetric thermal-to-visible face recognition. arXiv preprint arXiv:1801.01486, 2018.
-  P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004, 2016.
J. Johnson, A. Alahi, and L. Fei-Fei.
Perceptual losses for real-time style transfer and super-resolution.In European Conference on Computer Vision, pages 694–711. Springer, 2016.
-  H. Kazemi, M. Iranmanesh, A. Dabouei, S. Soleymani, and N. M. Nasrabadi. Facial attributes guided deep sketch-to-photo synthesis. In Computer Vision Workshops (WACVW), 2018 IEEE Winter Applications of, pages 1–8. IEEE, 2018.
-  D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
-  J. Li, J. Feng, and C.-C. J. Kuo. Deep convolutional neural network for latent fingerprint enhancement. Signal Processing: Image Communication, 60:52–63, 2018.
-  D. Maio, D. Maltoni, R. Cappelli, J. Wayman, and A. Jain. Fvc2004: Third fingerprint verification competition. Biometric Authentication, pages 31–35, 2004.
-  S. Motiian, Q. Jones, S. Iranmanesh, and G. Doretto. Few-shot adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 6670–6680, 2017.
-  Neurotechnology Inc., Verifinger. http://www.neurotechnology.com.
-  O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
-  A. Sankaran, T. I. Dhamecha, M. Vatsa, and R. Singh. On matching latent to latent fingerprints. In Biometrics (IJCB), 2011 International Joint Conference On, pages 1–6. IEEE, 2011.
A. Sankaran, P. Pandey, M. Vatsa, and R. Singh.
On latent fingerprint minutiae extraction using stacked denoising sparse autoencoders.In Biometrics (IJCB), 2014 IEEE International Joint Conference on, pages 1–7. IEEE, 2014.
-  A. Sankaran, M. Vatsa, and R. Singh. Latent fingerprint matching: A survey. IEEE Access, 2(982-1004):1, 2014.
-  A. Sankaran, M. Vatsa, and R. Singh. Multisensor optical and latent fingerprint database. IEEE Access, 3:653–665, 2015.
-  A. Sankaran, M. Vatsa, R. Singh, and A. Majumdar. Group sparse autoencoder. Image and Vision Computing, 60:64–74, 2017.
-  X. Si, J. Feng, and J. Zhou. Detecting fingerprint distortion from a single image. In 2012 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6, Dec 2012.
-  X. Si, J. Feng, J. Zhou, and Y. Luo. Detection and rectification of distorted fingerprints. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):555–568, March 2015.
-  S. Soleymani, A. Dabouei, H. Kazemi, J. Dawson, and N. M. Nasrabadi. Multi-level feature abstraction from convolutional neural networks for multimodal biometric identification. In 24th International Conference on Pattern Recognition (ICPR), 2018.
-  S. Soleymani, A. Torfi, J. Dawson, and N. M. Nasrabadi. Generalized bilinear deep convolutional neural networks for multimodal biometric identification. In IEEE International Conference on Image Processing (ICIP), 2018.
-  J. Svoboda, F. Monti, and M. M. Bronstein. Generative convolutional networks for latent fingerprint reconstruction. arXiv preprint arXiv:1705.01707, 2017.
-  Y. Tang, F. Gao, and J. Feng. Latent fingerprint minutia extraction using fully convolutional network. In Biometrics (IJCB), 2017 IEEE International Joint Conference on, pages 117–123. IEEE, 2017.
-  C. Watson, M. Garris, E. Tabassi, C. Wilson, R. McCabe, S. Janet, and K. Ko. Nist biometric image software, 2011.
-  WVU multimodal dataset, Biometrics and Identification Innovation Center. http://biic.wvu.edu/.
-  S. Yoon, J. Feng, and A. K. Jain. On latent fingerprint enhancement. Biometric Technology for Human Identification VII, 7667(1):228, 2010.
-  S. Yoon, J. Feng, and A. K. Jain. Latent fingerprint enhancement via robust orientation field estimation. In Biometrics (IJCB), 2011 International Joint Conference on, pages 1–8. IEEE, 2011.
-  S. Yoon, J. Feng, and A. K. Jain. Altered fingerprints: Analysis and detection. IEEE transactions on pattern analysis and machine intelligence, 34(3):451–464, 2012.