Latent fingerprints are unintentional fingerprints that are left on surfaces. They are challenging to identify because of many factors such as poor quality, background noise, distortion, small non-overlapping areas, etc. Hence, processing latent images requires a lot of effort and time to get acceptable recognition results compared to scanned or rolled fingerprints. Unlike rolled fingerprints which have a number of distinctive characteristics such as ridge flow, large number of minutiae, and little background noise, latent fingerprints only contain a small portion of the fingerprint ridge structure [22, 23]. Thus, there is an urgent need to use other features that are present even in small partial fingerprints.
Fingerprints are known to have distinctive characteristics, even in identical twins, and are unchanging throughout a person’s lifetime . Such discriminative power emerges from three levels of fingerprint features: (i) Level 1 (ridge flow); (ii) Level 2 (minutiae points, such as ridge bifurcations and endings); and (iii) Level 3 (scars, pores, incipient ridges, etc.). Automatic fingerprint identification systems (AFIS) have generally used minutiae as primary features to discriminate fingerprints from different subjects. While AFIS accuracy for matching full fingerprints to full fingerprints is sufficiently high , only a few commercial AFIS can work on latent images and even then with low accuracy. The reason is that most of AFIS primarily rely on minutiae for matching while many of latent images just contain a small part of a fingerprint, and thus few minutiae. As a result, the AFIS fails in those cases. In developing a robust latent AFIS, one should take care to handle the cases where the query latent contains only a small portion of the whole fingerprint ridge flow. Use of level 3 features is a natural solution for such scenarios.
Unlike scanned or rolled fingerprints, latent fingerprints are very sensitive to both inherent (e.g. scars, injuries) and background (e.g. friction ridges on a printed document) noise. Since sweat pores are on fingerprint ridges are quite visible to human eyes compared to the other level-3 features, pores have been used as complementary features to minutiae [28, 15, 30]. This argument is the basis for our usage of pores as main level-3 features for high resolution latent fingerprint matching. If successful, the use of pores for matching could be used in other applications domains involving partial fingerprints (due to small sensor size), such as smart bank cards and mobile devices. Pores could also be used to aid infant fingerprint recognition systems where minutiae are not always available, even at resolutions over 1500 ppi.
Ii Related Work
An end-to-end AFIS contains two stages: feature extraction and matching. The feature extraction stage converts input fingerprint images into corresponding templates while the matching stage uses those templates to compute a similarity score between the input fingerprint templates, resulting in a ranked list of candidate matches. Although many studies have tried to address level 3 feature extraction and matching on fingerprints, these only used high resolution images with no background noise or scanned fingerprints.
Ii-a Pore extraction.
With the advancement of capture technology, obtaining high resolution images is not an obstacle in forensic community. One of the pioneers in the extraction of pores as fingerprint features is Ray et al. , who used a modified minimum squared error approach on the NIST 4  dataset. Jain et al.  developed their techniques based on high quality scanned partial fingerprint images. Zhao et al.  proposed adaptive modeling for sweat pore extraction. They divided the image into small blocks and constructed local pore models for each region.
With the development of neural networks and high-powered GPUs, some studies have used a deep network approach to extract pores. Janget al. 
proposed a pore extraction technique based on convolutional neural networks. Again, they used high resolution partial fingerprints, not latents, to evaluate their approach. Genoveseet al.  utilized a neural network to extract pores from live-scanned fingerprints. Labati et al.  used a similar idea for pore extraction in heterogeneous fingerprint images. However, the lab-collected “pseudo latent” dataset mentioned in the paper is very different from those sourced from forensic agencies.
The excellent pore extraction results on scanned high resolution images with clear backgrounds do not reveal anything about their capability on operational latent fingerprint datasets. Besides, for rolled or live-scanned high-resolution fingerprint images, traditional approaches can achieve reasonable results.
Ii-B Pore matching
There have been several attempts made to use pores as a feature for matching. Jain et al.  designed a hand-crafted approach to utilize features from all 3 fingerprint feature levels. They used pre-defined thresholds for their matching system. Again, they only reported experimental results on their high-resolution lab-collected dataset. Chen and Jain  extracted an extended feature set as input for the matching step. They used a commercial off-the-shelf (COTS) matcher to evaluate their feature extraction. The dataset on which they evaluated is the partial fingerprint dataset from NIST SD30 111This dataset is no longer publicly available.. Zhao et al.  proposed a pore matcher based on a combination of the ICP  and RANSAC  algorithms. Again, they only evaluated their algorithm on their high-resolution live-scanned partial fingerprint images.
A few publications applied their algorithms on latent fingerprint images. Jain and Feng  combined pores with other manually marked features for matching. However, their approach is not fully automated, requiring human intervention for feature extraction. The work in  evaluated directly on latent fingerprint images using level-3 features. They also used other level 3 features such as dots, incipient ridges, and ridge edge protrusions (DIP) for the matching. However, their approach was also not fully automatic and relied on an existing COTS minutiae matcher to see the advantage of using extended feature sets. Although their approach can work reasonably well on simulated partial fingerprints, it failed when used independently without a COTS system. Other existing pore matching works ([8, 20]) were not applied or evaluated on latent images, or simply do not go beyond the pore extraction stage [30, 17, 19]. One of the reasons for the scarcity of latent fingerprint research using pores is the lack of publicly available operational latent fingerprint databases.
Iii Pre-processing and pore extraction
There are three reasons, in our opinion, for not using a neural network approach to extract pores in forensic latent fingerprint images: (i) latent fingerprints contain unexpected background noise and distortion; (ii) pores in latent fingerprints are very sensitive to image normalization techniques; and (iii) pores are very small compared to the overall image size, which are not suitable for object detection-like neural network approaches.
Iii-a Pore Extraction
Unlike other pore extraction methods using mathematical modeling [30, 28] or pixel values between ridge and valley  that are only suitable for images with clear background, our proposed approach can work on arbitrary input fingerprint images without remodeling for new fingerprint image type or resolution. Based on the observation that sweat pores are on the fingerprint ridges, we propose two stages for pore extraction: (i) ridge enhancement and (ii) pore extraction using connected components. Figure 1 shows the pipeline of our proposed pore extraction module. Note that we visualize pore extraction steps on high-resolution scanned images to allow readers to understand the approach more easily.
Iii-A1 Fingerprint ridge enhancement
Unlike scanned fingerprints which have a clear background, latent fingerprints need to be enhanced to obtain reliable friction ridges for further processing. We use several different enhancement techniques to get enhanced images for a given latent fingerprint input. Specifically, we apply the short-time Fourier transform (STFT) and cartoon-texture decomposition technique  to enhance the the original image and get sharper friction ridges. Since pores have small sizes and variable pixel values, global enhancement techniques such as contrast enhancement  or auto-encoders  are not suitable here. This is in contrast to systems that use minutiae or orientation field as features, where these enhancement techniques are appropriate. Figure 2 shows examples of latent fingerprint enhancement. Although the STFT and cartoon-texture decomposition enhancement approaches do not completely eliminate background noise, they are resistant to artifact creation, and can be used to convert the latent and rolled prints into a similar appearance, thus projecting them into the same input space and allowing for easier extraction of connected components. Hence, the system does not need to have different configurations for different types of input images.
Iii-A2 Neighbors connected component technique for pore extraction
to get a binary version of the image for the extraction of connected components. Because pores are local features, using the adaptive threshold technique can divide the input image into sub-regions and binarize that region based on its information. Thus, this technique can preserve location of pores.
Based on the observation that pores are small, have rounded shape and are located on the friction ridges, we regard the image as a densely connected graph where an edge exists between two vertices if and only if they have the same pixel color. The problem turns out to be the graph coloring problem , which is easily solved using graph traversal and some special data structures. We refer readers to  for more information about the solution to this problem. Because pores are small in size and vary based on input resolution, we create a user-input parameter for limiting the maximum pore area. By default, pixels is used for ppi image. Thus, components having size larger than are eliminated. The precise location of each pore is taken to be the mean of all pixels inside that pore region.
Figure 3 shows pore extraction results on different fingerprint types and resolutions to demonstrate the generalizability of our proposed pore extraction approach.
Iv Pore Matching With Graphs
Figure 4 visualizes our matching framework. Our approach comprises of two modules: minutiae vs. pore matching and re-ranking. Since pores are only used for exclusion in cases where the minutiae decision is not clear, we use pore matching as a complementary step for minutiae matching. Hence, given an input latent image, we normalize its resolution to ppi, apply our enhancement technique given in section III-A1, and input it to an end-to-end minutiae-based matcher . There are two cases in which the minutiae matcher does not provide strong discriminative power: (i) number of minutiae is small; and (ii) similarity scores between matched minutiae are not high enough (i.e. weak confidence) to deliver a decision.
Iv-a Pore matching using bipartite graphs
Before discussing the details of pore matching, we summarize the common steps for minutiae matching: (i) detect minutiae; (ii) construct minutiae descriptors using surrounding patches around minutiae by hand-crafted approaches or automatically extracted from convolutional networks; (iii) find minutiae correspondences by comparing distances (e.g. or norm, cosine criteria, etc.) between minutiae descriptors; (iv) eliminate wrong (low confidence) matches using relative position between minutiae; (v) return the final score as the sum of all minutiae similarity scores . Our proposed pore matching approach follows this procedure to find reliably matched pores. Note that we do not use a neural network approach to get pore descriptors (as in the construction of minutiae descriptors) because pores are dense and small, which contrasts with the sparse and relatively large number of attributes of minutiae. Since the surrounding area for each pore is small (i.e. pixels in ppi image), they are not suitable for deep networks.
Unlike scanned/rolled fingerprints, latent fingerprints usually have a small number of minutiae, and thus cannot be reliably matched using minutiae alone. The average number of minutiae in NIST SD27 is and the minimum and maximum number of minutiae are: and On the other hand, the average number of minutiae in rolled mates of NIST SD27 latents is . However, that does not mean we completely disregard the minutiae information. On the contrary, we use matched minutiae to guide initial pore correspondence. To increase system’s ability to handle extreme cases (fewer than matched minutiae), we consider two cases: fewer than minutiae matches, and at least minutiae matches. Specifically, given at least matched minutiae pairs, we construct a transformation matrix from a latent fingerprint to a rolled print. Since different impressions are not collected at the crime scenes in the same way, we introduce tolerance to deal with distortion. That means one latent pore can have a few matched pores in a given rolled print which satisfy the tolerance constraint. In the case of fewer than matched minutiae, we construct a Hamiltonian path  on both the latent print and rolled print pore graphs. Then, we apply statistical shape matching  to get matched pore pairs.
At this point, each pore in the latent print has a few candidate pores in the rolled print. We remove spurious matches by converting this problem into bipartite matching: finding the maximum set of matches in such a way that no two matches have a common endpoint. Because the algorithm finds the maximize number of matches, only very similar image pairs can have high similarity scores (as well as number of matches), where the pore matching score between two images is the sum of scores of all refined pore matches.
Iv-B Re-ranking with fusion
Since pores, in forensic practice, are not used directly for identification (instead for exclusion), we use pore scores as criteria to re-rank the minutiae matching-based rank list. Figure 5 presents our re-ranking process. First, we get pore scores for top candidates in the minutiae rank list. Second, we retrieve the pore index for each pore score. Third, we compute the weighted sum of minutiae and pore indices. Lastly, the final index is the ranking of the sum calculated in the previous step.
In this paper, three databases, NIST SD27 (1,000 ppi), NIST SD30, and PolyU-HRF , are used to evaluate the proposed end-to-end pore extraction and matching framework. Table I shows a summary of the three fingerprint databases. We also collected a database of 500 different fingers from 50 subjects at both 500 ppi and 1,000 ppi for our algorithm development and parameter tuning. Some example images from the NIST, PolyU-HRF, and our privately collected databases are shown in figure 6.
V-a Pore extraction evaluation
As shown in figure 3, our proposed pore extraction approach can work well on a variety of input image types and resolutions. However, with the scarcity of high-resolution fingerprint datasets with ground truth information on pores, we quantitatively evaluate the effectiveness of our approach on partial fingerprint images  for reference only. Note that our algorithm’s better pore extraction ability on non-latent high-resolution fingerprint images does not predict its capability on latent fingerprint images.
V-A1 Evaluation protocol
We use the precision, recall, and F1 score to evaluate the goodness of extracted pores compared to manual ground truth222Provided by the authors of . Let be true positive, be false positive, be false negative. Precision, recall, and F1 score are then defined as:
Table II shows the precision, recall, and F1 score of pore extraction on partial fingerprint images. The default confident threshold is . All of the precision, recall, and F1 score are in the range of where value of indicates the “perfect” compared to the ground truth.
V-A2 Running time
The pore extraction time depends on the number of pores in the input image. In the NIST SD27 1,000ppi dataset, the latent images, on average, take seconds for extraction, while the rolled prints (500 ppi upsampled to 1,000 ppi) take seconds on an Intel(R) Core(TM) i7-7700 CPU @3.6GHz, 32GB RAM desktop machine. Approximately, the computational complexity of pore extraction is an exponential function of the number of pores.
V-B Pore matching evaluation
We evaluate our proposed pore-based matcher on the NIST SD27 1,000 ppi dataset. We report the performance based on closed-set identification, where the true mated prints of the query are assumed to be in the gallery. Figure 9 shows the cumulative match characteristic (CMC) curves of our proposed pore-based matcher and minutiae-based matcher on latents in NIST SD27 1,000 ppi with background size up to .
By utilizing pore information, we improve more cases in rank-1 and cases at rank-10 compared to using minutiae matcher only. Thus, the overall rank-1 accuracy is with a background size of 255. This implies that pores provide some information that is complementary to the minutiae. However, when the background size increases, the pore based matcher no longer improves the minutiae based matcher and even decrease the rank retrieval for some of the latents. The reason for showing only a single curve for background is that the pores do not help minutiae, hence the minutiae matcher curve and fusion curve are the same. That means there are no cases which satisfy the pore matching criteria (e.g. pore scores are not significantly distinct or the minutiae scores between latent and imposter prints are already sufficiently large to make a decision). Further explanation is provided in section V-C.
Figure 10 shows examples of correct and incorrect matches in NIST SD27, as well as situations where re-ranking does and does not improve the rankings for genuine mates.
We also evaluate our matcher on the NIST SD30 dataset. Table III summarizes the results on the background size of . Since this is a rolled print dataset, with high numbers of minutiae and low background noise, the minutiae-based matcher alone performs extremely well. As in Table III, we clearly see that the pore based matcher helps improve the identification accuracy.
The current runtime of the pore-based matcher is, on average, 10 ms per match (matching one latent to one rolled print) on an Intel(R) Core(TM) i7-7700 CPU @3.6GHz, 32GB RAM desktop machine.
V-C Analysis of pore extraction and matching
In order to understand the performance and improve the recognition accuracy, we conducted ablation studies of our pore extraction and matching. Figure 11 compares the number of minutiae and pores automatically extracted in both latent and mated prints in NIST SD27 1,000 ppi dataset. As confirmed by the figure, the partial nature of latent prints leads to a smaller number of minutiae and pores relative to rolled prints. Another observation is that the number of pores is significantly larger than number of minutiae ( to times larger). While the number of pores in principle should help improve matching results by providing more reliable correspondences, they become very sensitive to noise if the pore distribution is too dense. This means we could find a match in a rolled print for nearly every pore in the latent print. Hence, the pore scores are not reliable.
Figure 12 shows the overlay between latent and rolled prints with their corresponding rank. The greater the transformation between the latent and its mated rolled print on the right, the more distortion it has. This distortion is one of the greatest challenges in the latent identification problem. This distortion is particularly difficult to handle in latent prints because there is insufficient surface area to capture reliable keypoints for rectification.
V-D Ablation study
In this section, we further analyze the pore features on infant fingerprint besides adult fingerprints (NIST databases, PolyU-HRF, etc.), which have been intensively studied in literature. The infant fingerprint database 333We contacted the authors of  for our experiment contains images at ppi resolution from infants collected from sessions. For further details of the database, we refer readers to .
Figure 13 shows the pore extraction on infant fingerprints. Although the pore size of infant prints are different from adult prints, the proposed pore extraction can also work well on those images. Note that we use the latent settings for both extraction and matching experiments for the infant database in this ablation study to demonstrate the portability of our approach. We follow the experimental setup of  in our verification experiment. Specifically, we used two state-of-the-art COTS matchers (COTS-A matcher is designed for latent fingerprint and COTS-B is utilized for plain (slap) prints), and the texture-based matcher from . The goal is to see whether using the scores of our pore matcher can help increase identification/verification accuracy. In our experiment, we fuse the scores of two thumbs of an infant and also scores of different impressions of the same thumb. Table IV shows the verification accuracy (TAR(%)@0.1% and 1.0% FAR) of each matcher and the fusion of all matchers on 2-3 months old infants.
As from the table, although the individual pore scores do not have the high TAR values compared to the two COTS matchers, which utilize minutiae information, fusing all scores together helps boost the performance. Figure 14 shows the genuine and imposter score histogram of infant fingerprints. Even though the distribution of imposter scores has a long tail, score level fusion between matchers helps separate those genuine and imposter scores.
In this paper, we have presented an end-to-end pore extraction and matching framework for arbitrary fingerprint types and qualities. To demonstrate the effectiveness of our proposed approach, we evaluated our approach in terms of qualitative, quantitative, and recognition accuracy. Experimental results and analysis reveal that including pore based matcher leads to only a marginal improvement in retrieval results compared to minutiae only based matcher. This can be explained in terms of large distortion in latent prints where pores are not clearly visible. However, pores do improve in some latents where the number of minutiae is neither too large nor too small. So, this is some evidence of the complementary characteristics of pores and minutiae, but generalization of these cases is difficult. The pore matcher could be further improved in the following ways: (i) improve fingerprint normalization steps to compensate for fingerprint distortion; and (ii) incorporate additional features such as ridge flow for the re-ranking step. Currently, the only 1,000 ppi latent database is the NIST SD27 with 255 image pairs. Lack of additional high resolution latent datasets (with associated mates and a large background) is a major stumbling block.
-  NIST Special Database 4. http://www.nist.gov/srd/nistsd4.cfm.
-  D. Bradley and G. Roth. Adaptive thresholding using the integral image. Journal of Graphics Tools, 12(2):13–21, 2007.
-  A. Buades, T. Le, J.-M. Morel, and L. Vese. Cartoon+ texture image decomposition. Image Processing On Line, 1:200–207, 2011.
-  K. Cao, D.-L. Nguyen, C. Tymoszek, and A. Jain. End-to-end latent fingerprint search. arXiv preprint arXiv:1812.10213, 2018.
-  U. Castellani, M. Cristani, S. Fantoni, and V. Murino. Sparse points matching by combining 3d mesh saliency with statistical descriptors. In Computer Graphics Forum, volume 27, pages 643–652. Wiley Online Library, 2008.
-  Y. Chen and A. K. Jain. Dots and incipients: extended features for partial fingerprint matching. In 2007 Biometrics Symposium, pages 1–6. IEEE, 2007.
-  S. Chikkerur, A. N. Cartwright, and V. Govindaraju. Fingerprint enhancement using stft analysis. Pattern Recognition, 40(1):198–211, 2007.
-  G. Dahia and M. P. Segundo. Cnn-based pore detection and description for high-resolution fingerprint recognition. arXiv preprint arXiv:1809.10229, 2018.
-  J. J. Engelsma, K. Cao, and A. K. Jain. Fingerprints: Fixed length representation via deep networks and domain knowledge. arXiv preprint arXiv:1904.01099, 2019.
-  J. J. Engelsma, D. Deb, A. K. Jain, P. S. Sudhish, and A. Bhatnager. Infant-prints: Fingerprints for reducing infant mortality. arXiv preprint arXiv:1904.01091, 2019.
-  M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381–395, 1981.
A. Genovese, E. Munoz, V. Piuri, F. Scotti, and G. Sforza.
Towards touchless pore fingerprint biometrics: a neural approach.
2016 IEEE Congress on Evolutionary Computation (CEC), pages 4265–4272. IEEE, 2016.
-  Y. Gurevich and S. Shelah. Expected computation time for hamiltonian path problem. SIAM Journal on Computing, 16(3):486–502, 1987.
-  M. Indovina, R. Hicklin, and G. Kiebuzinski. Elft-efs evaluation of latent fingerprint technologies: extended feature sets [evaluation# 1]. National Institute of Standards and Technology, US Department of Commerce NISTIR, 7775, 2011.
-  A. K. Jain, Y. Chen, and M. Demirkus. Pores and ridges: High-resolution fingerprint matching using level 3 features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(1):15–27, 2007.
-  A. K. Jain and J. Feng. Latent fingerprint matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):88–100, 2011.
-  H.-U. Jang, D. Kim, S.-M. Mun, S. Choi, and H.-K. Lee. Deeppore: fingerprint pore extraction using deep convolutional neural networks. IEEE Signal Processing Letters, 24(12):1808–1812, 2017.
-  T. R. Jensen and B. Toft. Graph coloring problems, volume 39. John Wiley & Sons, 2011.
-  R. D. Labati, A. Genovese, E. Muñoz, V. Piuri, and F. Scotti. A novel pore extraction method for heterogeneous fingerprint images using convolutional neural networks. Pattern Recognition Letters, 113:58–66, 2018.
-  F. Liu, Y. Zhao, and L. Shen. Feature guided fingerprint pore matching. In Chinese Conference on Biometric Recognition, pages 334–343. Springer, 2017.
-  PolyU-HRF Database. http://www4.comp.polyu.edu.hk/~biometrics/HRF/HRF~_old.htm.
-  D.-L. Nguyen, K. Cao, and A. K. Jain. Robust minutiae extractor: Integrating deep networks and fingerprint domain knowledge. In 2018 International Conference on Biometrics (ICB), pages 9–16. IEEE, 2018.
-  D.-L. Nguyen, K. Cao, and A. K. Jain. Automatic latent fingerprint segmentation. In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1–9. IEEE, 2019.
P. M. Pardalos, T. Mavridou, and J. Xue.
The graph coloring problem: A bibliographic survey.
Handbook of Combinatorial Optimization, pages 1077–1141. Springer, 1998.
-  M. Ray, P. Meenen, and R. Adhami. A novel approach to fingerprint pore extraction. In Proceedings of the Thirty-Seventh Southeastern Symposium on System Theory, 2005. SSST’05., pages 282–286. IEEE, 2005.
-  S. Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In 3dim, volume 1, pages 145–152, 2001.
-  J. A. Stark. Adaptive image contrast enhancement using generalizations of histogram equalization. IEEE Transactions on Image Processing, 9(5):889–896, 2000.
-  Q. Zhao, J. Feng, and A. K. Jain. Latent fingerprint matching: Utility of level 3 features. MSU Techical Report, 8:1–30, 2010.
-  Q. Zhao, D. Zhang, L. Zhang, and N. Luo. Adaptive fingerprint pore modeling and extraction. Pattern Recognition, 43(8):2833–2844, 2010.
-  Q. Zhao, L. Zhang, D. Zhang, and N. Luo. Direct pore matching for fingerprint recognition. In International Conference on Biometrics, pages 597–606. Springer, 2009.