Face recognition is a long standing challenge in the field of Artificial Intelligence (AI). The goal is to create systems that accurately detect, recognize, verify, and understand human faces. There are significant technical hurdles in making these systems accurate, particularly in unconstrained settings due to confounding factors related to pose, resolution, illumination, occlusion, and viewpoint. However, with recent advances in neural networks, face recognition has achieved unprecedented accuracy, largely built on data-driven deep learning methods. While this is encouraging, a critical aspect that is limiting facial recognition accuracy and fairness is inherent facial diversity. Every face is different. Every face reflects something unique about us. Aspects of our heritage - including race, ethnicity, culture, geography - and our individual identify - age, gender, and other visible manifestations of self-expression, are reflected in our faces. We expect face recognition to work equally accurately for every face. Face recognition needs to be fair. As we rely on data-driven methods to create face recognition technology, we need to ensure necessary balance and coverage in training data. However, there are still scientific questions about how to represent and extract pertinent facial features and quantitatively measure facial diversity. Towards this goal, Diversity in Faces (DiF) provides a data set of one million annotated human face images for advancing the study of facial diversity. The annotations are generated using ten well-established facial coding schemes from the scientific literature. The facial coding schemes provide human-interpretable quantitative measures of facial features. We believe that by making the extracted coding schemes available on a large set of faces, we can accelerate research and development towards creating more fair and accurate facial recognition systems.READ FULL TEXT VIEW PDF
Have you ever been treated unfairly? How did it make you feel? Probably not too good. Most people generally agree that a fairer world is a better world. Artificial Intelligence (AI) has enormous potential to make the world a better place. Yet, as we develop and apply AI towards addressing a broad set of important challenges, we need to make sure the AI systems themselves are fair and accurate. Recent advances in AI technology have produced remarkable capabilities for accomplishing sophisticated tasks, like translating speech across languages to augment communications and bridge cultures, improving complex interactions between people and machines, and automatically recognizing contents of video to assist in safety applications. Much of the recent power of AI comes from the use of data-driven deep learning to train increasingly accurate models by using growing amounts of data. However, the strength of these techniques can also be their inherent weakness. These AI systems learn what they are taught. If they are not taught with robust and diverse data sets, accuracy and fairness are at risk. For that reason, AI developers and the research community need to be thoughtful about what data they use for training. This is essential for developing AI systems which can help to make the world more fair.
The challenge in training AI systems is manifested in a very apparent and profound way with face recognition technology. Today, there can be difficulties in making face recognition systems that meet fairness expectations. The heart of the problem lies not with the AI technology itself, per se, but with how the systems are trained. For face recognition to perform as desired –- to be both accurate and fair –- training data must provide sufficient balance and coverage. The training data sets should be large enough and diverse enough to learn the many ways in which faces inherently differ. The images must reflect the diversity of features in faces we see in the world. This raises the important question of how we measure and ensure diversity for faces. On one hand, we are familiar with how faces may differ according to age, gender and skin color. But, as prior studies have shown, these dimensions are inadequate for characterizing the full range of diversity of faces. Dimensions like face symmetry, facial contrast, and the sizes, distances and ratios of the various attributes of the face (eyes, nose, forehead, etc.), among many others, are important.
Diversity in Faces () is a new large and diverse data set designed to advance the study of fairness and accuracy in face recognition technology. provides a data set of annotations of one million face images. The annotations are made on faces sampled from the publicly available YFCC-100M data set of 100 million images . The data set provides a comprehensive set of annotations of intrinsic facial features that includes craniofacial distances, areas and ratios, facial symmetry and contrast, skin color, age and gender predictions, subjective annotations, and pose and resolution. The facial coding schemes, summarized in Table 1, are among the strongest identified in the scientific literature and build a solid foundation to our collective knowledge. We believe that making these descriptors available will help accelerate the study of diversity and coverage of data towards creating more fair and accurate face recognition systems.
|#||Facial Coding Scheme||Reference|
|1||Craniofacial Distances||L. G. Farkas, Anthropometry of the Head and Face, Raven Press, 1994 .|
|2||Craniofacial Areas||L. G. Farkas, et. al, “International anthropometric study of facial morphology in various ethnic groups/races,” J Craniofac Surg. 2005 Jul;16(4), pp. 615-46 .|
|3||Craniofacial Ratios||N. Ramanathan, R. Chellappa, “Modeling Age Progression in Young Faces,” , 2006, pp. 387-394 .|
|4||Facial Symmetry||A. C. Little, B. C. Jones, L. M. DeBruine, “Facial attractiveness: evolutionary based research,” Philos Trans R Soc Lond B Biol Sci. 2011 Jun 12;366(1571), pp. 1638-59 .|
|5||Facial Contrast||A. Porcheron, E. Mauger, R. Russell, “Aspects of Facial Contrast Decrease with Age and Are Cues for Age Perception,” PLoS One 8(3), Mar. 6, 2013 .|
|6||Skin Color||A. Chardon I. Cretois and C. Hourseau, “Skin colour typology and suntanning pathways,” Intl. Journal of Cosmetic Science, Aug. 1991, 13(4), pp. 191-208 .|
|7||Age||R. Rothe, R. Timofte, L. Van Gool, “Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks,” L. Int J Comput Vis (2018) 126: 144 .|
|8||Gender||Same as above|
|9||Subjective Annotation||Z. Liu, P. Luo, X. Wang, X. Tang, “Deep Learning Face Attributes in the Wild,” IEEE Intl. Conf. on Computer Vision (ICCV), 2015 |
|10||Pose and Resolution||
X. Zhu, D. Ramanan, “Face Detection, Pose Estimation, and Landmark Localization in the Wild,”Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), 2012 .
In this paper, we describe the development and analysis of the data set. The paper is organized as follows: in Section 2, we review the state of face recognition technology and examine how different face image data sets are used today. We discuss some of the shortcomings from over-reliance on narrow data sets. In Section 3, we describe the process for creating the data set. In Section 4, we describe the implementation of the ten facial coding schemes. In Section 5, we provide a statistical analysis of the coding schemes extracted for the face images. In Section 6, we summarize and discuss future directions.
Face recognition is a long-standing topic in computer vision, and AI broadly. Computer-based face recognition was addressed as far back as the 1970s with Takeo Kanade‘s seminal thesis on recognizing faces using a set of manually defined points corresponding to nose, mouth, eyes and other features. Modest by today’s standards, his work processed photographs and conducted experiments involving identification of 20 people . Two decades later, a significant development came from Matthew Turk and Alex Pentland, who developed an appearance-based technique called eigenfaces that models faces holistically from image data . This kind of data-driven methodology was subsequently helped by numerous efforts of curating large and growing face image data sets. The community has built open evaluations around these data sets, such as MegaFace , MS-Celeb  and the NIST Face Recognition Vendor Test (FRVT)111https://www.nist.gov/programs-projects/face-recognition-vendor-test-frvt.
One prominent example of an early face data set and open evaluation is Labeled Faces in the Wild (LFW), which is comprised of face photos from the Web of individuals, mostly celebrities and public figures, captured in unconstrained conditions of lighting, pose and expression . LFW gained significant focus from the research community upon its release. Eventually, with the advent of deep learning techniques [16, 17], face recognition performance on LFW reached near-perfect results with accuracy [18, 19]. Megaface defined a follow-on larger data set comprised of faces from unique individuals which was made more difficult with the addition of million face image distractors . Although early results produced low accuracy in the range of , ultimately, performance reached near-perfect levels of [20, 21]. Other data sets and evaluations such as CelebA have brought focus to a wider set of problems in face recognition such as face attribute recognition. CelebA provides a data set of face images with annotations of attributes such as ‘eyeglasses,’ ‘smiling,’ and ‘mustache’ . State-of-art systems have achieved greater than mean accuracy across the CelebA attribute set and as high as for some attributes. Many other face data sets and evaluations have produced similar improvements using deep learning methods [14, 22, 23, 24, 8, 25, 26, 9, 27, 28, 29, 30, 31, 32, 33, 34]. The healthy progress on face recognition, as measured on these data sets and evaluations, has raised expectations in the technology.
. The reason is that different or broader distributions of faces, as well as varied environmental conditions, are found in real applications. Face recognition systems that are trained within only a narrow context of a specific data set will inevitably acquire bias that skews learning towards the specific characteristics of the data set. This narrow context appears as under-representation or over-representation of certain types of faces in many of the publicly available data sets. Table2 shows some of the big differences in distribution of age groups for seven prominent face image data sets. Generally, there is a skew away from younger and older ages. Some of the differences are quite dramatic. For example, of faces in IMDB-Face are for individuals 20-30 years of age, whereas IJB-C has of faces in this age group.
Similarly, Table 3 shows the distribution of gender and skin color/type for eight face image data sets. LFW is highly skewed towards male faces with corresponding to male. Six of the eight data sets have more male faces. A similar skew is seen with skin color/type when grouped coarsely into darker and lighter groups. Note that different methods were used for characterizing skin color/type in Table 3, and the meaning of darker and lighter is not the same across these eight data sets. For all but two data sets the distribution shows faces that are lighter. AgeDb is the most heavily skewed, with faces that are lighter. The Pilot Parliaments Benchmark (PPB) data set was designed to be balanced for gender and skin type, where a board certified dermatologist provided the ground-truth labels using the Fitzpatrick six-point system [36, 38]. However, the age distribution in PPB is skewed, having been built from official photos of members of parliaments, all adults, from six countries. Face recognition systems developed from skewed training data are bound to produce biased models. This mismatch has been evidenced in the significant drop in performance for different groupings of faces [39, 40, 41]. A published study showed that gender estimation from face images is biased against dark-skinned females over white-skinned males [36, 37]. Such biases may have serious impacts in practice. Yet much of the prior research on face recognition does not take these issues under consideration, having focused strongly on driving up accuracy on narrow data sets. Note also that the gender categorization in Table 3, as in much of the prior work, uses a binary system for gender classification that corresponds to biological sex – male and female. However, different interpretations of gender in practice can include biological gender, psychological gender and social gender roles. As with race and ethnicity, over-simplification of gender by imposing an incomplete system of categorization can result in face recognition technologies that do not work fairly for all of us. Some recent efforts, such as InclusiveFaceNet , show that imperfect categorization of race and gender can help with face attribute recognition. However, we expect that more nuanced treatment of race, ethnicity and gender is important towards improving diversity in face data sets.
The study of bias and fairness has recently gained broad interest in computer vision and machine learning[43, 44, 45]. Torralba and Efros  presented an evaluation of metrics related to bias and framed bias in visual classification as a domain transfer problem. Tommasi et al.  and Hoffman et al. 
conducted a similar evaluation with deep features, showing that data set bias can be reduced but not eliminated. Khosla et al.
proposed a method that learns bias vectors associated with individual data sets, as well as weights common across data sets, which are learned by undoing unwanted bias from each data set. Hardt et al.
proposed a framework for fairness called equalized odds, also referred to as disparate mistreatment, where the goal is to predict a true outcome based on labeled training data, while ensuring it is ‘non-discriminatory’ with respect to a chosen protected attribute. More recently, Burns et al.  addressed bias in image captioning, proposing a model that ensures equal gender probability when gender evidence is occluded in a scene, and otherwise predicts gender from relevant information when present. The problem of gender-neutral smile classification was addressed in . Bias in face detection for skin tone, pose and expression was studied in . Buolamwini et al.  proposed an approach to evaluate unwanted bias in face recognition and data sets with respect to phenotypic subgroups and introduced the PPB data set balanced by gender and skin type.
As described above, the last decade has seen an ever-growing collection of face recognition data sets. Table 4 summarizes many of the prominent face image data sets used for evaluating face recognition technology. Returning to Labeled Faces in the Wild (LFW) , it presented considerable technical challenges upon its release in 2007, whereas nearly perfect results are being attained today. Several data sets such as IJBC , UMD  and VGGFace [23, 22] provide a larger set of face images with a wider range of pose and lighting variations. Other large-scale face recognition data sets include MegaFace , MS-Celeb  and CASIA . Many other data sets have been proposed for different facial analysis tasks, such as age and gender classification [27, 54, 55, 24, 31, 56], facial expression analysis , memorability , attributes  and aging . Unlike the prior data sets, which focus on robustness under variations of pose, lighting, occlusion, and scale, the data set is aimed understanding diversity with respect to intrinsic facial features.
Given the above issues, we were motivated to create the data set to obtain a scientific and computationally practical basis for ensuring fairness and accuracy in face recognition. At one extreme the challenge of diversity could be solved by building a data set comprised from the face of every person in the world. However, this would not be practical or even possible, let alone the significant privacy concerns. For one, our facial appearances are constantly changing due to ageing, among other factors. At best this would give a solution for a point in time. Rather, a solution needs to come from obtaining or generating a representative sample of faces with sufficient coverage and balance. That, however, is also not a simple task. There are many challenging questions: what does coverage mean computationally? How should balance be measured? Are age, gender and skin color sufficient? What about other highly personal attributes that are part of our identity, such as race, ethnicity, culture, geography, or visible forms of self-expression that are reflected in our faces in a myriad of ways? We realized very quickly that until these questions were answered we could not construct a complete and balanced data set of face images.
We formulated a new approach that would help answer these questions. We designed the data set to provide a scientific foundation for research into facial diversity. We reviewed the scientific literature on face analysis and studied prior work in fields as diverse as psychology, sociology, dermatology, cosmetology and facial surgery. We concluded that no single facial feature or combination of commonly used classifications – such as age, gender and skin color – would suffice. Therefore, we formulated a novel multi-modal approach that incorporates a diversity of face analysis methods. Based on study of the large body of prior work, we chose to implement a solid starter-set of ten facial coding schemes. The criteria for selecting these coding schemes included several important considerations: (1) strong scientific basis as evidenced by highly cited prior work, (2) extracting the coding scheme was computationally feasible, (3) the coding scheme produced continuous valued dimensions that could feed subsequent analysis, as opposed to generating only categorical values or labels, and (4) the coding scheme would allow for human interpretation to help with our understanding.
We chose YFCC-100M  to be the source for the sample of images. There were a number of important reasons for this. Ideally, we would be able to automatically obtain any large sample of images from any source meeting any characteristics of diversity we desire. However, practical considerations prevent this, including the fact that various copyright laws and privacy regulations must be respected. YFCC-100M is one of the largest image collections, consisting of more than millions photos. It was populated by users of the Flickr photo service. There is a large diversity in these photos overall, where people and faces appear in an enormous number of ways. Also, importantly, a large portion of the photos have Creative Commons license. The downside of using YFCC-100M is that there is skew in the Flickr user community that contributed the photos. We cannot rely on the set of users or their photos to be inherently diverse. A consequence of this is that the set of images used in the is not completely balanced on its own. However, it still provides the desired basis for studying methods for characterizing facial diversity.
|(a) face detection (size)||(b) face keypoints (pose and iod)||(c) face rectification|
Each candidate photo from YFCC-100M was processed by first detecting the depicted faces with a Convolutional Neural Network (CNN) using the Faster-RCNN based object detector. Then each detected face as in (a) was processed using DLIB  to extract pose and landmark points as shown in (b) and subsequently assessed based on the width and height of the face region. Faces with region size less than 50x50 or inter-ocular distance of less than pixels were discarded. Faces with non-frontal pose, or anything beyond being slightly tilted to the left or the right, were also discarded. Finally, an affine transformation was performed using center points of both eyes, and the face was rectified as shown in (c).
While the YFCC-100M photo data set is large, not all images could be considered. Naturally, we excluded photos that did not contain a face. We also excluded black and white and grayscale photos and those with significant blur. Although face recognition needs to be robust for non-color photos, we deferred incorporating these images in the initial data set in order to focus on intrinsic facial variation rather than image variation due to color processing.
The YFCC-100M data set gives a set of URLs that point to the Flickr web page for each of the photos. The first step we took was to check whether the URL was still active. If so, we then checked the license. We proceeded with the download only if the license type was Creative Commons. Once we retrieved the photo, we processed it using face detection to find all depicted faces. For the face detection step, we used a Convolutional Neural Network (CNN) object detector trained for faces based on Faster-RCNN . For each detected face, we then extracted both pose and face key-points using the open source DLIB toolkit . If there was any failure in the image processing steps, we excluded the face from further consideration. We also removed faces of size less than pixels or with inter-ocular distance of less than pixels. We removed faces with substantial non-frontal pose. The overall process is shown in Figure 1.
Finally, we generated two instances of each face. One is a rectified instance whereby the center points of each eye are fixed to a specific location in the overall image. The second crops an expanded region surrounding each face to give 50% additional spatial context. This overall process filtered the million YFCC-100M photos down to approximately one million mostly frontal faces with adequate size. The surviving face images were the ones used for the data set. Note that the overall process of sampling YFCC-100M used only factors described above including color, size, quality and pose. We did not bias the sampling towards intrinsic facial characteristics or by using metadata associated with each photo, such as a geo-tag, date, labels or Flickr user name. In this sense, the data distribution is expected to closely follow the overall distribution of the YFCC-100M photos. In future efforts to grow the data set, we may relax some of the constraints based on size, pose and quality, or we may bias the sampling based on other properties. However, one million publicly available face images provides a good start. Given this compiled set of faces, we next process each one by extracting the ten facial coding schemes.
In this Section, we describe the implementation of the ten facial coding schemes and the process of extracting them from the face images. The advantage of using ten coding schemes is that it gives a diversity of methods and allows us to compare statistical measures for facial diversity. As described above, the ten schemes have been selected based on their strong scientific basis, computational feasibility, numerical representation and interpretability. Overall the chosen ten coding schemes capture multiple modalities of facial features, which includes craniofacial distances, areas and ratios, facial symmetry and contrast, skin color, age and gender predictions, subjective annotations, and pose and resolution. Three of the facial coding schemes are based on craniofacial features. As prior work has pointed out, skin color alone is not a strong predictor of race, and other features such as facial proportions are important [63, 64, 65, 6]. Face morphology is also relevant for attributes such as age and gender . We incorporated multiple facial coding schemes aimed at capturing facial morphology using craniofacial features [2, 3, 4]. The basis of craniofacial science is the measurement of the face in terms of distances, sizes and ratios between specific points such as the tip of the nose, corner of the eyes, lips, chin, and so on. Many of these measures can be reliably estimated from photos of frontal faces using landmark points of the head and face . To provide the basis for the three craniofacial feature coding schemes used in , we built on the subset of facial landmarks listed in Table 5. For brevity we adopt the abbreviations from  when referring to these facial landmark points instead of using the full anatomical terms.
|Anatomical term||Abbreviation||Anatomical term||Abbreviation|
|palpebrale superius||crista philtre|
|palpebrale inferius||labiale superius|
In order to extract the facial landmark points, we leveraged standard DLIB facial key-point extraction tools that provide a set key-points for each face. As shown in Figure 2, we mapped the DLIB key-points to the facial landmarks . These landmarks were used for extracting the craniofacial features. Note that for illustrative purposes, the example face used in Figure 2 was adopted from  and was generated synthetically using a progressive Generative Adversarial Network (GAN) model. The face does not correspond to a known individual person. However, the image is subject to license terms as per . In order to incorporate a diversity of approaches, we implemented three facial coding schemes for craniofacial features. The first, coding scheme 1, provides a set of craniofacial distance measures from . The second, coding scheme 2, provides an expanded set of craniofacial areas from . The third, coding scheme 3, provides a set of craniofacial ratios from .
The first coding scheme for craniofacial distances has been adopted from . It comprises eight measures which characterize all the vertical distances between elements in a face: the top of the forehead, the eyes, the nose, the mouth and the chin. In referring to the implementation of the coding scheme, we use the abbreviations from Table 5. We note that two required points, and , were not part of the set of 68 DLIB key-points. As such, we had to derive them in the following manner: was computed as the topmost point vertically above in the rectified facial image, and was computed from the vertical average of and . The eight dimensions of craniofacial distances are summarized in Table 6.
|intercanthal face height|
|eye fissure height (left and right)|
|orbit and brow height (left and right)|
|upper lip height|
|lower vermilion height|
|lateral upper lip heights (left and right)|
The second coding scheme is adopted from a later development from Farkas et al. . It comprises measures corresponding to different areas of the cranium. Similar to the craniofacial distances, the extraction of craniofacial areas relied on the mapped DLIB key-points to the corresponding facial landmarks. Table 7 summarizes the twelve dimensions of the craniofacial area features.
|Orbits intercanthal width|
|Orbits fissure length (left and right)|
|Orbits biocular width|
The third coding scheme comprises measures corresponding to different ratios of the face. These features were used to estimate age progression from faces in the age groups of 0 to 18 in . Similar to the above features, the craniofacial ratios used the mapped DLIB key-points as facial landmarks. Table 8 summarizes the eight dimensions of the craniofacial ratio features.
|Orbital width index (left and right)|
|Eye fissure index (left and right)|
|Vermilion height index|
|Mouth-face width index|
Facial symmetry has been found in psychology and anthropology studies to be correlated with subjective and objective traits including expression variation  and attractiveness . We adopted facial symmetry for coding scheme 4, given its intrinsic nature. To represent the symmetry of each face we computed two measures, following the work of Liu et al. . We processed each face as shown in Figure 3. We used three of the DLIB key-points detected in the face image to spatially normalize and rectify it to the following locations: the inner canthus of each eye (C1 and C2) to reference locations , and the philtrum C3 was mapped to . Next, the face mid-line (point in Figure 3(a)) was computed as the line passing through the mid-point of the line segment connecting (point in Figure 3(a)) and the philtrum C3.
We point out that although a face image is spatially transformed during rectification, facial symmetry with respect to the face mid-line is preserved according to the topological properties of the affine transformation . Each image is then cropped to 128x128 pixels to create a squared image with the face mid-line centered vertically. Next we convert the spatially transformed image to grayscale to measure intensity. Each point on this normalized face intensity image on the left of the face mid-line has a unique corresponding horizontally mirrored point on the other side of the face image (right of the mid-line). We also extract edges in this image to produce using a Sobel filter. Finally, we compute two facial symmetry measures based on density difference and edge orientation similarity as follows: for each pixel in the left 128x64 part ( and ) and the corresponding 128x64 right part ( and ) are computed as summarized in Table 9, where is the angle between the two edge orientations of images and at pixel . We compute the average value of and to be the two measures for facial symmetry.
|Edge orientation similarity|
It is interesting to notice that the two symmetry measurements capture facial symmetry from different perspectives: density difference is affected by the left-right relative intensity variations of a face, while edge orientation similarity is affected by the zero-crossing of the intensity field. Higher values of density difference correspond to more asymmetrical faces, while the higher the values of edge orientation similarity refer to more symmetrical faces.
Prior studies have shown that facial contrast is a cross-cultural cue for perceiving facial attributes such as age. An analysis of full face color photographs of Chinese, Latin American and black South African women aged – in  found similar changes in facial contrast with ageing across races and were comparable to changes with Caucasian faces. This study found that high-contrast faces were judged to be younger than low-contrast faces. The study also found that artificially increasing the aspects of facial contrast that decrease with age across diverse races makes faces look younger, independent of the ethnic origin of the face or cultural origin of the observers . On one hand, the age that you are is one dimension that needs to be addressed in terms of fairness and accuracy of face recognition. However, the age that you look, considering possible artificial changes, should not change requirements for fairness and accuracy.
We adopted facial regions contrast as the basis for coding scheme 5. To compute facial contrast, we measured contrast individually for each image color channel , , , corresponding to the CIE-Lab color space, for three facial regions: lips, eyes, and eyebrows, as shown in Figure 4. First, we defined the internal regions ringed by facial key points computed from DLIB for each of these facial parts (shown as the inner rings around lips, eyes, and eyebrows in Figure 4). Then, we expanded this region by 50% to define an outer region around each of these facial parts (shown as the outer rings in Figure 4). The contrast is then measured as the difference between the average pixel intensities in the outer and inner regions. This is repeated for each of the three CIE-Lab color channels. Given the three facial regions, this gives a total of nine measures, where the contrast values for the eyes and eyebrows are based on the average of the left and right regions. The computation is summarized in Table 10, where is the pixel intensity at for CIE-Lab channel and correspond to the outer and inner regions around each facial part .
|Facial region contrast||Measure|
Skin occupies a large fraction of the face. As such, characteristics of the skin influence the appearance and perception of faces. Prior work has studied different methods of characterizing skin based on skin color [7, 69, 70], skin type [7, 38] and skin reflectance 
. Early studies used Fitzpatrick skin type (FST) to classify sun-reactive skin types, which was also adopted recently in . However, to-date, there is no universal measure for skin color, even within the dermatology field. In a study of participants in South Africa, self-identified as either black, Indian/Asian, white, or mixed, Wilkes et al. found a high correlation between the Melanin Index (MI), which is frequently used to assign FST, with Individual Typology Angle (ITA) . Since a dermatology expert is typically needed to assign the FST, the high correlation of MI and ITA indicates that ITA may be a practical method for measuring skin color given the simplicity of computing ITA. In order to explore this further, we designed coding scheme 6 to use ITA for representing skin color . ITA has a strong advantage over Fitzpatrick in that it can be computed directly from an image. As in , we implemented ITA in the CIE-Lab space. For obvious practical reasons, we could not obtain measurements through a device directly applied on the skin of each individual, but instead converted the image to CIE-Lab space using standard image processing. The axis quantifies luminance, whereas quantifies absence or presence of redness, and quantifies yellowness. Figure 5 depicts the image processing steps for extracting the coding scheme 6 for skin color.
It is important to note that that ITA is a point measurement. Hence, every pixel corresponding to skin can have an ITA measurement. In order to generate a feature measure for the whole face, we extract ITA for pixels within a masked face region as shown in Figure 5(g). This masked region is determined in the following steps:
Extract the skin mask in the face (pixels corresponding to skin) using a deep neural network, as described in .
Extract regions corresponding to the chin, two cheeks and forehead using the extracted 68 DLIB key-points
Smooth the ITA values of each region to reduce outliers using an averaging filter
Pick the peak value of each region to give its ITA score
Average the values to give a single ITA score for each face
Table 11 gives the formula for computing the ITA values for each pixel in the masked face region.
|Individual Typology Angle (ITA)|
Age is an attribute we all possess and our faces are predictors of our age, whether it is our actual age or manipulated age appearance . As discussed in Section 4.5, particular facial features such as facial contrast are correlated with age. As an alternative to designing specific feature representations for predicting age, for coding scheme 7, we adopt a Convolutional Neural Network (CNN) that is trained from face images to predict age. We adopt the DEX model [74, 8] that is among the highest performing on some of the known face image data sets. The model is based on a pre-trained VGG16-face neural network for face identity that was subsequently fine-tuned on the IMDB-wiki data set  to predict age (years in the range 0-100). Since the DEX model was trained within a narrow context, it is not likely to be fair. However, our initial use here is to get some continuous measure of age in order to study diversity. Ultimately, it will require an iterative process of understanding diversity to make more balanced data sets and create more fair models. In order to predict age using DEX, each face was pre-processed as in 
. First, the bounding box was expanded by 40% both horizontally and vertically, then resized to 256x256 pixels. Inferencing was then performed on the 224x224 square cropped at the center of the image. Since softmax loss was used during the fine-tuning process, age prediction is output from the softmax layer, which is computed from, where are the softmax output probabilities for each of the 101 discrete years corresponding to each class , with .
Coding scheme 8 follows a similar process for gender prediction as for age prediction . We used the same pre-processing steps as described in Section 4.7 as well as the same neural network model and training pipeline. The only difference is that we use the DEX model to predict a continuous value score for gender between and , and not just report a binary output.
Coding scheme 9 aims at capturing age and gender but through subjective means rather than using a neural network-based predictive model. For each of the face images, we employed the Figure Eight crowd-sourcing platform  to obtain subjective human-labeled annotations of gender and age. The gender annotations used two class labels (male and female) and the age group labeling used seven classes ([0-3],[4-12],[13-19],[20-30],[31-45],[46-60],[61-]), as well as a continuous age value to be consistent with the automatic prediction labels. For each face, input was taken from three independent annotators. A weighted voting scheme was used to aggregate the labels, where the vote of each annotator was weighted according to their performance on a set of “gold standard” faces for which the ground truth was known.
The final coding scheme 10 provides information about pose and resolution. Although pose can only loosely be considered an intrinsic facial attribute, how we present our faces to cameras should not affect performance. We include resolution as it gives useful information to correlate with the other coding scheme features to provide further insight. In order to extract pose, we use the DLIB toolkit to compute a pose score of 0-4. Here, the values correspond as follows: 0-frontal, 1-rotated left, 2-rotated right, 3-frontal but tilted left, 4-frontal but tilted right. Along with pose, resolution is determined from the size of the bounding box of each face and inter-ocular distance (IOD), which is the distance between the center points of each eye.
In this Section, we report on the statistical analysis of the ten facial coding schemes in the data set. Intuitively, in order to provide sufficient coverage and balance, a data set needs to include data with diverse population characteristics. This type of analysis comes up in multiple disciplines, including bio-diversity [76, 77], where an important objective is to quantify species diversity of ecological communities. It has been reported that species diversity has two separate components: (1) species richness, or the number of species present, and (2) their relative abundances, called evenness. We use these same measures to quantify the diversity of face images using the ten facial coding schemes. We compute diversity using Shannon and scores and Simpson and scores 
. Additionally, we measure mean and variance for each of the feature dimensions of the ten facial coding schemes The computation of diversity is as follows: given individual
in a probability distribution for each feature measure, and thebeing the number of classes for the attribute, we compute:
Shannon and Simpson are diversity measures and Shannon and Simpson are evenness measures. To see how they work, consider a class problem (
) with uniform distribution (). These measures take the following values: Shannon , Shannon , Simpson , and Simpson . Evenness is constant at as expected. Shannon represents the diversity of classes (). For complex distributions, it may not be easy to understand the meaning of specific values of these scores. Generally, a higher diversity value is better than a lower value, whereas an evenness value closer to is better. Figure 6 illustrates these measures on two example distributions. Figure 6 (a) and (b) show how diversity and evenness values vary for a uniform distribution, respectively, as the number of classes increase from to . Figure 6 (c) and (d) show the same information for a random distribution.
Table 12 summarizes the diversity scores computed for the ten facial coding schemes in the data set. As described in Section 4, many of the coding schemes have multiple dimensions. Hence the table has more than ten rows. The craniofacial measurements across the three coding scheme types total features corresponding to craniofacial distances, craniofacial areas and craniofacial ratios. The diversity scores of the different dimensions of the remaining seven coding schemes can similarly be seen in Table 12.
|Facial Symmetry||Density difference||4.777||0.796||1.620||0.904||0.12||0.06|
|Edge or. similarity||5.005||0.834||1.692||0.944||0.01||0.01|
|Facial Contrast||Lips contrast||5.857||0.976||1.779||0.993||-0.07||0.09|
|Subjective Annotation||Gender labeling||2.000||1.000||0.693||1.000||0.49||0.50|
|Pose & Resolution||Pose||1.224||0.408||0.390||0.355||-0.01||0.31|
|Face Region Size||2.819||0.470||1.198||0.668||93.37||42.98|
Figure 7 summarizes the feature distribution for the craniofacial distances in coding scheme 1. The highest Simpson value is and the lowest is . The highest and lowest Shannon values are and . Based on the Shannon values, this feature dimension would typically map to classes. Evenness is generally balanced with highest Simpson and Shannon of and , respectively.
Figure 8 summarizes the feature distribution for the craniofacial areas in coding scheme 2. The highest Simpson value is and the smallest is . The highest Shannon value is and the lowest is . Compared to coding scheme 1, these values are in the similar range, mapping to classes. Evenness ranges between and .
Figure 9 summarizes the feature distribution for the craniofacial ratios in coding scheme 3. Unlike the previous coding scheme 2, the diversity values for this coding schemes have less variance. The largest Simpson value is and smallest is . Similarly, the largest Shannon value is and smallest is . This would map to approximately to classes. While Simpson has a range between to , Shannon ranges between to . The evenness of coding scheme 3 is similar to coding scheme 2.
Figure 10 summarizes the feature distribution for facial symmetry in coding scheme 4. The diversity value is in a middle range compared to the previous coding schemes. For example, the highest Simpson is and the largest Shannon is . The evenness values are lower as well with highest Simpson value being and highest Shannon value being . The Shannon value of translates to about classes.
Figure 11 summarizes the feature distribution for facial contrast in coding scheme 5. The highest Simpson value is and highest Shannon value is , which is equivalent to classes. The evenness factor Shannon is very close to indicating that the measures are close to even.
Figure 12 summarizes the feature distribution for skin color in coding scheme 6. The Simpson value is and Shannon value is which translates to about classes, which shows a good match with the number of bins we used. The evenness is weaker than a uniform distribution.
Figure 13(a) summarizes the feature distribution for age prediction in coding scheme 7, where we bin the age values into seven groups: [0-3],[4-12],[13-19],[20-30],[31-45],[46-60],[61-]. The Simpson and Shannon values are and . Because of the data distribution not being even, we can see a lower value around . The Shannon value of maps to classes.
Figure 13 also summarizes the feature distribution for gender prediction in coding scheme 8. Even though this has two classes, male and female, the confidence score ranges between . The gender score distribution is shown in Figure 13 (b). The Simpson is and Shannon is . The Shannon value translates to classes, which is beyond the typical two classes used for gender, possibly reflecting the presence of sub-classes. The Shannon evenness score of reflect some unevenness as well.
Figure 14 summarizes the feature distribution for the subjective annotations of age and gender for coding scheme 9. The Simpson for gender distribution is and Shannon is , indicating the equivalent classes to be near , which is understandable. The evenness is very high, indicating a nearly flat distribution. The Simpson is and Shannon is , resulting in a equivalent class index of . However, the evenness scores are low at , indicating unevenness, as is visible in the distribution of the annotated age scores.
Figure 15 summarizes the feature distribution for pose and resolution for coding scheme 10. Pose uses three dimensions from the output of DLIB face detection and the distribution is shown in 15 (a). When computing mean and variance for pose in Table 12, we used the following values: Frontal Tilted Left -1, Frontal 0, and Frontal Tilted Right 1. The IOD and box size distribution are shown in Figure 15 (b)-(c). The distances have been binned to six classes. The three class pose distribution has a Shannon value of . The Shannon value for IOD is (mapping to equivalent of classes) while for the box size it is , translating to classes.
Some observations come from this statistical analysis of the ten coding schemes on the face image data. One is that the many of the dimensions of the craniofacial schemes have high scores in diversity relative to the other coding schemes. Generally, they are higher than measures used for age and gender, whether using a predictive model or subjective annotation. Similarly, their evenness scores are also closer to one. What this shows is that there is higher variability in these measures, and they are capturing information that age and gender alone do not. Interestingly, facial regions contrast, which was designed to capture information about age, has a higher diversity score and better evenness than that for either neural network prediction of age or subjective human annotation of age. Again, it implies that this continuous valued feature of facial contrast is capturing information that goes beyond simple age prediction or labeling. The only feature dimension with lower diversity is understandably pose, which was a controlled variable in selecting images for the data set, since only mostly frontal faces were incorporated. In future work, we will use these methods to assess diversity of other face data sets, which will provide a basis for comparison and further insight.
We described the new Diversity in Faces () data set, which has been developed to help advance the study of fairness and accuracy in face recognition technology. provides a data set of annotations of publicly available face images sampled from the YFCC-100M data set of 100 million images. The annotations are defined from facial coding schemes that provide quantitative measures related to intrinsic characteristics of faces including craniofacial features, facial symmetry, facial contrast, skin color, age, gender, subjective annotations and pose and resolution. We described the process for generating the data set as well as the implementation and extraction of the ten facial coding schemes. We also provided a statistical analysis of the facial coding scheme measures on the one million images using measures of diversity, evenness and variance. For one, this kind of analysis has provided insight into how the total feature dimensions within the ten facial coding schemes provide measures of data set diversity for the one million images. While it may not yet be possible to conclude that the goal is to drive all of these feature dimensions to be maximally diverse and even, we believe the approach outlined in this work provides a needed methodology for advancing the study of diversity for face recognition.
There are multiple next directions for this work. Table 4 outlined many of the currently used face data sets. We plan to perform the equivalent statistical analysis on some of these other data sets using the ten coding schemes. This will provide an important basis for comparing data sets in terms of diversity. Using the statistical measures outlined in this paper, including diversity, evenness and variance, we will begin to answer questions of whether one data set is better than another, or where a data set falls short in terms of coverage and balance. We also strongly encourage others to build on this work. We selected a solid starting point by using one million publicly available face images and by implementing ten facial coding schemes. We hope that others will find ways to grow the data set to include more faces. As more insight comes from the type of analysis outlined in this paper, we see that an iterative process can more proactive sampling to fill in gaps. For example, as technologies like Generative Adversarial Networks (GANs) continue to improve [66, 78], it may be possible to generate faces of any variety to synthesize training data as needed. We expect that will be valuable to steer these generative methods towards gaps and blind spots in the training data, for example, to assist methods that automatically generate faces corresponding to a specific geographical area . We also hope that others will see ways to improve on the initial ten coding schemes and add new ones. Pulling together our collective efforts is the best way to make progress on this important topic. We hope that the data set provides a useful foundation for creating more fair and accurate face recognition systems in practice.
Age progression/regression by conditional adversarial autoencoder.In Intl. Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
Equality of opportunity in supervised learning.In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3315–3323. Curran Associates, Inc., 2016.