Automatic kinship recognition in visual media is essential for many real-world applications: e.g., kinship verification [6, 8, 11, 26, 29, 30, 31, 32], automatic photo library management [27, 22], historic lineage and genealogical studies , social-media analysis , along with many security applications involving missing persons, human trafficking, crime scene investigations, and even our overall human sensing capabilities– ultimately, enhancing surveillance systems used in both real-time (e.g., vBOLO  mission111vBOLO: joint effort of two DHS Centers of Excellence, ALERT & VACCINE (http://www.northeastern.edu/alert)222https://web-oup.s3-fips-us-gov-west-1.amazonaws.com/) or offline (e.g., searching for a subject in a large gallery [3, 24]
). Thus, a gallery of imagery annotated with rich family information should yield more powerful multimedia retrieval tools and complement many existing facial recognition systems (e.g., FBI’s NGI333https://www.eff.org/2014/fbi-to-have-52M-face-photos). However, even after several years (i.e., since 2010 ) there are only a few vision systems capable of handling such tasks. Hence, kin-based technology in the visual domain has yet to truly transition from research-to-reality.
We believe the reason that kinship recognition technology has not yet advanced to real-world applications is two-fold:
Current image datasets available for kinship tasks are not large enough to reflect the true data distributions of a family and their members.
Visual evidence for kin relationships are less discriminant than class types of other, more conventional machine vision problems (e.g., facial recognition or object classification), as many hidden factors affect the similarities of facial appearances amongst family members.
To that end, we introduce a large-scale image dataset for kinship recognition called Families in the Wild (FIW). To the best of our knowledge, FIW is by far the largest and most comprehensive kinship dataset available in the vision and multimedia communities (see Table 1). FIW includes unconstrained family photos of families, which is nearly more than the next-to-largest, Family101 . Also, it is from these families that image pairs for the relationship types (see Figure 1 & Table 2).
Thus far, attempts at image-based kinship recognition have focused on facial features, as is the case in this work. Kinship recognition is typically conducted in one or more of the following modes: (1) kinship verification, a binary classification problem, i.e., determine whether or not two or more persons are blood-relatives (i.e., kin or non kin); (2) family recognition, a multi-class classification problem, where the aim is to determine the family an individual belongs to; (3) kinship identification, a fine-grain categorization problem, with the goal of determining the types of relationships shared between two or more people. We focus on (1) and (2) in this paper, which we briefly discuss next.
Kinship Verification. Previous efforts mainly focused on the
parent-child relationship types. As research in psychology and computer vision found, different relationship types render different familiar features, and thekin relations are usually modeled independently. Thus, it is best to have more kin relationship types accessible– FIW provides pair-wise types (see Figure 1): existing types (i.e., parent- child and siblings), but with sample sizes scaling up to larger, and types being offered for the first time (i.e., grandparent-grandchild). Also, existing datasets contain, at most, only a couple of hundred pairs per category (i.e., in total). Such a insufficient amount leads to models overfitting the training data. Hence, existing models do not generalize well to new, unseen test data. However, FIW now makes pairs available.
Family Recognition. A challenging task that grows more difficult with more families. This is because families contain large intra-class variations, often overlapping between classes. Similar to conventional face recognition, when the targets are unconstrained faces in the wild  (i.e., variation in pose, illumination, expression, and scene) the difficulty level further increases, and the same being true for kinship recognition. These are, unfortunately, challenges that need to be overcome. Thus, FIW poses realistic challenges needed to be addressed before deploying to real-world applications.
Contributions. We make three distinct contributions:
We introduce the largest visual kinship dataset to date, Families in the Wild (FIW).444FIW will be available upon publication of this paper. FIW is complete with rich label information for family trees, from which kin relationship types were extracted in numbers that are orders of magnitude times larger than existing datasets. This was made possible with an efficient annotation tool and procedure (Section 2).
We provide several benchmarks on FIW for both kinship verification and family recognition, including various low-level features, metric learning methods, and pre-trained Convolutional Neural Network (CNN) models. See Section 3 for experimental settings and results.
We fine-tune two CNNs: one with a triplet-loss layer on top, and the other with a softmax loss. Both yield a significant boost in performance over all other benchmarks for both tasks (Section 3.2).
2 Families in the Wild
We now discuss the procedure followed to collect, organize, and label family photos of families with minimal manual labor. Then, we statistically compare FIW with other related datasets.
2.1 Building FIW
The goal for FIW was to collect around photos for families, each with at least family members. We now summarize the method for achieving this in a three step process, which is visually depicted in Figure 2.
Step 1: Data Collection. A list of over candidate families was made. To ensure diversity, we targeted groups of public figures worldwide by searching for <ethnicity OR country> AND <occupation> online (e.g., MLB [Baseball] Players, Brazilian Politicians, Chinese Actors, Denmark + Royal Family). Family photos were then collected using various search engines (e.g., Google, Bing, Yahoo) and social media outlets (e.g., Pinterest) to widen the search space. Those with at least 3 family members and family photos were added to FIW under an assigned Family ID (FID).
Step 2: Data Annotation. A tool was developed to quickly annotate a large corpus of family photos. All photos for a given FID are labeled sequentially. Labeling is done by clicking a family member’s face. Next, a face detector initializes a resizable box around the face. Faces unseen by the detector are discarded, as these are assumed to be poorly resolved. The tool then prompts for the name of the member via a drop-down menu. For starters, option new adds a member to a family under a unique member ID (MID), which prompts for the name, gender, and relationship types shared with others previously added to the current family (or FID). From there onward, labeling family members is just a matter of clicking.
Step 3: Dataset Parsing. Two sets of labels are generated in Step 2: (1) image-level, containing names and corresponding facial locations; (2) family-level, containing a relationship matrix that represents the entire family tree. Relationship matrices are then referenced to generate lists of member pairs for the 11 relationship types. Next, face detections are normalized and cropped with , then are stored according the previously assigned FID MID. Lastly, lists of image pairs are generated for both kinship verification and family recognition.
2.2 Database Statistics
Our FIW dataset far outdoes its predecessors in terms of quantity, quality, and purpose. FIW contains family photos of different families. There are about images per family that include at least and as many as family members. We compare FIW to related datasets in Table 1 and 2. Clearly, FIW provides more families, identities, facial images, relationship types, and labeled pairs– the pair count of FIW is orders of magnitude bigger than the next- to-largest (i.e., KFW-II).
3 Experiments On FIW
In this section, we first discuss visual features and related methods used to benchmark the FIW dataset. We then report and review all benchmark results. Finally, we discuss the two methods used to fine-tune the pre-trained CNN model: (1) training a triplet-loss for kinship verification and (2) learning a softmax classifier for family recognition. Top scores were obtained for both in the respective task.
|Dataset||No. Family||No. People||No. Faces||Age Varies||Family Trees|
3.1 Features and Related Methods
All features and methods covered here were used to bench- mark FIW. First, we review handcrafted features, Scale In- variant Feature Transformation (SIFT) and Local Binary Patterns (LBP), which are both widely used in kinship verification  and facial recognition . Next, we introduce VGG-Face, the pre-trained CNN model used here as an off- the-shelf feature extractor. Lastly, we review other related metric learning methods.
with a stride of. Thus, there were a total of
blocks for each image, yielding a feature vector of length.
|KFW-II ||Sibling Face ||Group Face ||Family 101||FIW (Ours)|
LBP  has been frequently used for texture analysis and face recognition, as it describes the appearance of an image in a small, local neighborhood around a pixel. Once again, we followed the feature settings of  by first resizing each facial image to , and then extracting LBP features from non-overlapping blocks with a radius of 2 pixels and number of neighbors (i.e., samples) set to 8. We then generated a 256D histogram from each, yielding a final feature vector of length .
VGG-Face CNN  uses a “Very Deep” architecture with very small convolutional kernels (i.e., ) and convolutional stride (i.e., pixel). This model was pre-trained on over million images of celebrities. For this, each face image was resized to and then fed-forward to the second-to-last fully-connected layer (i.e., fc7) of the CNN model, producing a D feature vector.
Metric Learning methods are commonly employed in the visual domain, some designed specifically for kinship recognition [5, 16, 18] and some as generic metric learning methods . We chose two representative methods from these two categories to report benchmarks on, Neighborhood Repulsed Metric Learning (NRML)  and Information Theoretic Metric Learning (ITML) .
3.2 Fine-Tuned CNN Model
Weights of deep networks are trained on larger amounts of generic source data, then fine-tuned on target data, which utilize a wider, more readily available source domain that resembles the target in either modality, view, or both 
. Following this notion, and motivated by recent success with deep learning on faces[17, 23, 25], we fine-tuned the VGG-Face model, improving results for both kinship verification and family recognition.
Kinship Verification. In kinship verification, for each fold we select the families which have more than images in the rest four folds. images of each family are used to fine-tune the model and the rest for validation. An average of images are selected. The experimental results of this part can be found in Table 3 and Figure 3. This is so far the best results on FIW. Specifically, we remove the last fully-connected layer which is used to identify people and employed a triplet-loss  as the loss function. The second-to-last fully-connected layer was the only non-frozen layer of the original CNN model, with an initial learning rate of that decreased by a factor of every iterations (out of ). Batch size was
images, and other network settings were the same as the original VGG-Face model. Training was done on a single GTX Titan X with about 10GB GPU memory. Fine-tuning was done using the renown Caffe framework.
As visual kinship recognition is essentially related to facial features, we froze the weights in the lower levels of the VGG-Face network, and replaced the topmost layer with a new softmax layer to classify thefamilies. Other settings are the same as those used for kinship verification.
3.3 Experimental Settings
In this section, we provide benchmarks on FIW using all features and related methods mentioned in the previous section. Dimensionality of each feature is reduced to 100D using PCA. Experiments were done following a 5-fold cross-validation protocol. Each fold was of equal size and with no family overlap between folds.
Experiment 1: Kinship Verification.
We randomly select an equal number of positive and negative pairs for each fold family. Cosine similarity is computed for each pair in the test fold. The average verification rate of all folds is reported in Table3, showing that kinship verification is a challenging task. While some relation are relatively easy to recognize, e.g., B-B, SIBS, S-S through SIFT, LBP, and VGG-Face features, results of other relations such as parent- child are still below 70.0%. Clearly, VGG-Face features are much better than hand-craft features. Notice grandparent- child pairs typically have higher accuracies than parent-child, which we believe is due to the differences in sample sizes. We also compare with the state-of-the-art metric learning methods, NRML and ITML (see Figure 3). Showing improved scores for the low-level features, but still outperformed by the CNN model fine-tuned on FIW.
Experiment 2: Family Recognition. We again follow the 5-fold cross-validation protocol with no family overlap. Families with 6 or more members, from which the 5 members with the most images were used. The results in Table 4 are from 316 families with 7,772 images. Folds were made up of one member for each family. Multi-class SVM was used to model VGG-Face features for each family (i.e., one-vs-rest). We then improved the top-1 classification accuracy from 12.31.8 (%, VGG-Face) to 13.31.6 (%, our fine-tuned model).
We introduced a large-scale dataset of family photos captured in natural, unconstrained environments (i.e., Families in the Wild). An annotation tool was designed to quickly generate rich label information for photos of families. Emphasis was put on diversity (i.e., families worldwide), data distribution (i.e., at least 3 members and 8 photos per family), sample sizes (i.e., multiple instances of each member, and at various ages), quality (i.e., only faces seen by the detector), and quantity (i.e., much more data and new relationship types).
There are many interesting directions for future work on our dataset. We currently only verify whether a pair of images is kin or non kin. However, also predicting the kin relationship type could lead to more value and practical usefulness. Thus, bringing us closer to doing fine-grain categorization on entire family trees. FIW will be an ongoing effort that will continually grow and evolve. See project page for downloads, updates, and to learn more about FIW.
This material is based upon work supported by the U.S. Department of Homeland Security, Science and Technology Directorate, Office of University Programs, under Grant Award 2013-ST-061-ED0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security.
We would also like to thank all members of SMILE Lab who helped with the process of collecting and annotating the FIW dataset.
-  T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE TPAMI, 28, 2006.
-  M. Almuashi, S. Z. M. Hashim, D. Mohamad, M. H. Alkawaz, and A. Ali. Automated kinship verification and identification through human facial images: a survey. Multimedia Tools and Applications, 2015.
-  N. Crosswhite, J. Byrne, O. M. Parkhi, C. Stauffer, Q. Cao, and A. Zisserman. Template adaptation for face verification and identification. CoRR, abs/1603.03958, 2016.
-  J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon. Information-theoretic metric learning. In Proc. ACM ICML, 2007.
A. Dehghan, E. Ortiz, R. Villegas, and M. Shah.
Who do i look like? determining parent-offspring resemblance via gated autoencoders.In Proc. IEEE Conference on CVPR, 2014.
-  H. Dibeklioglu, A. Salah, and T. Gevers. Like father, like son: Facial expression dynamics for kinship verification. In Proc. IEEE ICCV, 2013.
-  R. Fang, A. Gallagher, T. Chen, and A. Loui. Kinship classification by modeling facial feature heredity. In 20th IEEE ICIP, 2013.
-  R. Fang, K. D. Tang, N. Snavely, and T. Chen. Towards computational models of kinship verification. In IEEE ICIP, 2010.
-  R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection & semantic segmentation. CoRR, abs/1311.2524, 2013.
-  Y. Guo, H. Dibeklioglu, and L. van der Maaten. Graph-based kinship recognition. In ICPR, pages 4287–4292, 2014.
-  J. Hu, J. Lu, J. Yuan, and Y.-P. Tan. Large Margin Multi-metric Learning for Face and Kinship Verification in the Wild. Springer, 2015.
-  G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, UMass, Amherst, 2007.
-  Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. CoRR, abs/1408.5093, 2014.
D. E. King.
Dlib-ml: A machine learning toolkit.Journal of Machine Learning Research, 2009.
-  D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 2004.
-  J. Lu, X. Zhou, Y.-P. Tan, Y. Shang, and J. Zhou. Neighborhood repulsed metric learning for kinship verification. IEEE TPAMI, 2014.
-  O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In Proc. BMVC, 2015.
-  X. Qin, X. Tan, and S. Chen. Tri-subject kinship verification: Understanding the core of A family. CoRR, abs/1501.02555, 2015.
M. S. S. Xia and Y. Fu.
Kinship verification through transfer learning.In Proc. IJCAI, 2011.
-  F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proc. IEEE Conference on CVPR, 2015.
-  M. Shao, S. Xia, and Y. Fu. Genealogical face recognition based on ub kinface database. In IEEE CVPR WORKSHOPS, 2011.
-  M. Shao, S. Xia, and Y. Fu. Identity and Kinship Relations in Group Pictures. Springer, 2014.
-  Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10,000 classes. In Proc. IEEE Conference on CVPR, 2014.
-  D. Wang, C. Otto, and A. K. Jain. Face search at scale: 80 million gallery. CoRR, abs/1507.07242, 2015.
-  M. Wang, Z. Li, X. Shu, J. Tang, et al. Deep kinship verification. In IEEE 17th International Workshop on MMSP, 2015.
-  S. Xia, M. Shao, and Y. Fu. Toward kinship verification using visual attributes. In IEEE ICPR, 2012.
-  S. Xia, M. Shao, J. Luo, and Y. Fu. Understanding kin relationships in a photo. IEEE Transactions on Multimedia, 2012.
-  F. Xiong, M. Gou, O. Camps, and M. Sznaier. Person re-identification using kernel-based metric learning methods. In ECCV, 2014.
-  H. Yan, J. Lu, W. Deng, and X. Zhou. Discriminative multimetric learning for kinship verification. IEEE Transactions on Information Forensics and Security, 2014.
-  H. Yan, J. Lu, and X. Zhou. Prototype-based discriminative feature learning for kinship verification. IEEE Transactions on Cybernetics, 2015.
-  X. Zhou, J. Hu, J. Lu, Y. Shang, and Y. Guan. Kinship verification from facial images under uncontrolled conditions. In Proc. ACM-MM, 2011.
-  X. Zhou, J. Lu, J. Hu, and Y. Shang. Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments. In Proc. ACM-MM, 2012.