1 Introduction
Given a large corpus of images, we want to cluster them such that images semantically related are grouped in one cluster. Semantics of an image refer to the information that image carries. For example, the face on the image is usually used to identify who. The background of the image refers to the location where the person was. All components together can convey what story has happened. In our case, we focus on two entities: who and where.
While there has been considerable work on automatic face recognition [2, 23] in images and even a modest effort on location recognition [4, 3], the coupling of the two is basically unexplored. An image which contains both people and location implies the cooccurrence of instances in two domains. For example, multiple photos taken at the same private location increase the confidence that similar faces on those photos are from a same person. Within a short time window, the same person on several photos indicates the affinity of locations.
Our framework, shown in Fig. 1, consists of two domains: people and locations. We take into account the interrelation between two domains to enhance clustering in each domain. Three types of relations in people and location domains are considered: (1) peoplepeople (2) locationlocation (3) peoplelocation. A set of image patches is extracted and described in each domain. The similarity between patches within each domain is defined based on the visual appearances. The cooccurrence constraints are satisfied if patches from two domains appear in a same image. This relationship reflects the consistency of clustering results which is not embodied from visual appearances in a single domain.
We formulate the clustering task as an optimization problem which aims to minimize the within cluster distances and maximize the consistency across domains. We show this problem can convert to the semisupervised kernel kmeans clustering similar to
[12]. However, we generate clustering results for two domains at the same time. During the iterative clustering process, constraints across domains and within domains keep updated. The main idea is that the clustering result in one domain can aid the clustering in the other domain. We validate our approach with photos gathered from personal albums and a set of public photos crawled from Flickr.Our contributions are threefold: 1) we propose a coclustering algorithm for image clustering, focusing on people and locations; the algorithm couples both domains and explores underlying crossdomain relations; 2) our algorithm can simultaneously produce the clustering results of people and location, and outperforms clustering separately on each domain and the baseline coclustering algorithm; 3) our algorithm is formulated as an optimization problem, which can be solved by through semisupervised kernel kmeans. It is robust and converges fast in practice.
2 Related Work
Face is an important kind of visual objects in images, which is crucial to identify people. In recent years, there have been a lot of efforts in face detection [22], recognition [2, 23] and clustering [1]
. The basic idea is to either represent a face as one or multiple feature vectors, or parameterize the face based on some template or deformable models. In addition to treating faces as individual objects, some researchers have been seeking for help from context information, such as background, people cooccurrence, etc. Davis et al.
[5] developed a contextaware face recognition system that exploits GPStags, timestamps, and other metadata. Lin et al. [14] proposed a unified framework to jointly recognize the people, location and event in a photo collection based on a probabilistic model.Most location clustering algorithms are relying on the bag of words model [17]. Largescale location clustering has been recently demonstrated in [13, 24], which use the GPS information to reduce the largescale task down into a set of smaller tasks. Hays et al. [9]
proposed an algorithm for estimating a distribution over geographic locations from the query image using a purely datadriven scene matching approach. They leveraged a dataset of over 6 million GPStagged images from the Flickr. When the temporal data is available in the corpora, it also helps to localize sequences of images.
However, clustering in people and location domains are usually treated as separate tasks. Location patches in photos with faces are not well exploited. While GPS information of the photo is not easily accessible, we propose a coclustering algorithm, which simply use patches of the photo itself to discover the correlation in these two domains.
3 Our Approach
In this section, we present the coclustering framework to simultaneously cluster images in people and location domains. We have two major steps. The first step is preprocessing. We extract face and location patches from the corpus of images, and compute the visual features. The next step is coclustering. The peoplepeople, locationlocation and peoplelocation relations are generated and updated. We describe the detail for each step below.
3.1 Preprocessing
We here describe how to extract features from people and location domains, and discover the relations between both domains.
People Domain: We use ViolaJones face detector [22] to extract face patches from an image. To obtain high accuracy, a nested detector is applied to reduce the false positive rate. Every face will have a corresponding face patch. All face patches are normalized to the same size. We adopt the algorithm in [21] to detect seven facial landmarks from each extracted face patch. For each input face patch, four landmarks (outer eye corners and mouth corners) are registered to the predefined positions using the perspective transform. Then all seven facial landmarks are aligned by the computed perspective transform. For each landmark, two SIFT descriptors of different scales are extracted to form the face descriptor. We build a face graph over all face patches in the image collection. In the graph, each vertex represents a face patch. The weight of the edge is the similarity of face descriptors of two face patches.
Location Domain For each image, Hessian affine covariant detector [16] is used to detect interest points. The SIFT descriptor [15] is extracted on every interest point. The method similar to the work of Heath et al. [10]
is used to discover the shared locations in the image collection. The contentbased image retrieval
[17] is applied to find top related images, and avoid quadratic pairwise comparisons. Lowe’s ratio test [15] is used to find the initial correspondences. RANSAC [7] is used to estimate the affine transform between a pair of images and compute feature correspondences between images. For every location patch, two types of features are extracted: a bag of visual words [20] and a color histogram. The bag of words descriptor summarizes the frequency that prototypical local SIFT patches occur. It captures the appearance of component objects. For images taken in an identical location, this descriptor will typically provide a good match.The color histogram characterizes certain scene regions well. These two types of features are concatenated to represent the location patch. A location graph is built similarly to the face graph. Each vertex in the graph represents a location patch. The weight of the edge is the similarity of location descriptors of two location patches.Interrelations across Domains To cocluster across the people and location domains, several basic assumptions are made as follows.
Cannot Match Link. One person cannot appear twice in one image. Therefore, there is a cannot match link between a pair of face patches in the same image. Here we do not consider the exceptions like the photo collage or mirrors in the image. If two locations are far away according to the ground truth e.g. GPS signals, and two face patches appear in these two locations during a short time period, there is a cannot match link between this pair of patches. This assumption comes from that people cannot teleport within a short time period, for example, one people cannot appear in San Francisco and in New York within an hour.
Must Match Link. Two location patches are connected by a must match link if there is an affine transform found between them in the location graph construction. Because the links verified by RANSAC have high accuracy, we trust that they connect patches in the same location. Two location patches are connected by a must match link if they appear in the same image. Two different buildings may appear in the same image, therefore, in our setting, one location is defined as an area which may contain different backgrounds. Two location patches are connected by a must match link if they cooccur with the same people within a short time period. This assumption also comes from the fact that people cannot move too fast.
Possible Match Link.
Two face patches that appear in the same location but not in the same image probably belong to the same people, due to the strong cooccurrence between the location and the face. This is true if the place has special meaning to the person, for example, his/her home or office, where he/she visits frequently. However, the assumption is not always true. For example, at tourist attractions, every people would take photos there. Therefore, the locations do not contribute much for the clustering in people domain. A weight is needed for the locations to distinguish the public locations and private locations. Private location is more helpful for clustering in people domain, while public location will introduce many noise.
3.2 Problem Formulation
We formulate our people and location coclustering as an optimization problem. Given a set of feature vectors , the goal of the standard kmeans in each domain is to find a way disjoint partitioning such that the following objective is minimized:
(1) 
where is the cluster center of . The matrix is defined as pairwise squared Euclidean distances among the data points, such that . We introduce an indicator vector for the cluster .
(2) 
where is the size of cluster , and gives the sum of over all and in . Now the matrix is defined such that the th column of is equal to . is an orthonormal matrix, . Let be the number of face patches and be the number of location patches. is the number of face clusters and is the number of location clusters. By considering the relations between people and location domains, we write the objective as:
minimize  (3)  
subject to  
and are pairwise squared Euclidean distance matrices in people and location domains. To integrate the must match constraints and cannot match constraints, the distance of the must match link is set to and the distance of the cannot match link is set to . and with constraints and are the standard kmeans optimization problems in people and location domains respectively.
The binary peoplelocation cooccurrence matrix is defined as: the th column of is the location patches that cooccur with the face patch . For example, if the first column of is , which means the first face patch cooccurs with the third and the fifth location patches in the same image.
is a clustering of location patches which is based on the face clustering result . Our goal is to maximize the consistency between the location clustering and . Location patches are weighted differently to reflect different semantic meanings of the people and location interactions. It is not difficult to discover the similarity between the definitions of and except the weight matrix and . optimizes the consistency that locations cooccur with the same people during a short time period should be one location. is a binary diagonal matrix that nonzero entries on the diagonal indicate these location patches are taken within a short time period. For example, means the second, the fourth and the fifth location patches have similar timestamps. There are time constraints that are automatically learned from the metadata of images.
optimizes the consistency that private locations are useful to identify people. is a diagonal weight matrix. It defines a score for each location patches. The private locations have larger weights and the private locations have small weights. The diagonal matrix is defined as:
(4) 
where approximates at public locations such as landmarks and it is approximate at private locations. is the location cluster that belongs to. is the number of people appear in location .
3.3 Alternative Optimization
The optimization problem (3) is not convex when the optimization variables involve and . Therefore, we use the alternative optimization by fixing variables in one domain and optimize on other variables and do this iteratively. When fixing variables, e.g. . The problem becomes a semisupervised kernel kmeans problem, which can be solved easily. We solve the problem following this sequence until the convergence: . The first and are computed using the standard kernel kmeans without crossdomain relations. After the initial clustering results are known, the weight matrix can be computed using equation (4) and in the following iteration, the semisupervised kernel kmeans is used to integrate the crossdomains relations.
3.3.1 Semisupervised Kernel Kmeans
We now briefly describe the existing semisupervised kernel kmeans algorithm [12]. The objective is written as the minimization of:
(5) 
where is the set of must match link constraints, is the set of cannot match link constraints, is the penalty cost for violating a constraint between and , and refers to the cluster label of . The first term in this objective function is the standard kmeans objective function, the second term is a reward function for satisfying must match link constraints, and the third term is a penalty function for violating cannot match link constraints. The penalties and rewards are normalized by cluster size: if there are two points that have a cannot match link constraint in the same cluster, we will penalize higher if the corresponding cluster is smaller. Similarly, we will reward higher if two points in a small cluster have a must match link constraint. Thus, we divide each by the size of the cluster that the points are in.
Let be the similarity matrix and let be the matrix such that . Then, . By replacing in the trace minimization, the problem is equivalent to the minimization of . We calculate as , which is a constant and can be ignored in the optimization. This leads to a maximization of . If we define a matrix , our problem is expressed as a maximization of and is mathematically equivalent to unweighted kernel kmeans [6].
3.3.2 Alternative Optimization
If is fixed and is optimized. The objective can be written as:
(6) 
The objective can be written as:
(7) 
We obtain the following optimization problem:
maximize  (8)  
subject to  
where
is the affinity matrix in the location domain. This optimization problem can be solved by setting the kernel matrix
. Similarly, if is fixed and is optimized. Using the fact that we can rewrite the and , and obtain the following optimization problem:maximize  (9)  
subject to  
where is the affinity matrix in the face domain. This optimization problem can be solved by setting the kernel matrix .
4 Evaluations
We conduct experiments on two datasets to validate our approach. The first dataset contains images collected from personal albums with labeled ground truth. The second one uses a larger dataset crawled from online photo service: Flickr. We choose Kmeans with constraints [11] as the baseline algorithm. We also compare the performance of clustering on the each single domain by normalize cut [19] and Kmeans without any constraint. We use the RandIndex [18] to evaluate the performance of the clustering.
4.1 Personal Albums
This dataset contains images collected from personal albums. In total it has people and locations. In the location domain, the top image candidates are selected for the pairwise geometric verification. For each image, the bounding box of matched interest points is extracted as the location patch. A bag of words histogram ( visual words), bin color histogram are extracted from each location patch. The dimension size of features in the location domain is . We use a weight ratio of for BoW:color features. All feature vectors are normalized. In total, there are face patches and location patches.
In the dataset, each image associates a timestamp in the Exif header. The meanshift [8] is used to cluster images in the time sequence and a matrix is defined for each cluster of images. We cluster the face and location patches using the normalized cuts based on their appearance features as the baseline. Kmeans with constraints are also compared by adding the initial must match links and cannot match links in each domain. Figure 2 shows results in the people domain and the location domain.
From Figure 2, we observe the steady improvement on the clustering results when the number of clusters is larger than 2. The kmeans with constraints are quite sensitive to the number of clusters. The best RandIndex values of methods across all K values are ordered as: Coclustering, kmeanswithconstraints and Normalize cuts. The values for these methods except Coclustering do not vary much. The performance gain of Coclustering in the location domain is very significant. It’s mainly resulted from the must match link within the location domain. For the people domain, the difference in the clustering performance is very big, however, the steady increase over K is still promising.
4.2 Online Photo Sets
Dataset preparation: We use 140 names of public figures to query Flickr and filter out images without geolocation information. In total, we collect 53,800 images. We then filter out images without faces. The ground truth of the people domain is obtained directly from names. The ground truth of the location is obtained by clustering the longitude and latitude associated with images. We use the agglomerative clustering to discover location clusters. We consider each geolocation data including the longitude and the latitude as a point in the two dimensional space. In this dataset, we set the number of locations to be .
Figure 3 shows RandIndex values on the people domain and the locatin domain comparing kmeans, Normalized cuts and Coclustering over different K values. The improvement is not as big as that in the personal album dataset. It is mainly caused by the noise of the image set. The ground truth of the location domain is clustered by geolocation information which is not necessary equal to the location in the image. The ground truth of the people domain could also contain noise e.g. different people with the same name may appear together within one cluster. One future work is to find efficient algorithm to deal with the noise.
5 Conclusion
We present a novel algorithm to cocluster the people and location simultaneously. The relations across domains are used to enhance the clustering in single domain. We validate our approach using two datasets, and the experiment show that our algorithm performs better than clustering in the single domain and the baseline coclustering algorithm.
References
 [1] T. Berg, A. Berg, J. Edwards, M. Maire, R. White, E. LearnedMiller, and D. Forsyth. Names and faces in the news. CVPR, 2004.
 [2] Z. Cao, Q. Yin, X. Tang, and J. Sun. Face recognition with learningbased descriptor. CVPR, 2010.
 [3] C.Y. Chen and K. Grauman. Clues from the beaten path: Location estimation with bursty sequences of tourist photos. In CVPR, 2011.
 [4] D. Chen, G. Baatz, Köser, S. Tsai, R. Vedantham, T. Pylvanainen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk. Cityscale landmark identification on mobile devices. 2011.
 [5] M. Davis, M. Smith, J. Canny, N. Good, S. King, and R. Janakiraman. Towards contextaware face recognition. In ACM MM, 2005.

[6]
I. S. Dhillon, Y. Guan, and B. Kulis.
Kernel kmeans: spectral clustering and normalized cuts.
In KDD, 2004.  [7] M. Fischler and R. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981.

[8]
K. Fukunaga and L. Hostetler.
The estimation of the gradient of a density function, with applications in pattern recognition.
IEEE Transactions on Information Theory, 1975.  [9] J. Hays and A. Efros. IM2GPS: estimating geographic information from a single image. CVPR, 2008.
 [10] K. Heath, N. Gelfand, M. Ovsjanikov, M. Aanjaneya, and L. Guibas. Image webs: Computing and exploiting connectivity in image collections. In CVPR, 2010.
 [11] S. R. Kiri Wagstaff, Claire Cardie and S. Schroedl. Constrained kmeans clustering with background knowledge. In ICML, 2000.
 [12] B. Kulis, S. Basu, I. Dhillon, and R. Mooney. Semisupervised graph clustering: A kernel approach. In ICML, 2005.
 [13] X. Li, C. Wu, C. Zach, S. Lazebnik, and J.M. Frahm. Modeling and recognition of landmark image collections using iconic scene graphs. In ECCV, 2008.
 [14] D. Lin, A. Kapoor, G. Hua, and S. Baker. Joint people, event, and location recognition in personal photo collections using crossdomain context. In ECCV, 2010.
 [15] D. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV, 2004.
 [16] K. Mikolajczyk and C. Schmid. Scale & affine invariant interest point detectors. IJCV, 2004.
 [17] J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR, 2007.
 [18] W. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 1971.
 [19] J. Shi and J. Malik. Normalized cuts and image segmentation. TPAMI, 2000.
 [20] J. Sivic and A. Zisserman. Video Google: A text retrieval approach to object matching in videos. In ICCV, 2003.
 [21] M. Uřičář, V. Franc, and V. Hlaváč. Detector of facial landmarks learned by the structured output SVM. In VISAPP, 2012.
 [22] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In CVPR, 2001.
 [23] Q. Yin, X. Tang, and J. Sun. An associatepredict model for face recognition. In CVPR, 2011.
 [24] Y.T. Zheng, M. Zhao, Y. Song, H. Adam, U. Buddemeier, A. Bissacco, F. Brucher, T.S. Chua, and H. Neven. Tour the world: building a webscale landmark recognition engine. In CVPR, 2009.
Comments
There are no comments yet.