Deep Imbalanced Learning for Face Recognition and Attribute Prediction

Data for face analysis often exhibit highly-skewed class distribution, i.e., most data belong to a few majority classes, while the minority classes only contain a scarce amount of instances. To mitigate this issue, contemporary deep learning methods typically follow classic strategies such as class re-sampling or cost-sensitive training. In this paper, we conduct extensive and systematic experiments to validate the effectiveness of these classic schemes for representation learning on class-imbalanced data. We further demonstrate that more discriminative deep representation can be learned by enforcing a deep network to maintain inter-cluster margins both within and between classes. This tight constraint effectively reduces the class imbalance inherent in the local data neighborhood, thus carving much more balanced class boundaries locally. We show that it is easy to deploy angular margins between the cluster distributions on a hypersphere manifold. Such learned Cluster-based Large Margin Local Embedding (CLMLE), when combined with a simple k-nearest cluster algorithm, shows significant improvements in accuracy over existing methods on both face recognition and face attribute prediction tasks that exhibit imbalanced class distribution.


page 10

page 12


Attribute Adaptive Margin Softmax Loss using Privileged Information

We present a novel framework to exploit privileged information for recog...

Class Rectification Hard Mining for Imbalanced Deep Learning

Recognising detailed facial or clothing attributes in images of people i...

Imbalanced Deep Learning by Minority Class Incremental Rectification

Model learning from class imbalanced training data is a long-standing an...

Long-Tailed Recognition Using Class-Balanced Experts

Classic deep learning methods achieve impressive results in image recogn...

ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classification

Graph contrastive learning (GCL) has attracted a surge of attention due ...

Discriminative Sparse Neighbor Approximation for Imbalanced Learning

Data imbalance is common in many vision tasks where one or more classes ...

Striking the Right Balance with Uncertainty

Learning unbiased models on imbalanced datasets is a significant challen...

1 Introduction

Many data in face analysis domain naturally exhibit imbalance in their class distribution. For instance, the numbers of positive and negative face pairs in face verification [1, 2] are highly skewed since it is easier to obtain face images with different identities (negative) than faces with matched identity (positive) during data collection. For face attribute prediction [3], it is comparatively easy to find persons with “normal-sized nose” attribute from web images than that of “big-nose”. Such face recognition and attribute prediction problems provide perfect testbeds for studying generic imbalanced learning algorithms, either under closed- or open-set protocol [4]. Indeed, without handling the imbalance issue conventional methods tend to be biased toward the majority class with poor accuracy for the minority class [5, 6].

Deep representation learning has recently achieved great success due to its high learning capacity, but still cannot escape from the negative impact of imbalanced data. To counter such negative effect, one often chooses from a few available options, which have been extensively studied in the past [6, 7, 8, 9, 10, 5, 11, 12, 13, 14, 15, 16]. The first option is re-sampling, which aims to balance the class priors by under-sampling the majority class or over-sampling the minority class (or both). For instance, Oquab et al. [17] resampled the number of foreground and background image patches for learning a convolutional neural network (CNN) for object classification. The second option is cost-sensitive learning, which assigns higher misclassification costs to the minority class than to the majority. Caesar et al. [18]

proposed to calibrate an ensemble of SVMs with inverse class frequencies as costs to combat class imbalance in semantic segmentation. Similarly in deep CNNs, the loss function is rescaled with the inverse 

[19], relative [20] and median [21]

class frequencies, respectively for semantic segmentation, face attribute prediction and multi-task scene understanding. For image edge detection 

[22], the softmax loss of CNN is regularized with equal weights for the positive and negative edge classes. An alternative [23] goes beyond conventional cost-sensitive strategies by re-weighting the contributions of spatial image pixels based on their actual observed losses for semantic segmentation.

Can these methods help the deep face recognition and attribute prediction tasks where data imbalance is barely handled? Are these methods the most effective way to deal with data imbalance in the context of deep representation learning? The aforementioned options are well studied for the ‘shallow’ model [24] but their implications have not yet been systematically studied for deep representation learning. Importantly, such schemes are well-known for some inherent limitations. For instance, over-sampling can easily introduce undesirable noise with increased computational cost and overfitting risk. Under-sampling is often preferred [9]

but it may remove valuable information. Cost-sensitive learning approaches often design costs using heuristics or static class label statistics.

Such nuisance factors can be equally applicable to the recent deep imbalanced learning methods based on such common schemes. Methods in this line [25, 26, 27, 28] all fail to provide noticeable improvements in performance. Two advances [29, 30] excel by providing new insights. Wang et al. [29] proposed a meta-network to transfer knowledge (model parameters) from the majority class to minority class. Thus the model dynamics between many-shot and few-shot

models is learned, achieving superior classification results on existing imbalanced datasets like ImageNet 

[31]. Dong et al. [30] proposed a Class Rectification Loss (CRL) that can further handle imbalanced multi-label attributes and is more related to our goal. CRL performs hard mining for the minority attribute classes in each batch, and enforces their feature constraints to rectify the learning bias of the conventional cross entropy loss. However, regularization for only minority class cannot guarantee equal learning for all classes, and the learning quality and speed can be hindered by the hard mining online that sees one batch at a time, where a global characterization of feature space is lacking for correct regularization.

In this paper, we wish to investigate a more effective approach for deep imbalanced learning. We show its important applications to face recognition111Face recognition can be categorized as face identification (i.e.

classify one face to a specific identity) and face verification (

i.e., determine whether a pair of faces belong to the same identity). and attribute prediction from ubiquitously imbalanced datasets. Note such tasks can be evaluated under either closed- or open-set protocol. The open-set protocol is harder since the testing classes may be unseen from the training classes. It usually requires discriminative feature representations with built-in large margins, which are embodied in our approach.

Fig. 1: Example of class imbalance for the binary face attribute “wear hat”. Our method aims to separate the cluster distributions both within and between classes. This effectively reduces the class imbalance in local neighborhoods and forms balanced local class boundaries that are insensitive to the imbalanced size of remaining class samples.

Our method is motivated by the observation that the minority class often contains very few instances with high degree of visual variability. The scarcity and high variability make the genuine neighborhood of these instances easy to be invaded by other imposter nearest neighbors222An imposter neighbor of a data point is another data point with a different class label, .. Such invasion will confuse the underlying class boundaries formed by either a local or global classifier. To this end, we propose to learn an embedding with a CNN to ameliorate the invasion. The CNN is trained with a maintained index of clusters for each class, which we will update continuously throughout training. Our objective, then, would enforce margins between hard-mined clusters in the local neighborhood from both the same and different classes. Such margins introduce a tight constraint for reducing local data imbalance, leading to much more balanced class boundaries locally (Fig. 1). We demonstrate the margins can be deterministically derived on a hypersphere feature space. We also study the effectiveness of classic schemes of re-sampling and cost-sensitive learning in our context.

Using the learned feature representation, we show the evaluation can be simply done by a soft k-nearest-cluster metric which is consistent with our learning objective. The proposed approach, called Cluster-based Large Margin Local Embedding (CLMLE), achieves the new state-of-the-art performance on several face recognition datasets using only small training data. CLMLE also drastically outperforms the standard softmax and triplet losses and surpasses the recent imbalanced learning methods. For face attribute prediction, CLMLE achieves superior performance measured by the balanced accuracy across multiple attributes.

A preliminary version of this work has been published in [32]. This work extends the initial method LMLE [32] in significant ways. (1) Sampling of quintuplets (composed of 5 data points) in LMLE is generalized to the sampling of entire clusters. This alleviates the training inefficiencies of LMLE due to the exponential growth of quintuplet number. Also, penalizing the overlap between cluster distributions is much more coherent than penalizing individual quintuplets or triplets, leading to faster and better convergence than LMLE and triplet loss [33]. (2) A new CLMLE loss function is proposed, with customized cluster re-sampling and cost-sensitive learning techniques. (3) We design angular margins to be enforced between the involved cluster distributions. This is more natural than enforcing Euclidean distance on a hypersphere manifold as in LMLE. (4) The effectiveness of CLMLE is validated in the imbalanced tasks of not only face attribute prediction but also face recognition, where the additional open-set scenario demonstrates the superior generalization ability of CLMLE.

2 Related Work

Previous efforts to tackle the class imbalance problem can be mainly divided into two groups: data re-sampling [6, 7, 9, 10, 5, 11, 15] and cost-sensitive learning [8, 12, 13, 16, 14]. The former group aims to alter the training data distribution to learn equally good classifiers for all classes, usually by random under-sampling and over-sampling techniques. The latter group, instead of manipulating samples at the data level, operates at the algorithmic level by adjusting misclassification costs. A comprehensive literature survey can be found in [5, 6].

A well-known issue with replication-based random over-sampling is its tendency to overfit. More radically, it does not increase any information and fails in solving the fundamental “lack of data” problem for the minority class. To this end, SMOTE [7]

creates new non-replicated examples by interpolating neighboring minority class instances. Several variants of SMOTE 

[10, 11, 15] followed for improvements. However, their broadened decision regions are still error-prone by synthesizing borderline examples. Therefore under-sampling is often preferred to over-sampling [9], although potentially valuable information may be removed. Cost-sensitive methods avoid these issues by directly imposing heavier cost on misclassifying the minority class. Currently, how to determine the cost representation is still an open problem. Commonly-agreed practices include using the inverse class frequency or pre-defined misclassification costs, and they are typically applied to SVMs [12]

and decision trees 

[16]. Boosting [13] offers another natural way to embed the costs in example weights. Many other methods follow this philosophy of designing classifier ensemble (e.g.[14, 8]) to combat imbalance. In [14, 8]

, the authors combined cost sensitivity with bagging which is less vulnerable to noise than boosting, and generated cost-sensitive version of random forests.

Deep imbalanced learning. To our knowledge, only few works [25, 26, 27, 28, 34, 35, 29, 30] approach imbalanced learning via deep models. Jeatrakul et al. [25] treated the Complementary Neural Network as an under-sampling technique, and combined it with SMOTE-based over-sampling to re-balance data. Zhou and Liu [28] studied data resampling for training cost-sensitive neural networks. In [26, 27]

, the cost-sensitive deep features and the cost parameter are jointly optimized. All these works can be seen as direct “deep” extensions of traditional imbalanced learning techniques. Alternatives 

[34, 35] simply tune the networks to maximize a class-balanced accuracy measure. More recently, Ren et al. [36] proposed to reweight batch samples based on their gradient directions online, which can combat imbalance. However, a clean unbiased validation set is needed to represent the target distribution. Liu et al. [37] proposed to maximize the hyperspherical margin regardless of class imbalance, while Wang et al. [29] proposed a meta-learning approach that transfers model parameters from the majority to minority class. Both methods achieve good classification results on imbalanced datasets.

Unfortunately, none of the above works takes into account the data structure of imbalanced classes which helps learning. One exception is the Class Rectification Loss (CRL) in [30], where “hard” minority classes are searched in each batch and are regularized in feature space to rectify the learning bias of conventional loss, e.g., cross entropy. However, feature regularization for only minority class cannot guarantee equal learning for all classes. We propose here a “structure-aware” approach by enforcing large margins between intra-class and inter-class clusters. This way, balanced class boundaries can be equally drawn for every class from its involved local clusters. We show the clusters provide a global characterization of class distributions, leading to faster and better convergence than purely online methods like CRL [30] where the global information is missing. We also show our cluster separation rule can be easily applied to both training and testing, where the clustering process only incurs negligible computational cost during training.

Deep face recognition. Softmax loss has been pioneering effective CNN models for deep face recognition [1, 2], including recognition under the open-set protocol [4]. However, open-set recognition, unlike the closed-set one, cannot be addressed as a classification problem of known face identities as in softmax. Open-set scenario usually requires more discriminatively learned features with built-in margin. Recent L-Softmax loss [38], A-Softmax loss [39], and Large Margin Cosine Loss (LMCL) [40] generalize softmax by enforcing large angular margin between classes to enhance feature discrimination. Other works adopt ideas from metric learning, and combine softmax with contrastive [41, 42, 43], center loss [44] or marginal loss [45] for improvements. Another popular choice is the triplet loss [33, 46], which leads to state-of-the-art performance. The recent methods that handle class imbalance augment the minority classes in the image space [47] and feature space [48], respectively. Others align the feature centers [49] or weight norms [50] of the minority classes to the majority. Zhang et al. [51]

proposed a range loss to minimize the intra-class variance based on the largest intra-class distances (ranges) computed regardless of the imbalanced class size. Our method complements these methods by providing a data structure-aware loss function that enforces margins between local data to reduce the imbalance in any local neighborhood.

Deep face attribute prediction. Face attributes are useful as mid-level features for many applications like face verification [52, 3]. It is challenging to predict them from unconstrained face images due to the large facial variations. Most existing methods utilize part-based models to extract features from the localized part regions, and then train SVM classifiers to predict the presence of an array of face attributes, e.g., “male” and “smile”. For example, Kumar et al. [3] extracted HOG-like features from various local face regions for attribute prediction. Recent deep learning methods [53, 54, 55] excel by learning powerful features. Kalayeh et al. [55] further combined a deep semantic segmentation network to guide attribute prediction to the corresponding local region. These studies, however, share a common drawback: they neglect the class imbalance issue in those relatively rare attributes like “big nose” and “bald”. To our knowledge, only two works handle class imbalance in attribute prediction. The Mixed Objective Optimization Network (MOON) [20] re-weights attributes in a cost-sensitive manner, and the Class Rectification Loss (CRL) [30] performs online regularization for minority attribute classes in batch. Our method shows a stronger imbalanced learning ability with a new loss function.

3 Learning Deep Representation from Class-Imbalanced Data

Given an imagery dataset with imbalanced class distribution, our goal is to learn an embedding function from an image into a feature space , such that the embedded features are discriminative with local class imbalance ameliorated. We constrain this embedding to live on a -dimensional hypersphere, i.e.. Such normalization is commonplace in existing embedding methods (e.g., triplet embedding [33]), in order to achieve scale invariance under different image conditions, e.g., lighting, contrast and so on.

To achieve the above learning goal, we start by giving a brief review of the challenges with existing embedding methods that hinder their performance on class-imbalanced data. They will motivate our work to follow.

3.1 Challenges with Existing Embedding Methods

Triplet loss. The triplet loss [33] and contrastive loss [41] are two popular approaches to learn a Euclidean embedding function . Some triplet variants [56, 57] were recently proposed for improvements. For simplicity here, we only use the vanilla triplet loss to exemplify the inefficacy of this line of methods when handling class imbalance.

Triplet embedding is trained on a set of triplets where is the anchor point, associated with the positive and negative examples and . The sampling of triplets is usually semantic, informed by class labels: come from the same class, and come from different classes. Then the goal is to enforce semantic similarity between the feature embeddings extracted by one CNN with parameters (we will omit later for brevity). More precisely, the objective is to push away the negative example from the anchor in the embedding space by a Euclidean distance margin compared to the positive example :


where is the Euclidean distance.

The cost function is defined as:


where denotes the hinge function.

It is widely observed that the convergence rate and performance of triplet embeddings are hindered by two factors: 1) the cubic growth of the number of triplets, and 2) the penalty on individual triplets not being necessarily consistent. Hard negative mining [33] (semi-hard negative mining in the paper) is one of the ways to improve the triplet quality and hence the learning efficiency. However, hard mining still cannot conquer the limitation of the triplet loss in capturing the structure of imbalanced data. Specifically, the similarity information is only extracted at the class-level by triplets. This tends to equally collapse the intra-class variation and encourage unimodal discrimination. But it is easy to tell that the difficulty of collapsing different classes varies. Fig. 2 shows an imbalanced binary case. Obviously it is more difficult to collapse the majority class with larger variation than minority class. In case of collapsing failure, it would immediately lead to the invasion of imposter neighbors or even domination of the majority class in local neighborhood.

Fig. 2: The 2-D feature space of triplet loss [33], A-Softmax loss [39], Large Margin Local Embedding (LMLE) [32] and the proposed Cluster-based Large Margin Local Embedding (CLMLE). Class imbalance is exemplified in a binary-class case. The triplet and A-Softmax losses enforce Euclidean and angular margins respectively at class-level, assuming each class can be captured by a single mode. Such unimodal discrimination imposes too strong of a requirement, and may fail to collapse the majority class with larger variation and lead to class overlap. The LMLE enforces Euclidean margins among quintuple examples sampled from the intra- and inter-class local clusters. Such constraint preserves discrimination in local neighborhood and helps form local class boundaries that are insensitive to the imbalanced class size. The proposed CLMLE samples the entire cluster distributions instead to address the inefficiency and inconsistency issues with quintuplet sampling in LMLE. CLMLE also naturally facilitates the derivation of angular margins between cluster distributions on the unit hypersphere.

A-Softmax loss. The A-Softmax loss in SphereFace [39] enhances the discrimination power of Softmax by imposing the angular margin in a hypersphere space. However, it still suffers from the unimodal assumption of class distributions, which is insufficient on imbalanced data (see Fig. 2). In the traditional Softmax loss, the class score for sample can be written as where is the -th column of fully connected layer , and is the angle between and . In A-Softmax loss, each is normalized and the loss becomes:


where has the class label , and are the other labels in the training set . The function incorporates an angular margin for the class on unit hypersphere. Obviously, due to the enforcement of angular margins at class level, the intra-class structures are lost again and the issues with triplet loss on imbalanced data apply to the A-Softmax loss as well.

Large Margin Local Embedding (LMLE). Our previously proposed LMLE [32] addresses the class-imbalance issue by assuming that each class’s distribution can be represented as a variant number of clusters. Then we are able to draw balanced class boundaries only among the involved local clusters, not at the whole class level anymore. To this end, quintuplet instances (see Fig. 2), sampled from clusters both within and between classes, are used as hard examples to form implicit local boundaries:

  • : an anchor,

  • : the anchor’s most distant within-cluster neighbor,

  • : the nearest within-class neighbor of the anchor, but from a different cluster,

  • : the most distant within-class neighbor of the anchor,

  • : the nearest between-class neighbor of the anchor.

We wish to ensure that the following relationship holds in the embedding space:


Such a fine-grained relationship embraces the multi-modality of class distribution: it preserves not only locality across the same-class clusters but also discrimination between classes. As a result, it is capable of preserving discrimination in any local neighborhood, and forming local class boundaries with the most discriminative samples. Other irrelevant samples in a class are effectively “ignored” for class separation, making the local boundaries insensitive to imbalanced class sizes. To enforce the relationship in Eq. (3.1), a triple-header hinge loss is formulated to constrain three margins between the four Euclidean distances:


where are the slack variables, are the enforced Euclidean distance margins that can be explicitly determined on the hypersphere manifold.

Despite being effective on imbalanced data, quintuplets have similar sampling issues as triplets — the exponential growth of quintuplet number and the potential inconsistency between sampled quintuplets, both of which can hinder the convergence rate and quality. This work generalizes to sample the entire cluster distributions for learning. We will demonstrate the significantly improved learning efficiency and consistency, and hence better discrimination in the imbalanced context.

3.2 Cluster-based Large Margin Local Embedding

We propose to learn Cluster-based Large Margin Local Embedding (CLMLE) from all the examples in contextual clusters, rather than only hard examples (quintuplets) in clusters. Clustering techniques are employed to capture the local distributions of clusters for every class. Since our feature vectors

are always normalized onto the unit hypersphere (see Fig. 2), we choose the spherical k-means algorithm for clustering. Suppose we have the training set with sample and class label . Then for each class , we have cluster assignments:


where are the cluster centroids in class . We generate clusters with the same size to ensure an equal complexity to collapse these clusters during embedding learning, and to draw implicit balanced boundaries between them. The number of clusters is adaptively determined for each class that has size . Note here we use the inner product as the similarity metric, which is more suitable than Euclidean distance on the unit hypersphere. It also facilitates the derivation of our angular margins, as will be elaborated next.

During training, we are interested in a global characterization of data neighborhoods and penalizing any overlap of cluster distributions in a coherent way. Specifically, at each training iteration, we retrieve for a query cluster its nearest clusters from both the same and different classes in the neighborhood. We define as the centroid of cluster , and let denote the class label of . Then local discrimination can be achieved by making all the clusters inter-distinct and intra-compact, so that each cluster is able to represent a unique mode for future discrimination. Concretely, we maximize the intra-cluster similarity between and its containing cluster centroid , while minimizing the inter-cluster similarity between and other cluster centroids . We use the inner product again to characterize similarity and define our loss function as follows:


where is an angular margin on the unit circle, the desired angular gap between the cluster centroid and others.

The above loss function actually penalizes the probability assigned to the example

of a particular cluster under the distribution of another by a large margin . This will gradually collapse each cluster distribution into a small region and form safe margins between adjacent regions on the hypersphere. More importantly, those “marginal” clusters near the class bounds will implicitly draw a balanced classification boundary between them. As a result, the class imbalance is effectively reduced locally, ignoring the impact of other far-away clusters. To guarantee sufficient class separation between those marginal clusters, we inject the information of class label into Eq. (8), enforcing a larger angular margin for those between-class clusters than for same-class clusters:


where and are the inter-cluster angular margins enforced between different classes and within the same class, respectively. We will introduce how to explicitly derive and later.

Comparisons with related embedding methods. The new loss in Eq. (9) is similar in spirit to the probabilistic model of Nearest Class Mean (NCM) [58]. The main difference is that we adopt a non-uniform assumption for the class distribution, each modeled as a mixture of homogeneous clusters. This is well suited to class-imbalanced data and allows for local discrimination using only a balanced subset of clusters. While in NCM, the homogeneous class assumption is adopted, which is invalid in imbalanced settings. Another difference is that we learn deep feature representations rather than use fixed ones as in NCM.

When compared with the embedding methods based on individual triplets [33] or quintuplets [32], our method has two main merits: 1) Our sampling of entire clusters is linear to the size of training data , thus significantly avoids the sampling complexity of triplets and quintuplets and improves the training efficiency. 2) Our loss informed of entire cluster distributions has a sufficient insight of contextual neighborhoods. By separating all clusters at once, the training is more coherent and globally consistent than training with individual triple or quintuple examples. Fig. 3 demonstrates our CLMLE indeed converges much faster than the triplet loss [33] and quintuplet-based LMLE [32], while achieving better performance at the same time. In comparison to other state-of-the-art embedding methods (e.g., Center loss [44]), CLMLE strikes a better performance-speed trade-off.

The recent Class Rectification Loss (CRL) [30] performs regularization for the “hard” minority classes in every batch. By contrast, our CLMLE method performs equal learning for both minority and majority classes over batches. More importantly, the clustering step for all classes informs our learning process of the global feature distribution. It encourages faster and better learning convergence than the purely online method CRL, at negligible cost. On the CelebA dataset, for instance, CLMLE converges about 15 times faster than CRL.

Fig. 3: Speed and performance analyses in face recognition (on LFW [1]) and face attribute (on CelebA [53]) tasks. We compare convergence speed by the number of seen training data, which is fair and independent of differences in GPU, batch size used and other factors. We also report training times for the face attribute task. The compared methods are triplet loss [33], Center loss [44], A-Softmax [39], LMCL [40], Large Margin Local Embedding (LMLE) [32] and our Cluster-based Large Margin Local Embedding (CLMLE). CLMLE strikes a good performance-speed trade-off.

Angular margin derivation. One good characteristic of our loss function in Eq. (9) is that the margins and can be explicitly derived following a geometric intuition. These margins are designed angular, which translates well to the inner product based similarity metric on a unit hypersphere. Angular margins also share the favorable properties of scale and rotation invariance.

Fig. 4 illustrates how to set and properly. Obviously, their lower bounds are zero. Then we derive their theoretical upper bounds and in 2D space, to provide a reasonable parameter range for grid search during training. Recall that denotes the inter-cluster angular margin from different classes, and denotes the one within the same class. It is easy to first find the angular gap would peak when all clusters or even the whole classes are collapsed into single points on the hypersphere and they are all widely separated in between. In such extreme case, each unnormalized cluster centroid is identical to the cluster members thus has unit norm too. The inner product in Eq. (9) now depends on the angle only, while the intra-cluster similarity is fixed as . Hence we can define the maximum angular margin in the form of . Easily, we have when each class becomes a single point with the maximum inter-class angle . when the considered clusters are most far apart in their belonging class that occupies a proportion of the hyperspherical space.

Fig. 4: Extreme 2D cases for deriving the upper bounds of inter-cluster angular margins between class () and within class ().

Cluster updates. It is worth noting that the cluster assignment for each class in Eq. (6) is initialized using the features from a pre-trained CNN. Since the feature representations are updated continuously during training, we should gradually update the clusters as well on the newly learned features to reflect their true distribution. To this end, for each class, we maintain a running index of clusters and refresh it after a fixed number of iterations using the latest features. The computational cost of the clustering algorithm of spherical k-means is negligible compared to the cost of learning CNN features.

Fig. 5: The 2-D feature space using t-SNE [59] and pairwise feature similarity for one binary face attribute from the CelebA dataset [53]. We only show 2 Positive Clusters (PC) and 5 Negative Clusters (NC) to represent the class imbalance. The embedding of a pre-trained model, our CLMLE, and triplet embedding are compared. We can see that between-class clusters (with different colors) are well separated in CLMLE, but they are overlapped in triplet embedding, leading to overlapping binary score distributions.

Embedding visualization. Fig. 5 visualizes the 2-D space of the initial embedding and final converged CLMLE in an imbalanced face attribute example. Triplet embedding [33] as a representative unimodal learning method, is included for comparison. Since triplets operate at the class-level and homogeneously collapse each class, we see their inability to capture the fine-grained variations (represented by clusters) within the imbalanced classes, which leads to a large overlap between the minority (positive) and majority (negative) classes. Such issue equally applies to the other unimodal methods of the contrastive loss [41], advanced triplet loss [56], and angular losses like A-Softmax [39] and L-Softmax [38]. By contrast, our CLMLE explores the multimodal property of class by enforcing angular margins both within and between classes. As demonstrated in the figure, this helps to learn unique clusters of balanced size in each class, which are able to draw balanced local class boundaries for discrimination. We will show this can also boost the feature generalization in open-set scenarios (for face verification) that often require discriminative features with built-in margins. It is worth noting that Wang et al. [60] aim to learn fine-grained similarity within class as well, but they do not explicitly rely on clustering techniques to model the within-class variations.

3.3 Overall Training Procedure with Re-Sampling and Cost-Sensitivity

During training, we construct one mini-batch with a query cluster and its retrieved nearest clusters by computing . We greedily make sure the retrieved clusters would come from both the same and different classes. Since the cluster size is not small, the batch size will be very large even if we only retrieve a small number of clusters . This is not viable for training due to memory constraints. On the other hand, we observed empirically that using only a few clusters (i.e., low cluster diversity) hurts performance. Thus we choose to sample a small portion of data in each considered cluster to increase the cluster number while maintaining a reasonable batch size. In practice, we randomly sample 20 examples out of from each of clusters. The batch size is 240. The cluster centroid in Eq. (9) is then approximated as . Such cluster data sampling is simply repeated during CNN training. It avoids large information loss as in traditional random under-sampling techniques. When compared with over-sampling, it introduces no artificial noise.

So far, one important problem is still left unaddressed: how to sample the query cluster in mini-batch. We found a random sampling strategy is not ideal for performance. Here we simply follow the common practices of re-sampling and cost-sensitive techniques as detailed below. Section 5 will quantify their efficacy systematically.

Re-sampling of query cluster . To ensure adequate learning for all classes, we sample evenly from both majority and minority classes. To determine the exact query cluster to use from the chosen class, we borrow the idea of  [23] to pick as the one with the highest observed loss in class. This allows us to adapt to the current feature distribution and focus on the hardest cluster that has large overlap with neighboring ones. In practice, the loss of a cluster is computed by averaging the losses of cluster members that are cached online.

Cost-sensitive learning in batch. Note the query and retrieved clusters are most likely to be class-imbalanced in one mini-batch. We follow the commonly used cost-sensitive approach [18, 19] to scale the loss of each sample by the inverse class frequency in mini-batch. For multi-way classification (e.g., in face identification), this effectively gives those less frequent classes more importance. For the problem of predicting multiple attribute labels from a face, cost-sensitivity helps even more by being able to re-balance all labels simultaneously. Note re-sampling multi-label data is structurally infeasible because sampling to balance one label will affect the sampling of others.

Overall training procedure for CLMLE:

  1. Cluster for each class by spherical k

    -means using the latest features. Initially, we use the features extracted by the pre-trained CNN.

  2. For CNN training, repeatedly construct mini-batches with one query cluster (with highest loss) from a random class, and the nearest clusters retrieved from both the same and different classes.

  3. Randomly sample 20 examples for each of the clusters in batch, and compute their cluster centroids.

  4. Compute the loss in Eq. (9) with cost-sensitivities (by inverse class frequency). Back-propagate the gradients to update the CNN parameters and feature embeddings.

  5. Alternate between step 1 and 2-4 periodically until convergence (often within 5 alternation rounds).

4 Fast Evaluation with Nearest Clusters

Our learned CLMLE offers crucial feature representations for accurate evaluation on imbalanced data. We choose a simple k-nearest cluster algorithm which is consistent with our training objective. The nearest neighbor rule is appealing due to its non-parametric nature, and it is easy to extend to new classes without retraining.

Specifically, for a query , we retrieve its nearest clusters from all the training classes by computing , and decide its class label by:


where the label is predicted as the class whose least similar cluster is more similar than the clusters from other classes by the largest margin.

Such testing procedure makes the nearest neighbor search a function of the cluster number rather than of the example number . We further speed up the cluster-wise search using the KD-tree [61] whose runtime is logarithmic in the cluster number () with a complexity of . This leads to up to three orders of magnitude speedup over standard example-wise search in practice, making it easy to scale to large datasets. It is worth mentioning that we use such a simple testing algorithm to show the efficacy of our learned features. Better performance is expected with the use of more elaborated algorithms.

5 Experiments

Fig. 6: Imbalanced data distribution for face recognition and face attribute prediction. (a) Long-tailed distribution of image number per class on CASIA-WebFace dataset [43]. (b) 40 binary face attributes on CelebA dataset [53], each with imbalanced positive and negative samples.

We study the face recognition and face attribute prediction tasks, both with large-scale imbalanced datasets (see Fig. 6). The face recognition task is cast as either an identification (multi-way classification of a face) or verification (binary classification of a face pair) problem, evaluated in the open-set scenario. As shown in the figure, the available training dataset of CASIA-WebFace [43] is highly imbalanced. Some classes of the 10k subjects have hundreds of images, while about 39% of them have no more than 20 images. The average number of images per class is 42.8. Such data imbalance makes it difficult for learning equally good class representations which are essential for comparing paired images from different classes. The face attribute prediction task is cast as a (closed-set) multi-task classification problem. We aim to predict 40 binary attributes simultaneously, each with imbalanced positive and negative samples (e.g., for “Bald” attribute: 2% vs. 98%). In this case, the variant imbalance level in multi-label data makes the problem even harder.

Parameters. Our CNN is trained using Caffe with fixed momentum 0.9, weight decay . The learning rate starts with 0.1 and 0.001 for face recognition and attribute prediction respectively, and is divided by 10 when the performance plateaus. We have cluster size and clusters in one batch. The down-sampled clusters leads to the batch size 240. For testing, the optimal number of retrieved clusters in Eq. (10) is searched from on validation set. The task-specific network architecture and prior features for initial clustering are summarized in Table I. Note the prior features are not critical to the final results because we will gradually update the deep features in alternation with the clustering process. The alternation happens every 2k - 5k iterations, depending on the task-specific convergence rate. Usually different prior features converge to similar results. For our face verification (on LFW [1]) and attribute prediction experiments, different prior features lead to accuracy difference of no more than 0.02% (pre-trained by softmax vs. triplet) and 0.3% (pre-trained by face recognition vs. multi-attribute softmax classification) respectively.

Computational cost. For face recognition and face attribute prediction, it takes about 1.5 and 3 days respectively to train CLMLE on GPU (NVIDIA Tesla K40), both converged within 5 alternation rounds. The clustering process in each round incurs negligible cost compared to feature learning. CLMLE has about 1.3 - 2 times faster convergence than its earlier version LMLE [32] (see Fig. 3

) thanks to the avoidance of quintuplet-based sampling and learning. For testing, it takes 10ms to extract features, and the cluster-wise kNN search is typically 1000 times faster than standard kNN. This enables real-time application to large-scale problems with hundreds of thousands to millions of samples.

Task Network Prior features
Face recognition Same w.r.t. [39] Pre-trained by softmax
Face attributes Same w.r.t. [41] DeepID2 features in [41]
TABLE I: The CNN architecture and prior features for initial clustering in our considered imbalanced tasks.

5.1 Experimental Settings

Face recognition. All faces and landmarks are detected by MTCNN [62] for the training and testing images. The detected landmarks (two eyes, nose and mouth corners) are used for similarity transformation, and the faces are cropped to pixels. Each pixel (in ) in cropped images is normalized by subtracting 127.5 and then dividing by 128.

For training, we use the same CNN architecture as in [39, 40] with 64 convolutional layers based on residual units. This enables fair comparison with the recent strong methods and their variants. For the same reason, we use a small training set, the publicly available CASIA-WebFace dataset [43] which contains 0.45M face images from 10,575 identities. The training images with identities appearing in our test sets are already removed. Note the scale of our training data is much smaller than that of other private datasets used in DeepFace [4] (4M), VGGFace [63] (2M) and FaceNet [33] (200M). In practice, we apply data augmentation by flipping the training images horizontally.

For testing, we extract the deep features from the output of the FC1 layer. The features of the original image and flipped image are concatenated to obtain the final face representation. The inner product between normalized features is then calculated as the similarity score. In the closed-set scenario with fixed class set, face recognition can be done by using the k-nearest cluster rule in Eq. (10); while in open-set scenarios with new classes, identification and verification are respectively conducted by ranking and thresholding the similarity scores following the convention, which will be shown to still benefit from our learned CLMLE.

We test on three popular large-scale datasets: LFW [1], YTF [64] and MegaFace [2]. LFW dataset contains 13,233 web images from 5,749 face identities captured in unconstrained conditions. YTF dataset contains 3,425 videos from 1,595 identities. Each video varies from 48 to 6,070 frames, with an average length as 181.3 frames. Both datasets exhibit large facial variations in pose, expression and lighting. We follow the standard protocol of unrestricted with labeled outside data [1] for both datasets, and test on 6k face pairs from LFW and 5k video pairs from YTF. MegaFace dataset is a very challenging benchmark for face recognition at the million scale of distractors. The gallery set in MegaFace contains more than 1M images from 690k identities, while the probe set consists of two existing datasets — Facescrub and FGNET. We choose the larger Facescrub dataset as our probe set which contains 106,863 face images from 530 celebrities. We report the face identification and verification results under two protocols (small or large training set).

For evaluation, we not only use the simple metrics of the identification or verification accuracy, but also the True Accept Rate (TAR) at fixed False Accept Rate (FAR) as well as ROC curves that can take into account any imbalance in testing pairs.

Face attributes. We use the CelebA dataset [53] that contains 202,599 images from 10,177 identities, each with about 20 images. Every face image is annotated with 40 attributes and 5 key points to align the image to pixels. We partition the dataset following [53]: the first 162,770 images (i.e., 8k identities) for training (10k images for validation), the following 19,867 images for training SVM classifiers for the PANDA [54] and ANet [53] methods, and the remaining 19,962 images for testing. The identities are non-overlapping in these splits. During training, horizontal flipping is applied for data augmentation. We use the CNN architecture from [41] as in ANet [53], LMLE [32] and CRL [30]. One extra 64-d FC layer is learned for each binary attribute via Eq. (9) in a multi-task manner.

For testing, we extract the deep features for each attribute from its FC layer, and classify attributes via Eq. (10) under the closed-set classification scenario. To account for the imbalanced positive and negative samples for each attribute, we adopt a balanced accuracy metric , where and are the numbers of positive and negative samples, while and are the numbers of true positives and true negatives. Note this metric differs from the one employed in [53]i.e. which can be biased to the majority class.

5.2 Face Recognition

Method #Nets Train data LFW YTF
DeepFace [4] 3 4M 97.35 91.4
FaceNet [33] 1 200M 99.63 95.1
Web-scale [65] 4 4.5M 98.37 -
VGG Face [63] 1 2.6M 98.95 97.3
DeepID2+ [42] 25 0.3M 99.47 93.2
Baidu [46] 1 1.3M 99.13 -
Center Face [44] 1 0.7M 99.28 94.9
Marginal loss [45] 1 4M 99.48 95.98
Noisy Softmax [66] 1 WebFace+ 99.18 94.88
CoCo loss [67] 1 2M 99.86 -
Range loss [51] 1 1.5M 99.52 93.7
Augmentation [47] 1 WebFace 98.06 -
Center invariant loss [49] 1 WebFace 99.12 93.88
Feature transfer [48] 1 4.8M 99.37 -
Softmax loss 1 WebFace 97.88 93.1
Softmax+Contrastive [41] 1 WebFace 98.78 93.5
Triplet loss [33] 1 WebFace 98.70 93.4
L-Softmax loss [38] 1 WebFace 99.10 94.0
Softmax+Center loss [44] 1 WebFace 99.05 94.4
SphereFace (A-Softmax) [39] 1 WebFace 99.42 95.0
CosFace (LMCL) [40] 1 WebFace 99.33 96.1
LMLE [32] 1 WebFace 99.51 95.8
CLMLE 1 WebFace 99.62 96.5
TABLE II: Face verification accuracy (%) on LFW [1] and YTF [64] datasets. Most methods in the first cell use large-scale outside data that are not publicly available (+ denotes data expansion). The second cell includes recent imbalanced learning methods. The state-of-the-art loss functions in the second-to-last cell and ours in the last cell use the small training data (CASIA-WebFace [43]: 0.45M) and the same 64-layer CNN model for fair comparison.

Experiments on LFW and YTF. Table II lists the face verification accuracy on the two datasets. Existing state-of-the-art face verification systems (in the first cell) either use large training data or model ensemble. For example, both the top performing CoCo loss [67] on LFW (99.86%) and VGG Face [63] on YTF (97.3%) use more than 2M training images (the training set is regarded as small only if it contains no more than 0.5M images). While the proposed method only uses the small, publicly available training data (CASIA-WebFace [43] with 0.45M images) and a single model. Our CLMLE achieves the best performance in this setting - 99.62% on LFW and 96.5% on YTF. By taking into account the class imbalance during training, our single CLMLE model even outperforms or performs closely to the FaceNet [33] which uses around 200M outside training data, and DeepID2+ [42] which ensembles 25 models.

For a fair comparison with recent loss functions, we use the same WebFace training data and the same 64-layer CNN architecture. The compared loss functions are trained with their default hyper-parameters. One can observe that our CLMLE considerably outperforms the Softmax variants (including L-Softmax [38], A-Softmax [39], and LMCL [40]) and metric learning methods (including contrastive [41], triplet [33] and center loss [44]). They all suffer from the unimodal assumption of class distribution which is not suitable in class-imbalanced scenarios.

When compared with the recent imbalanced learning methods trained on WebFace or larger data (second cell in Table II), CLMLE is shown to achieve consistent gains. The methods of minority class augmentation in image space [47] and feature space [48] obtain sub-optimal results, while the range loss [51] and center invariant loss [49] are not as discriminative as our CLMLE. Our approach demonstrates superior feature discrimination by enforcing large margins between local clusters from imbalanced classes. Moreover, CLMLE improves over its previous version LMLE [32] by discriminating entire cluster distributions rather than individual quintuplets sampled from them. Our large margin feature learning method is particularly suitable for generalization test in the open-set face recognition problem, as verified by the above experiments.

Method Protocol Rank 1 Veri.
Beijing FaceAll_Norm_1600 Large 64.80 67.11
Google - FaceNet v8 Large 70.49 86.47
NTechLAB - facenx_large Large 73.30 85.08
SIATMMLAB TencentVision Large 74.20 87.27
DeepSense V2 Large 81.29 95.99
YouTu Lab Large 83.29 91.34
Vocord - deepVo V3 Large 91.76 94.96
SIAT_MMLAB Small 65.23 76.72
DeepSense - Small Small 70.98 82.85
SphereFace - Small Small 75.76 90.04
Beijing FaceAll V2 Small 76.66 77.60
GRCCV Small 77.67 74.88
FUDAN-CS_SDS Small 77.98 79.19
CoCo loss [67] Small 76.57 -
Softmax loss Small 54.85 65.92
Softmax+Contrastive [41] Small 65.21 78.86
Triplet loss [33] Small 64.79 78.32
L-Softmax loss [38] Small 67.12 80.42
Softmax+Center loss [44] Small 65.49 80.14
SphereFace (A-Softmax) [39] Small 72.72 85.56
CosFace (LMCL) [40] Small 77.11 89.88
Range loss [51] Small 72.94 83.62
LMLE [32] Small 78.53 89.45
CLMLE Small 79.68 91.85
TABLE III: Face recognition on MegaFace Challenge 1 [2] under the protocols of small and large training set. “Rank 1” refers to rank-1 face identification accuracy (%) with 1M distractors, and “Veri.” refers to face verification TAR (%) under FAR. The state-of-the-art loss functions in the second-to-last cell and imbalanced learning loss functions in the last cell use the small training data (CASIA-WebFace [43]: 0.45M) and the same 64-layer CNN model for fair comparison.

Experiments on MegaFace Challenge 1. The feature generalization can be further tested in the MegaFace Challenge 1 [2] where the gallery set contains more than 1M distractors. For the face identification task which aims to match one probe image to the images with the same person in the gallery, the large MegaFace gallery set poses a big challenge and naturally causes data imbalance. For face verification, we should decide if an image pair contains the same person or not. There are 4 billion negative pairs generated between the probe and gallery sets, which are imbalanced with respect to the available positive pairs.

Table III first reports the face identification and verification results in terms of the simple rank-1 identification accuracy and verification True Accept Rate (TAR) at fixed False Accept Rate (FAR). Our training set CASIA-WebFace [43] has 0.45M images, thus we follow the small training set protocol. Under this protocol, our CLMLE performs best for both the identification and verification tasks. For some models trained under the large training set protocol, e.g., Google-FaceNet v8 with 500M images and NTechLAB-facenx_large with 18M images, they are also beaten by our CLMLE with small training data. In comparison to the recent models trained with the same small data and network architecture (in second-to-last cell), our CLMLE shows better results and superior generalization ability again. We further implemented the range loss [51] (a reportedly competitive imbalanced learning method) and LMLE [32] in the same settings. For the range loss, we formed a weighted combination of it and softmax loss, with weights suggested by the authors. It is evident from the table that CLMLE outperforms in the imbalance handling ability.

Fig. 7: CMC and ROC curves of top performing methods with 1M distractors on MegaFace Challenge 1 [2]. Note the FaceNet v8 follows the large training set protocol, while other methods follow the small one.
Fig. 8: Challenging pairs (green: positive pair; red: negative pair) that are correctly recognized by our method.
Method Protocol Rank 1 Veri.
3DiVi Large 57.04 66.45
NEC Large 62.12 66.84
GRCCV Large 75.77 74.84
SphereFace (A-Softmax) [39] Large 71.17 84.22
CosFace (LMCL) [40] Large 74.11 86.77
Range loss [51] Large 69.54 82.67
LMLE [32] Large 74.76 87.78
CLMLE Large 76.26 89.41
TABLE IV: Face recognition on MegaFace Challenge 2 [68] under the large training set protocol. “Rank 1” refers to rank-1 face identification accuracy (%) with 1M distractors, and “Veri.” refers to face verification TAR (%) under FAR. The SphereFace [39], CosFace [40] and imbalanced learning methods in the last cell use the same 64-layer CNN model for fair comparison.

Fig. 7 shows the Cumulative Match Characteristics (CMC) curves (for face identification) as well as the Receiver Operating Characteristic (ROC) curves (for verification) that can take into account any imbalance in testing pairs. Note the MegaFace gallery set contains different scales of distractors, from 10 to 1M with increasing difficulty. The figure shows the curves at 1M scale for the top performing methods under small and large protocols. We can see that our CLMLE method achieves the new state-of-the-art performance (see Fig. 8 for some visual results).

Experiments on MegaFace Challenge 2. For MegaFace Challenge 2 [68], all the algorithms use the same training data provided by MegaFace and follow the large training set protocol. The training set includes 4.7M images from 672k identities. The gallery set contains 1M images that are different from those in Challenge 1. Table IV illustrates our CLMLE attains the top performance for both face identification (76.26% rank-1 accuracy) and face verification (89.41% TAR under FAR). This validates the importance of tackling imbalance for feature representation learning, and our CLMLE outperforms its previous version LMLE and the range loss [51] for imbalanced learning. Moreover, our large margin nature is shown to significantly improve the open-set recognition performance. By contrast, the range loss as a regularization method over softmax, is more suitable to closed-set classification problems.

5.3 Face Attribute Prediction

Table V compares our CLMLE method for face attribute prediction with the state-of-the-art methods of Triplet-kNN [33], PANDA [54] and ANet [53] which are trained on the same data and tuned to their best performance. The attributes and their mean per-class accuracies are given in the ascending order of class imbalance level (positive class rate-50%). This is to highlight the impact of class imbalance on performance. Note the CelebA dataset [53] also poses challenges to joint attribute prediction, where the variant imbalance levels of different attributes need to be handled simultaneously.


Mouth Open


Wear Lipstick

High Cheekbones


Heavy Makeup

Wavy Hair

Oval Face

Pointy Nose

Arched Eyebrows

Black Hair

Big Lips

Big Nose


Straight Hair

Brown Hair

Bags Under Eyes

Wear Earrings

No Beard


Imbalance level 1 2 2 3 5 8 11 18 22 22 23 26 26 27 28 29 30 30 31 33 35
Triplet-kNN [33] 83 92 92 91 86 91 88 77 61 61 73 82 55 68 75 63 76 63 69 82 81
PANDA [54] 85 93 98 97 89 99 95 78 66 67 77 84 56 72 78 66 85 67 77 87 92
ANet [53] 87 96 97 95 89 99 96 81 67 69 76 90 57 78 84 69 83 70 83 93 90
Down-sampling [9] 78 87 90 91 80 90 89 70 58 63 70 80 61 76 80 61 76 71 70 88 88
MOON [20] 82 94 93 94 87 98 91 79 65 66 78 86 59 72 79 68 81 70 80 88 92
Over-sampling [9] 77 89 90 92 84 95 87 70 63 67 79 84 61 73 75 66 82 73 76 88 90
Cost-sensitive [5] 78 89 90 91 85 93 89 75 64 65 78 85 61 74 75 67 84 74 76 88 90
AFFACT [69] 83 94 93 94 88 98 91 84 66 66 80 87 62 76 82 75 83 76 86 94 92
CRL-I [30] 83 95 93 94 89 96 84 79 66 73 80 90 68 80 84 73 86 80 83 94 95
MOON-D [20] 82 94 93 94 87 98 91 81 69 71 82 88 66 77 84 75 85 81 86 94 94
AFFACT-D [69] 83 94 93 94 88 98 92 86 72 72 84 89 69 79 86 82 86 83 89 95 95
LMLE [32] 88 96 99 99 92 99 98 83 68 72 79 92 60 80 87 73 87 73 83 96 98
CLMLE 90 97 99 98 94 99 98 87 72 78 86 95 66 85 90 80 89 82 86 98 99

Blond Hair

Bushy Eyebrows

Wear Necklace

Narrow Eyes

5 o’clock Shadow

Receding Hairline

Wear Necktie


Rosy Cheeks





Wear Hat

Double Chin

Pale Skin

Gray Hair




Imbalance level 35 36 38 38 39 42 43 44 44 44 44 44 45 45 45 46 46 46 48
Triplet-kNN [33] 81 68 50 47 66 60 73 82 64 73 64 71 43 84 60 63 72 57 75 71.55
PANDA [54] 91 74 51 51 76 67 85 88 68 84 65 81 50 90 64 69 79 63 74 76.95
ANet [53] 90 82 59 57 81 70 79 95 76 86 70 79 56 90 68 77 85 61 73 79.58
Down-sampling [9] 85 75 66 61 82 79 80 85 82 85 78 80 68 90 80 78 88 60 79 77.45
MOON [20] 91 79 57 58 78 69 82 97 70 84 66 82 71 91 66 69 81 68 83 78.59
Over-sampling [9] 90 80 71 65 85 82 79 91 90 89 83 90 76 89 84 82 90 90 92 81.48
Cost-sensitive [5] 89 79 71 65 84 81 82 91 92 86 82 90 76 90 84 80 90 88 93 81.60
AFFACT [69] 91 80 70 64 86 76 90 99 79 90 76 91 74 95 74 75 87 74 89 82.69
CRL-I [30] 95 84 74 72 90 87 88 96 88 96 87 92 85 98 89 92 95 94 97 86.60
MOON-D [20] 94 86 76 75 90 88 93 99 89 94 87 95 86 96 88 91 94 93 96 87.02
AFFACT-D [69] 94 87 81 79 92 88 94 99 91 96 89 95 91 98 90 92 95 95 97 88.84
LMLE [32] 99 82 59 59 82 76 90 98 78 95 79 88 59 99 74 80 91 73 90 83.83
CLMLE 99 88 69 71 91 82 96 99 86 98 85 94 72 99 87 94 96 82 95 88.78
TABLE V: Mean per-class accuracy (%) and class imbalance level (positive class rate-50%) of each of the 40 binary attributes on CelebA dataset [53]. Attributes are sorted by the imbalance level in an ascending order. The second and third cells list state-of-the-art methods and imbalanced learning methods, respectively. Note the results of balanced accuracy are different from the classification accuracy reported by ANet [53]. Also, MOON (-D) and AFFACT (-D) use networks of VGG-16 and ResNet-50, much larger than ours as described in [41]

. The suffix ’D’ denotes domain adaptation to the balanced attribute distribution, the underlying distribution suggested by our balanced evaluation metric.

Fig. 9: Relative gains over the state-of-the-art methods without imbalance handling mechanism across the 40 imbalanced attributes on CelebA [53].

It is shown that CLMLE outperforms these methods across all attributes, with an average gap of about 9% over ANet. Considering most face attributes exhibit high class imbalance with an average positive class rate of only 23%, such improvements are nontrivial and prove the efficacy of our learned features on imbalanced data. Although PANDA and ANet are capable of learning robust feature representations by ensemble and multi-task learning respectively, they ignore the imbalance issue and thus struggle for highly imbalanced attributes, e.g., “Bald”. By contrast, our method performs well on such attributes. Our advantage over these non-imbalanced learning methods is more evident when observing the relative accuracy gains for different attributes. Fig. 9 shows the gains tend to be larger for those attributes with a higher class imbalance level.

The bottom cells of Table V illustrate the results of imbalanced learning methods. We can see that our CLMLE outperforms traditional re-sampling and cost-sensitive learning techniques, as well as the previous LMLE method [32] and recent CRL-I [30], all of which use the same network architecture [41] for fair comparison. CRL-I denotes regularizing the minority class with Instance level hard mining. However it still cannot guarantee equal learning for all classes and has inferior results. Thanks to the authors of MOON [20] (with adaptive re-weighting) and AFFACT [69], we further compare against their provided results of balanced accuracy from using VGG-16 and ResNet-50 networks respectively. Our CLMLE still outperforms the two methods by large margin (about 10% and 6% on average), even using a much smaller network [41]. On top of larger networks, the MOON-D and AFFACT-D variants are further fine-tuned to completely balanced attribute distribution, the same as the underlying distribution suggested by our balanced evaluation metric. While CLMLE, without assuming the target attribute distribution or using large networks, is able to achieve comparable or even better performance.

Table VI compares CLMLE with state-of-the-art AttCNN [70] and SSP [55] in terms of two evaluation metrics. AttCNN addresses the multi-label imbalance problem by selective data learning. However, only average classification accuracy is reported, and our superiority still holds for this metric. The Semantic Segmentation-based Pooling (SSP) method does not handle imbalance, but adds a segmentation network to improve the attribute network. Our CLMLE is quite comparable in both metrics without increasing network capacity. Fig. 10 shows some challenging images with highly imbalanced attributes that are correctly predicted by CLMLE but not by SSP.

Method AttCNN [70] SSP [55] CLMLE
Average balanced acc. - 88.24 88.78
Average classification acc. 90.97 91.67 91.13
TABLE VI: Average balanced accuracy and classification accuracy (%) for the 40 binary attributes on CelebA dataset [53].
Fig. 10: Most imbalanced face attributes (from Table V) that are correctly predicted by our method.
Method LFW CelebA
CLMLE 99.62 88.78
uniform cluster sampling 99.52 88.23
no cost-sensitive learning 99.57 87.46
TABLE VII: Ablation study in terms of the face verification accuracy (%) on LFW and average balanced accuracy (%) for attribute prediction on CelebA.
Triplet+ 42.2 32.2 65.2 59.8
CE+CRL [30] 45.2 37.4 71.2 64.5
LMLE [32] 44.3 38.1 71.9 65.7
CLMLE 47.4 39.7 73.5 68.8
Balanced baseline 48.2 78.3
TABLE VIII: Mean class-balanced accuracy (%) on CIFAR-100 [71] and MNIST-rot-back-image [72] datasets with simulated class imbalance (with parameter - the smaller, the more imbalanced). Triplet+ denotes triplet loss [33] with over-sampling and cost sensitivity. Balanced baseline denotes the CE+CRL results under class-balanced setting.

5.4 Ablation Study

Table VII quantifies the benefits of the re-sampling and cost-sensitive components adopted in our CLMLE method. Both the classic schemes prove themselves as useful for our imbalanced tasks. The sampling of query cluster is designed to be non-uniform in class - we pick the one with the highest observed loss. Results show that it hurts performance if we choose uniform cluster sampling instead. It is more so for face recognition, because compared to the binary attribute prediction problem, face recognition has far more classes whose discrimination is more sensitive to the hard modes (clusters). The cost-sensitive scheme is used by us to re-balance the classes in batch at score level. Note when this scheme is applied to perfectly class-balanced batches, it would have no effects. However in the case of multi-label attributes, class-balanced batch samples for one attribute will almost certainly be imbalanced for the other attributes. This is when the class costs can help. Our ablation study confirms that the cost-sensitivity has a larger impact on CelebA attributes. To highlight the effects of our k-nearest cluster classifier in Eq. (10), we replace it with the regular instance-wise kNN classifier. As expected, we observe much lower speed and a bit worse results (88.59%) for attributes.

5.5 Generic Imbalanced Image Classification

In addition to the face recognition and attribute prediction tasks with real-world imbalanced data, we have also tested CLMLE on generic image classification problems (single-label & multi-class). We experiment on standard benchmarks CIFAR-100 [71] and MNIST-rot-back-image [72] but with simulated class imbalance. As in [30], the simulated class-imbalanced data follow a power-law distribution where and are the largest and smallest sizes of classes . The parameter controls the degree of class imbalance — the smaller is, the more imbalanced the distribution is. We follow the same experimental settings of [30] and [32] for CIFAR-100 and MNIST respectively, including data usage, network architecture and hyper-parameters. Table VIII reports classification results under two imbalnaced ratios , and compares CLMLE with some imbalanced learning baselines Triplet+, CE (Cross-Entropy)+CRL [30] and LMLE [32]. Classification results under the class-balanced scenario are reported using CE+CRL as a balanced baseline. We can see that CLMLE’s superiority still holds for imbalanced image classification.

6 Conclusion

Class imbalance is common in many vision tasks, including face recognition and attribute prediction. Contemporary deep representation learning methods typically adopt class re-sampling or cost-sensitive learning schemes. Through extensive experiments, we have validated their usefulness and further demonstrated that the proposed Cluster-based Large Margin Local Embedding (CLMLE) works remarkably well when just combined with a simple k-nearest cluster classifier. CLMLE maintains inter-cluster angular margins both within and between classes, thus carving much more balanced class boundaries locally. Our feature learning converges fast and achieves the new state-of-the-art performance on existing face recognition and attribute benchmarks.


This work is supported by SenseTime Group Limited and the General Research Fund sponsored by the Research Grants Council of the Hong Kong SAR (CUHK 14241716, 14224316, 14209217).


  • [1] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database for studying face recognition in unconstrained environments,” University of Massachusetts, Amherst, Tech. Rep. 07–49, October 2007.
  • [2] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard, “The MegaFace benchmark: 1 million faces for recognition at scale,” in CVPR, 2016.
  • [3] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, “Describable visual attributes for face verification and image search,” TPAMI, vol. 33, no. 10, pp. 1962–1977, 2011.
  • [4] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “DeepFace: Closing the gap to human-level performance in face verification,” in CVPR, 2014.
  • [5] H. He and E. A. Garcia, “Learning from imbalanced data,” TKDE, vol. 21, no. 9, pp. 1263–1284, 2009.
  • [6] H. He and Y. Ma, Imbalanced Learning: Foundations, Algorithms, and Applications.   Wiley-IEEE Press, 2013.
  • [7] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: synthetic minority over-sampling technique,” JAIR, vol. 16, no. 1, pp. 321–357, 2002.
  • [8] B. Krawczyk, M. Woźniak, and G. Schaefer, “Cost-sensitive decision tree ensembles for effective imbalanced classification,” ASC Journal, vol. 14, pp. 554–562, 2014.
  • [9] C. Drummond and R. C. Holte, “C4.5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling,” in ICMLW, 2003.
  • [10] H. Han, W.-Y. Wang, and B.-H. Mao, “Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning,” in ICIC, 2005.
  • [11] T. Maciejewski and J. Stefanowski, “Local neighbourhood extension of SMOTE for mining imbalanced data,” in ICDM, 2011.
  • [12] Y. Tang, Y.-Q. Zhang, N. Chawla, and S. Krasser, “SVMs modeling for highly imbalanced classification,” TSMC, vol. 39, no. 1, pp. 281–288, 2009.
  • [13] K. M. Ting, “A comparative study of cost-sensitive boosting algorithms,” in ICML, 2000.
  • [14] C. Huang, C. C. Loy, and X. Tang, “Discriminative sparse neighbor approximation for imbalanced learning,” TNNLS, vol. PP, no. 99, pp. 1–11, 2017.
  • [15] H. He, Y. Bai, E. A. Garcia, and S. Li, “ADASYN: Adaptive synthetic sampling approach for imbalanced learning,” in IJCNN, 2008.
  • [16] Z. H. Zhou and X. Y. Liu, “On multi-class cost-sensitive learning,” in AAAI, 2006.
  • [17] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in CVPR, 2014.
  • [18] H. Caesar, J. Uijlings, and V. Ferrari, “Joint calibration for semantic segmentation,” in BMVC, 2015.
  • [19] M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich, “Feedforward semantic segmentation with zoom-out features,” in CVPR, 2015.
  • [20] E. M. Rudd, M. Günther, and T. E. Boult, “MOON: A mixed objective optimization network for the recognition of facial attributes,” in ECCV, 2016.
  • [21] D. Eigen and R. Fergus, “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” in ICCV, 2015.
  • [22] W. Shen, X. Wang, Y. Wang, X. Bai, and Z. Zhang, “DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection,” in CVPR, 2015.
  • [23]

    S. R. Bulò, G. Neuhold, and P. Kontschieder, “Loss max-pooling for semantic image segmentation,” in

    CVPR, 2017.
  • [24] D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?” JMLR, vol. 11, pp. 625–660, 2010.
  • [25] P. Jeatrakul, K. Wong, and C. Fung, “Classification of imbalanced data by combining the complementary neural network and SMOTE algorithm,” in ICONIP, 2010.
  • [26] S. H. Khan, M. Hayat, M. Bennamoun, F. A. Sohel, and R. Togneri, “Cost-sensitive learning of deep feature representations from imbalanced data,” TNNLS, vol. PP, no. 99, pp. 1–15, 2018.
  • [27]

    C. L. Castro and A. P. Braga, “Novel cost-sensitive approach to improve the multilayer perceptron performance on imbalanced data,”

    TNNLS, vol. 24, no. 6, pp. 888–899, 2013.
  • [28] Z.-H. Zhou and X.-Y. Liu, “Training cost-sensitive neural networks with methods addressing the class imbalance problem,” TKDE, vol. 18, no. 1, pp. 63–77, 2006.
  • [29] Y.-X. Wang, D. Ramanan, and M. Hebert, “Learning to model the tail,” in NIPS, 2017.
  • [30] Q. Dong, S. Gong, and X. Zhu, “Class rectification hard mining for imbalanced deep learning,” in ICCV, 2017.
  • [31] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” IJCV, vol. 115, no. 3, pp. 211–252, 2015.
  • [32] C. Huang, Y. Li, C. C. Loy, and X. Tang, “Learning deep representation for imbalanced classification,” in CVPR, 2016.
  • [33] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in CVPR, 2015.
  • [34] S. Wang, W. Liu, J. Wu, L. Cao, Q. Meng, and P. J. Kennedy, “Training deep neural networks on imbalanced data sets,” in IJCNN, 2016.
  • [35]

    W. W. Ng, G. Zeng, J. Zhang, D. S. Yeung, and W. Pedrycz, “Dual autoencoders features for imbalance classification problem,”

    PR, vol. 60, pp. 875–889, 2016.
  • [36] M. Ren, W. Zeng, B. Yang, and R. Urtasun, “Learning to reweight examples for robust deep learning,” in ICML, 2018.
  • [37] W. Liu, R. Lin, Z. Liu, L. Liu, Z. Yu, B. Dai, and L. Song, “Learning towards minimum hyperspherical energy,” in NeurIPS, 2018.
  • [38] W. Liu, Y. Wen, Z. Yu, and M. Yang, “Large-margin softmax loss for convolutional neural networks,” in ICML, 2016.
  • [39] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in CVPR, 2017.
  • [40] H. Wang, Y. Wang, Z. Zhou, X. Ji, Z. Li, D. Gong, J. Zhou, and W. Liu, “CosFace: Large margin Cosine loss for deep face recognition,” in CVPR, 2018.
  • [41] Y. Sun, Y. Chen, X. Wang, and X. Tang, “Deep learning face representation by joint identification-verification,” in NIPS, 2014.
  • [42] Y. Sun, X. Wang, and X. Tang, “Deeply learned face representations are sparse, selective, and robust,” in CVPR, 2015.
  • [43] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” arXiv preprint, arXiv:1411.7923, 2014.
  • [44] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in ECCV, 2016.
  • [45] J. Deng, Y. Zhou, and S. Zafeiriou, “Marginal loss for deep face recognition,” in CVPRW, 2017.
  • [46] J. Liu, Y. Deng, T. Bai, and C. Huang, “Targeting ultimate accuracy: Face recognition via deep embedding,” arXiv preprint, arXiv:1506.07310, 2015.
  • [47] I. Masi, A. T. an Trãn, T. Hassner, J. T. Leksut, and G. Medioni, “Do we really need to collect millions of faces for effective face recognition?” in ECCV, 2016.
  • [48] X. Yin, X. Yu, K. Sohn, X. Liu, and M. Chandraker, “Feature transfer learning for deep face recognition with long-tail data,” arXiv preprint, arXiv:1803.09014, 2018.
  • [49] Y. Wu, H. Liu, J. Li, and Y. Fu, “Deep face recognition with center invariant loss,” in Thematic Workshop of ACM-MM, 2017.
  • [50] Y. Guo and L. Zhang, “One-shot face recognition by promoting underrepresented classes,” arXiv preprint, arXiv:1707.05574, 2018.
  • [51] X. Zhang, Z. Fang, Y. Wen, Z. Li, and Y. Qiao, “Range loss for deep face recognition with long-tailed training data,” in ICCV, 2017.
  • [52]

    T. Berg and P. N. Belhumeur, “POOF: Part-Based One-vs-One Features for fine-grained categorization, face verification, and attribute estimation,” in

    CVPR, 2013.
  • [53] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in ICCV, 2015.
  • [54] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev, “PANDA: Pose aligned networks for deep attribute modeling,” in CVPR, 2014.
  • [55] M. M. Kalayeh, B. Gong, and M. Shah, “Improving facial attribute prediction using semantic segmentation,” in CVPR, 2017.
  • [56] K. Sohn, “Improved deep metric learning with multi-class N-pair loss objective,” in NIPS, 2016.
  • [57] C. Huang, C. C. Loy, and X. Tang, “Local similarity-aware deep feature embedding,” in NIPS, 2016.
  • [58] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka, “Distance-based image classification: Generalizing to new classes at near-zero cost,” TPAMI, vol. 35, no. 11, pp. 2624–2637, 2013.
  • [59]

    L. van der Maaten and G. Hinton, “Visualizing high-dimensional data using t-SNE,”

    JMLR, vol. 9, pp. 2579–2605, 2008.
  • [60]

    J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu, “Learning fine-grained image similarity with deep ranking,” in

    CVPR, 2014.
  • [61] C. Silpa-Anan and R. Hartley, “Optimised KD-trees for fast image descriptor matching,” in CVPR, 2008.
  • [62] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” SPL, vol. 23, no. 10, pp. 1499–1503, 2016.
  • [63] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in BMVC, 2015.
  • [64] L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in CVPR, 2011.
  • [65] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Web-scale training for face identification,” in CVPR, 2015.
  • [66] B. Chen, W. Deng, and J. Du, “Noisy softmax: Improving the generalization ability of DCNN via postponing the early softmax saturation,” in CVPR, 2017.
  • [67] Y. Liu, H. Li, and X. Wang, “Rethinking feature discrimination and polymerization for large-scale recognition,” in NIPSW, 2017.
  • [68] A. Nech and I. Kemelmacher-Shlizerman, “Level playing field for million scale face recognition,” in CVPR, 2017.
  • [69] M. Günther, , A. Rozsa, and T. E. Boult, “Affact: Alignment free facial attribute classification technique,” in IJCB, 2017.
  • [70] E. Hand, C. Castillo, and R. Chellappa, “Doing the best we can with what we have: Multi-label balancing with selective learning for attribute prediction,” in AAAI, 2018.
  • [71] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  • [72] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Bengio, “An empirical evaluation of deep architectures on problems with many factors of variation,” in ICML, 2007.