Convolutional Neural Networks (ConvNets) have achieved state-of-the-art performance on various visual recognition tasks such as image classification , object detection  and semantic segmentation . The availability of a huge set of training images is one of the most important factors for their success. However, it is difficult to collect sufficient training images with unambiguous labels in domains such as age estimation , head pose estimation , multi-label classification and semantic segmentation. Therefore, exploiting deep learning methods with limited samples and ambiguous labels has become an attractive yet challenging topic.
Why is it difficult to collect a large and accurately labeled training set? Firstly, it is difficult (even for domain experts) to provide exact labels to some tasks. For example, the pixels close to object boundaries are very difficult to label for annotators in semantic segmentation. In addition, pixel labeling is a time-consuming task that may limit the amount of training samples. Another example is that people’s apparent age and head pose is difficult to describe with an accurate number. Secondly, it is very hard to gather complete and sufficient data. For example, it is difficult to build an age dataset covering people from 1 to 85 years old, and ensure that every age in this range has enough associated images. Similar difficulties arise in head pose estimation, where head poses are usually collected at a small set of angles with a 10 or 15 increment. Thus, the publicly available age, head pose and semantic segmentation datasets are small scale compared to those in image classification tasks.
These aforementioned small datasets have a common characteristic, i.e., label ambiguity, which refers to the uncertainty among the ground-truth labels. On one hand, label ambiguity is unavoidable in some applications. We usually predict another person’s age in a way like “around 25”, which indicates using not only 25, but also neighboring ages to describe the face. And, different people may have different guesses towards the same face. Similar situations also hold for other types of tasks. The labels of pixels at object boundaries are difficult to annotate because of the inherent ambiguity of these pixels in semantic segmentation. On the other hand, label ambiguity can also happen if we are not confident in the labels we provide for an image. In the multi-label classification task, some objects are clearly visible but difficult to recognize. This type of objects are annotated as Difficult in the PASCAL Visual Object Classes (VOC) classification challenge , e.g., the chair in the third image of the first row in Fig. 1.
There are two main types of labeling methods: single-label recognition (SLR) and multi-label recognition (MLR). SLR assumes one image or pixel has one label and MLR assumes that one image or pixel may be assigned multiple labels. Both SLR and MLR aim to answer the question of which labels can be used to describe an image or pixel, but they can not describe the label ambiguity associated with it. Label ambiguity will help improve recognition performance if it can be reasonably exploited. In order to utilize label correlation (which may be considered as a consequence of label ambiguity in some applications), Geng et al. proposed a label distribution learning (LDL) approach for age estimation  and head pose estimation . Recently, some improvements of LDL have been proposed. Xing et al. proposed two algorithms named LDLogitBoost and AOSO-LDLogitBoost to learn general models to relax the maximum entropy model in traditional LDL methods . Furthermore, He et al. generated age label distributions through weighted linear combination of the input image’s label and its context-neighboring samples . However, these methods are suboptimal because they only utilize the correlation of neighboring labels in classifier learning, but not in learning the visual representations.
Deep ConvNets have natural advantages in feature learning. Existing ConvNet frameworks can be viewed as classification and regression models based on different optimization objective functions. In many cases, the softmax loss and loss are used in deep ConvNet models for classification  and regression problems 
, respectively. The softmax loss maximizes the estimated probability of the ground-truth class without considering other classes, and theloss minimizes the squared difference between the estimated values of the network and the ground-truth. These methods have achieved satisfactory performance in some domains such as image classification, human pose estimation and object detection. However, existing deep learning methods cannot utilize the label ambiguity information. Moreover, a well-known fact is that learning a good ConvNet requires a lot of images.
In order to solve the issues mentioned above, we convert both traditional SLR and MLR problems to label distribution learning problems. Every instance is assigned a discrete label distribution according to its ground-truth. The label distribution can naturally describe the ambiguous information among all possible labels. Through deep label distribution learning, the training instances associated with each class label is significantly increased without actually increase the number of the total training examples. Fig. 1 intuitively shows four examples of label distribution for different recognition tasks. Then, we utilize a deep ConvNet to learn the label distribution in both feature learning and classifier learning. Since we learn label distribution with deep ConvNets, we call our method DLDL: Deep Label Distribution Learning. The benefits of DLDL are summarized as follows:
DLDL is an end-to-end learning framework which utilizes the label ambiguity in both feature learning and classifier learning;
DLDL not only achieves more robust performance than existing classification and regression methods, but also effectively relaxes the requirement for large amount of training images, e.g., a training face image with ground-truth label 25 is also useful for predicting faces at age 24 or 26;
DLDL (only single model without ensemble) achieves better performance than the state-of-the-art methods on age and head pose estimation tasks. DLDL also improves the performance for multi-label classification and semantic segmentation.
The rest of this paper is organized as follows. We first review the related work in Section II. Then, Section III proposes the DLDL framework, including the DLDL problem definition, DLDL theory, label distribution construction and training details. After that, the experiments are reported in Section IV. Finally, Section V presents discussions and the conclusion is given in Section VI.
Ii Related Work
In the past two decades, many efforts have been devoted to visual recognition, including at least image classification, object detection, semantic segmentation, and facial attribute (apparent age and head pose) estimation. These works can be divided into two streams. Earlier research was mainly based on hand-crafted features, while more recent ones are usually deep learning methods. In this section, we briefly review these related approaches.
Methods based on hand-crafted features usually include two stages. The first stage is feature extraction. The second stage learns models for recognition, detection or estimation using these features. SVM, random forest and neural networks have commonly been used during the learning stage. In addition, Geng et al. proposed the label distribution learning approach to utilize the correlation among adjacent labels, which further improved performance on age estimation  and head pose estimation .
Although important progresses have been made with these features, the hand-crafted features render them suboptimal for particular tasks such as age or head pose estimation. More recently, learning feature representation has shown great advantages. For example, Lu et al.  tried to learn cost-sensitive local binary features for age estimation.
Deep learning has substantially improved upon the state-of-the-art in image classification , object detection , semantic segmentation  and many other vision tasks. In many cases, the softmax loss is used in deep models for classification . Besides classification, deep ConvNets have also been trained for regression tasks such as head pose estimation  and facial landmark detection . In regression problems, the training procedure usually optimizes a squared loss function. Satisfactory performance has also been obtained by using Tukey’s biweight function in human pose estimation . In terms of model architecture, deep ConvNet models which use deeper architecture and smaller convolution filters (e.g., VGG-Nets  and VGG-Face ) are very powerful. Nevertheless, these deep learning methods do not make use of the presence of label ambiguity in the training set, and usually require a large amount of training data.
A latest approach, in Inception-v3 
, is based on label smoothing (LS). Instead of only using the ground-truth label, they utilize a mixture of the ground-truth label and a uniform distribution to regularize the classifier. However, LS is limited to the uniform distribution among labels rather than mining labels’ ambiguous information. We believe that label ambiguity is too important to ignore. If we make good use of the ambiguity, we expect the required number of training images for some tasks could be effectively reduced.
In this paper, we focus on how to exploit the label ambiguity in deep ConvNets. Age and head pose estimation from still face images are suitable applications of the proposed research. In addition, we also extend our works to multi-label classification and semantic segmentation.
Iii The Proposed DLDL Approach
In this section, we firstly give the definition of the DLDL problem. Then, we present the DLDL theory. Next, we propose the construction methods of label distribution for different recognition tasks. Finally, we briefly introduce the DLDL architecture and training details.
Iii-a The deep label distribution learning problem
Given an input image, we are interested in estimating a category output (e.g., age or head pose angles). For two input images and with ground-truth labels and , and are supposed to be similar to each other if the correlation of and is strong, and vice versa. For example, the correlation between faces aged 32 and 33 should be stronger than that between faces aged 32 and 64, in terms of facial details that reflect the age (e.g., skin smoothness). In other words, we expect high correlation among input images with similar outputs. The label distribution learning approach [4, 7]
exploited such correlations in the machine learning phase, but used features that are extracted ignoring these correlations. The proposed DLDL approach, however, is an end-to-end deep learning method which utilizes such correlation information in both feature learning and classifier learning. We will also extend DLDL to handle other types of label ambiguity beyond correlation.
To fulfill this goal, instead of outputting a single value for an input , DLDL quantizes the range of possible values into several labels. For example, in age estimation, it is reasonable to assume that , and it is a common practice to estimate integer values for ages. Thus, we can define the set as the ordered label set for age estimation. The task of DLDL is then to predict a label distribution , where is the estimated probability that should be predicted to be years old. By estimating an entire label distribution, the deep learning machine is forced to take care of the ambiguity among labels.
Specifically, the input space of our framework is , where , and are the height, width, and number of channels of the input image, respectively. DLDL predicts a label distributionvector , where is the label set defined for a specific task (e.g., the above). We assume is complete, i.e., any possible value has a corresponding member in . A training data set with instances is then denoted as . We use boldface lowercase letters like to denote vectors, and the -th element of is denoted as . The goal of DLDL is to directly learn a conditional probability mass function from , where is the parameters in the framework.
Iii-B Deep label distribution learning
Given an instance with label distribution , we assume that
is the activation of the last fully connected layer in a deep ConvNet. We use a softmax function to turn these activations into a probability distribution, that is,
Given a training data set , the goal of DLDL is to find to generate a distribution that is similar to .
There are different criteria to measure the similarity or distance between two distributions. For example, if the Kullback-Leibler (KL) divergence is used as the measurement of the similarity between the ground-truth and predicted label distribution, then the best parameter is determined by
Thus, we can define the loss function as:
and the derivative of softmax (Eq. 1) is well known, as
where is 1 if
, and 0 otherwise. According to the chain rule, for any fixed, we have
Thus, the derivative of with respect to is
Once is learned, the label distribution of any new instance can be generated by a forward run of the network. If the expected class label is a single one, DLDL outputs , where
Prediction with multiple labels is also allowed, which could be a set where is a predefined threshold. If the expected output is a real number, DLDL predicts the expectation of , as
where . This indicates that DLDL is suitable for both classification and regression tasks.
Iii-C Label distribution construction
The ground-truth label distribution is not available in most existing datasets, which must be generated under proper assumptions. A desirable label distribution must satisfy some basic principles: (1) should be a probability distribution. Thus, we have and . (2) The probability values should have difference among all possible labels associated with an image. In other words, a less ambiguous category must be assigned high probability and those more ambiguous labels must have low probabilities. In this section, we propose the way to construct label distributions for age estimation, head pose estimation, multi-label classification and semantic segmentation.
For age estimation, we assume that the probabilities should concentrate around the ground-truth age . Thus, we quantize to get
using a normal distribution. For example, the apparent age of a face is labeled by hundreds of users. The ground-truth (including a mean
and a standard deviation) is calculated from all the votes. For this problem, we find the range of the target (e.g., ), quantize it into a complete and ordered label set , where is the label set size and are all possible predictions for . A label distribution is then , where is the probability that (i.e., for ). Since we use equal step size in quantizing
, the normal p.d.f. (probability density function) is a natural choice to generate the ground-truthfrom and :
where . Fig. (a)a shows a face and its corresponding label distribution. For problems where is unknown, we will show that a reasonably chosen also works well in DLDL.
For head pose estimation, we need to jointly estimate pitch and yaw angles. Thus, learning joint distribution is also necessary in DLDL. Suppose the label set is, where is a pair of values. That is, we want to learn the joint distribution of two variables. Then, the label distribution can be represented by an matrix, whose -th element is . For example, when we use two angles (pitch and yaw) to describe a head pose, is a pair of pitch and yaw angles. Given an instance with ground-truth mean and covariance matrix , we calculate its label distribution as
where . In the above, we assume , that is, the covariance matrix is diagonal. Fig. (b)b shows a joint label distribution with head pose and .
For multi-label classification, a multi-label image always contains at least one object of the class of interest. There are usually multiple labels for an image. These labels are grouped into three different levels, including Positive, Negative and Difficult in the PASCAL VOC dataset . A label is Positive means an image contains objects from that category, and Negative otherwise. Difficult indicates that an object is clearly visible but difficult to recognize. Existing multi-label methods often view Difficult as Negative, which leads to the loss of useful information. It is not reasonable either if we simply treat Difficult as Positive. Therefore, a nature choice is to use label ambiguity. We define different probabilities for different types of labels, as
for Positive, Difficult and Negative labels, respectively. Furthermore, an normalization is applied to ensure :
where equals , or if the label is Positive, Difficult or Negative, respectively. The label distribution is shown for a multi-label image in Fig. (c)c.
For semantic segmentation, we need to label a pixel as belonging to one class if it is a pixel inside an object of that class, or as the background otherwise. Let denote the annotation of the -th pixel, where (assuming there are categories and 0 for background). Fully Convolutional Networks (FCN) have been an effective solution to this task. In FCN , a ground-truth label means that and for all . However, it is very difficult to specify ground-truth labels for pixels close to object boundaries, because labels of these pixels are inherently ambiguous. We propose a mechanism to describe the label ambiguity in the boundaries. Considering a Gaussian kernel matrix , we replace the original label distribution with , as
where , , is the kernel size, and, and , and the generated label distribution is
Fig. (d)d gives the semantic label distribution for a bird image which shows that the ambiguity is encoded in the label distributions.
Iii-D The DLDL architecture and training details
. The ZF-Net consists five convolution layers, followed by three fully connected layers. The VGG-Nets architecture includes 16 or 19 layers. We modify the last fully connected layer’s output based on the task and replace the original softmax loss function with the KL loss function. In addition, we use the parameter ReLU
for ZF-Net. In our network, the input is an order three tensorand the output may be a vector (age estimation and multi-label classification), a matrix (head pose estimation) or a tensor (semantic segmentation).
In this paper, we train the deep models in two ways:
Training from scratch.
For ZF-Net, the initialization is performed randomly, based on a Gaussian distribution with zero mean and 0.01 standard deviation, and biases are initialized to zero. The coefficient of the parameter ReLU is initialized to 0.25. The dropout is applied to the last two fully connected layers with rate 0.5. The coefficient of weight decay is set to
. Optimization is done by Stochastic Gradient Descent (SGD) using mini-batches of 128 and the momentum coefficient is 0.9. The initial learning rate is set to 0.01. The total number of epochs is about 20.
Fine-tuning. Three pre-trained models including VGG-Nets (16-layers and 19-layers) and VGG-Face (16-layers) are used to fine-tune for different tasks. We remove these pre-trained models’ classification layer and loss layer, and put in our label distribution layer which is initialized by the Gaussian distribution and the KL loss layer. The learning rates of the convolutional layers, the first two fully-connected layers and the label distribution layer are initialized as 0.001, 0.001 and 0.01, respectively. We fine-tune all layers by back propagation through the whole net using mini-batches of 32. The total number of epochs is about 10 for age estimation and 20 for multi-label classification.
We evaluate DLDL on four tasks, i.e., age estimation, head pose estimation, multi-label classification and semantic segmentation. Our implementation is based on MatConvNet .111http://www.vlfeat.org/matconvnet/ All our experiments are carried out on a NVIDIA K40 GPU with 12GB of onboard memory.
Iv-a Age estimation
Datasets. Two age estimation datasets are used in our experiments. The first is Morph , which is one of the largest publicly available age datasets. There are 55,134 face images from more than 13,000 subjects. Ages range from 16 to 77. Since no TRAIN/TEST split is provided, 10-fold cross-validation is used for Morph.
The second dataset is from the apparent age estimation competition, the first competition track of the ICCV ChaLearn LAP 2015 workshop . Compared with Morph, this dataset (ChaLearn) consists of images collected in the wild, without any position, illumination or quality restriction. The only condition is that each image contains only one face. The dataset has 4,699 images, and is split into 2,476 training (TRAIN), 1,136 validation (VAL) and 1,087 testing (TEST) images. The apparent age (i.e., how old does this person look like) of each image is labeled by multiple individuals. The age of face images range from 3 to 85. For each image, its mean age and the corresponding standard deviation are given. Since the ground-truth for TEST images are not published, we train on the TRAIN split and evaluate on the VAL split of ChaLearn images.
Baselines. To demonstrate the effectiveness of DLDL, we firstly consider two related methods as baselines: ConvNet+LS (KL) and ConvNet+LD (-div). The former uses label smoothing (LS)  as ground-truth and KL divergence as loss function. The latter uses label distribution (LD) as ground-truth and divergence  as loss function, which is
In addition, we also compare DLDL with the following baseline methods:
BFGS-LDL Geng et al. proposed the label distribution learning approach (IIS-LLD) for age and head pose estimation. They used traditional image features. To further improve IIS-LLD, Geng et al.  proposed a BFGS-LDL algorithm by using the effective quasi-Newton optimization method BFGS.
R-ConvNet ConvNets are also successively trained for regression tasks. In R-ConvNet, the ground-truth label (age and pose angle) is projected into the range by the mapping , where and are the maximum and minimum values in the training label set. During prediction, the R-ConvNet regression result is reverse mapped to get
. To speed up convergence, the last fully connected layer is followed a hyperbolic tangent activation function, which maps to . The squared , and -ins loss functions are used in R-ConvNet.
We use the same preprocessing pipeline for all compared methods, including face detection, facial key points detection and face alignment, as shown in Fig2. We employ the DPM model  to detect the main facial region. Then, the detected face is fed into cascaded convolution networks  to get the five facial key points, including the left/right eye centers, nose tip and left/right mouth corners. Finally, based on these facial points, we align the face to the upright pose. Data augmentation are only applied to the training images for ChaLearn. For one color input training image, we generate its gray-scale version, and left-right flip both color and gray-scale versions. Thus, every training image turns into 4 images.
We define for both datasets. The label distribution of each image is generated using Eq. 10. The mean is provided in both Morph and ChaLearn. The standard deviation , however, is provided in ChaLearn but not in Morph. We simply set in Morph. Experiments for different methods are conducted under the same data splits.
|M-S ConvNets ||3.63||-||-|
|VGG (softmax, Exp) 3||-||6.08||0.51|
|VGG (softmax, Exp) 2,3||-||3.22||0.28|
|VGG (softmax, Exp) 2,3||2.68||3.25||0.28|
|BFGS-LDL (KL, Max)||3.940.05||7.81||0.57|
|BFGS-LDL (KL, Exp)||3.850.05||6.79||0.53|
|C-ConvNet (softmax, Max)||3.020.05||9.48||0.63|
|C-ConvNet (softmax, Exp)||2.860.05||7.95||0.58|
|ConvNet+LS (KL, Max)||2.960.13||8.64||0.59|
|ConvNet+LS (KL, Exp)||5.020.13||11.58||0.77|
|ConvNet+LD (-div, Max)||2.570.04||0.47|
|ConvNet+LD (-div, Exp)||2.570.04||0.46|
|DLDL (KL, Max)||2.510.03||0.44|
|DLDL (KL, Exp)||2.520.03||0.44|
|DLDL+VGG-Face (KL, Max)3||2.420.01||3.62||0.32|
|DLDL+VGG-Face (KL, Exp)3||2.430.01||3.51||0.31|
1 Used 80% of Morph images for training and 20% for evaluation;
2 Used additional external face images (i.e., IMDB-WIKI);
3 Used pre-trained model (i.e., VGG-Nets or VGG-Face).
Evaluation criteria. Mean Absolute Error (MAE) and Cumulative Score (CS) are used to evaluate the performance of age estimation. MAE is the average difference between the predicted and the real age:
where and are the estimated and ground-truth age of the -th testing image, respectively. CS is defined as the accuracy rate of correct estimation:
where is the number of correct estimation, i.e., testing images that satisfy . In our experiment, . In addition, a special measurement (named -error) is defined by the ChaLearn competition, computed as
Results. Table I lists results on both datasets. The upper part shows results in the literature. The middle part shows the baseline results. The lower part shows the results of the proposed approach. The first term in the parenthesis behind each method is the loss function corresponding to the method. Max or Exp represent predicting according to Eq. 8 or 9, respectively. Since cross-validation is used in Morph, we also provide its standard deviations.
From Table I, we can see that DLDL consistently outperforms baselines and other published methods. The difference between DLDL (KL, Max) and its competitor C-ConvNet (softmax, Max) is 0.51 on Morph
. This gap is more than 6 times the sum of their standard deviations (0.03+0.05), showing statistically significant differences. The advantage of DLDL over R-ConvNet, C-ConvNet and ConvNet+LS suggests that learning label distribution is advantageous in deep end-to-end models. DLDL has much better results than BFGS-LDL, which shows that the learned deep features are more powerful than manually designed ones. Compared to ConvNet+LD (-div), DLDL (KL) achieves lower MAE on both datasets. It indicates that KL-divergence is better than -divergence for measuring the similarity of two distributions in this context.
We find that C-ConvNet and R-ConvNet are not stable. The R-ConvNet () method, although being the second best method for ChaLearn, is inferior to C-ConvNet (softmax, Exp) for Morph. In addition, we also find that Eq. 9 is better than Eq. 8 in many cases, which suggests that Eq. 9 is more suitable than Eq. 8 for age estimation.
Fine-tuning DLDL. Instead of training DLDL from scratch, we also fine-tune the network of VGG-Face . On the small scale ChaLearn dataset, the MAE of DLDL is reduced from 5.34 to 3.51, yielding a significant improvement. The -error of DLDL is reduced from 0.44 to 0.31, which is close to the best competition result 0.28  on the validation set. In , external training images (260,282 additional external training images with real age annotation) were used. DLDL only uses the ChaLearn dataset’s 2,476 training images and is the best among ChaLearn teams that do not use external data . In the competition, the best external-data-free -error is 0.48, which is worse than DLDL’s. However, the idea in  to use external data is useful for further reducing DLDL’s estimation error.
Fig. (a)a and Fig. (b)b show the CS curves on ChaLearn and Morph datasets. At every error level, our DLDL fine-tuned VGG-Face always achieves the best accuracy among all methods. It is noteworthy that the CS curves of DLDL (KL, Max) and ConvNet (-div, Max) are very close to that of the DLDL+VGG-Face (KL, Max) on Morph even without lots of external data and very deep model. This observation supports the idea that using DLDL can achieve competitive performance even with limited training samples.
In Fig. (g)g, we show some examples of face images from the ChaLearn validation set and predicted label distributions by DLDL (KL, Exp). In many cases, our solution is able to accurately predict the apparent age of faces. Failures may come from two causes. The first is the failure to detect or align the face. The second is some extreme conditions of face images such as occlusion, low resolution, heavy makeup and old photos.
Iv-B Head pose estimation
Datasets. We use three datasets in head pose estimation: Pointing’04 , BJUT-3D  and Annotated Facial Landmarks in the Wild (AFLW) . In them, head pose is determined by two angles: pitch and yaw. Pointing’04 discretizes the pitch into 9 angles and the yaw into 13 angles . When the pitch angel is or , the yaw angle is always set to . Thus, there are 93 poses in total. The head images are taken from 15 different human subjects in two different time periods, resulting in images.
BJUT-3D contains 500 3D faces (250 male and 250 female people), acquired by a CyberWare Laser Scanner in an engineered environment. 9 pitch angles and 13 yaw angles are used. There are in total 93 poses in this dataset, similar to that in Pointing’04. Therefore, face images are obtained.
|Methods||Description||MAE (lower is better)||Acc (higher is better)|
|Methods||Description||MAE (lower is better)||Acc (higher is better)|
Unlike Pointing’04 and BJUT-3D, the AFLW is a real-world face database. Head pose is coarsely obtained by fitting a mean 3D face with the POSIT algorithm . The dataset contains about 24k faces in real-world images. We select 23,409 faces to ensure pitch and yaw angles within .
Implementation details. The head region is provided by bounding box annotations in Pointing’04 and AFLW. The BJUT-3D does not contain background regions. Therefore, we will not perform any preprocessing.
In DLDL, we set in Pointing’04 and in BJUT-3D for constructing label distributions. For AFLW, ground-truth of head pose angles are given as real numbers. Ground-truth (pitch and yaw) angles are divided from to in steps of , so we get (pitch, yaw) pair category labels. We set for AFLW. Since the discrete Jeffrey’s divergence is used in LDL , we implement BFGS-LDL with the Kullback-Leibler divergence. All experiments are performed under the same setting, including data splits, input size and network architecture.
To validate the effectiveness of DLDL for head pose estimation, we use the same baselines as age estimation. Our experiments show that Eq. 9 has lower accuracy than Eq. 8. Hence, we use Eq. 8 in this section.
Evaluation criteria. Three types of prediction values are evaluated: pitch, yaw, and pitch+yaw, where pitch+yaw jointly estimates the pitch and yaw angles. Two different measurements are used, which is MAE (Eq. 17) and classification accuracy (Acc). When we treat different poses as different classes, Acc measures the pose class classification accuracy. In particular, the MAE of pitch+yaw is calculated as the Euclidean distance between the predicted (pitch, yaw) pair and the ground-truth pair; the Acc of pitch+yaw is calculated by regarding each (pitch, yaw) pair as a class. For R-ConvNet, we only report its MAE but not Acc, because its predicted value are continuous real numbers. All methods are tested with 5-fold cross validation for Pointing’04 and BJUT-3D following . For AFLW, 15,561 face images are randomly chosen for training, and the remaining 7,848 for evaluation. The setup is similar to the recent literature  (14,000 images for training and the rest 7,041 images for testing).
|Description||MAE (lower is better)||Acc (higher is better)|
Results. Tables II, III and IV show results on Pointing’04, BJUT-3D and AFLW, respectively. Pointing’04 is small scale with only 2,790 images. We observe that BFGS-LDL (with hand-crafted features) has much lower MAE and much higher accuracy than deep learning methods C-ConvNet, R-ConvNet and ConvNet+LS. One reasonable conjecture is that C-ConvNet, R-ConvNet and ConvNet+LS are not well-learned with only small number of training images. DLDL, however, successfully learns the head pose. For example, its accuracy for pitch+yaw is 73.15 (and C-ConvNet is only 42.97). That is, DLDL is able to perform deep learning with few training images, while C-ConvNet R-ConvNet and ConvNet+LS have failed for this task.
On BJUT-3D and AFLW which have enough training data, we observe that many deep learning methods show higher performance than BFGS-LDL. DLDL achieves the best performance: it has much lower MAE and higher accuracy than other methods. Another observation is also worth mentioning. Although R-ConvNet is better than C-ConvNet when label is dense such as age estimation and head pose estimation on AFLW, it is obviously worse than C-ConvNet on BJUT-3D and pointing’04 for head pose estimation which have sparse labels. In other words, the performance of C-ConvNet and R-ConvNet are not very robust, while the proposed method consistently achieves excellent performance.
Fig. (c)c shows the pitch+yaw CS curves on the AFLW dataset. There is an obvious gap between DLDL and baseline methods at every error level. Fig. (g)g shows the predicted label distributions for different head poses on the AFLW testing set using the DLDL model. Our approach can estimate head pose with low errors but may fail under some extreme conditions. It is noteworthy that DLDL may produce more incorrect estimations when both yaw and pitch are large (e.g., ). The reason might be that there are much fewer training examples for large angles than for other angles.
Iv-C Multi-label classification
Datasets. We evaluate our approach for multi-label classification on the PASCAL VOC dataset : PASCAL VOC2007 and VOC2012. There are 9,963 and 22,531 images in them, respectively. Each image is annotated with one or several labels, corresponding to 20 object categories. These images are divided into three subsets including TRAIN, VAL and TEST sets. We train on the TRAINVAL set and evaluate on the TEST
set. The evaluation metric is average precision (AP) and mean average precision (mAP), complying with the PASCAL challenge protocols.
We denote our methods as Images-Fine-tuning-DLDL (IF-DLDL) and Proposals-Fine-tuning-DLDL (PF-DLDL) when ConvNets are fine-tuned by images and proposals of images, respectively. Details of these two variants are explained later in this section. We compare the proposed approaches with the following methods:
VGG+SVM . This method densely extracted 4,096 dimensional ConvNet features at the penultimate layer of VGG-Nets pre-trained on ImageNet. These features from different scales (smallest image side ) were aggregated by average pooling. Then, these averaged features from two networks (“Net-D” containing 16 layers and “Net-E” containing 19 layers) were further fused by stacking. Finally,  normalized the resulting image features and used these features to train a linear SVM classifier for multi-label classification.
HCP . HCP proposed to solve the multi-label object recognition task by extracting object proposals from the images. The method used image label and square loss to fine-tune a pre-trained ConvNet. Then, BING  or EdgeBoxes 
was used to extract object proposals, which were used to fine-tune the ConvNet again. Finally, scores of these proposals were max-pooled to obtain the prediction.
Fev+Lv . This approach transformed the multi-label object recognition problem into a multi-class multi-instance learning problem. Two views (label view and feature view) were extracted for each proposal of images. Then, these two views were encoded by a Fisher vector for each image.
IF-VGG- and IF-VGG-KL. We fine-tune the VGG-Nets with square loss and multi-label cross-entropy loss  and use them as our IF-DLDL’s baselines. They are trained using the same setting.
Implementation details. According to the ground-truth labels, we set different probabilities for all possible labels on PASCAL VOC dataset. In our experiments, , , . Finally, similar to label smoothing, a uniform distribution is added to , where .
IF-DLDL. Following , each training image is individually rescaled by randomly sampling in the range [256, 512]. We randomly crop patches from these resized images. We also adjust the pooling kernel in the pool5 layer from to . Max-pooling and Avg-pooling are used at pool5 to train two ConvNets. We obtain four ConvNet models thought fine-tuning “Net-D” and “Net-E”. At the prediction stage, the smaller side of each image is scaled to a fixed length . Each scaled image is fed to the fine-tuned ConvNets to obtain the 20-dim probability outputs. These probability outputs from different scales and different models are averaged to form the final prediction.
PF-DLDL. Following , we further fine-tune IF-DLDL models with proposals of images to boost performance. For each training image, we employ EdgeBoxes  to produce a set of proposal bounding boxes which are grouped into clusters by the normalized cut algorithm . For each cluster, the top proposals with higher predictive scores generated by EdgeBoxes are resized into square shapes (i.e., ). As a result, we can obtain proposals for an image. Finally, these resized proposals are fed into a fine-tuned IF-DLDL model to obtain prediction scores and these scores are fused by max-pooling to form the prediction distribution of the image. This process can be learned by using an end-to-end way. In our implementation, we set and at the training and the prediction stage, respectively. Similar to IF-DLDL, we also average fuse prediction scores of different models to generate the final prediction.
Results. In Table V, we compare single model results (average AP of all classes) on VOC2007. Our PF-DLDL defeats all the other methods. Compared with Fev+Lv , 1.7% improvement can be achieved by PF-DLDL even without using the bounding box annotation. Compared with HCP-VGG , our PF-DLDL can achieve 92.3% mAP, which is significantly higher than their 90.9%. This further indicates that it is very important to learn a label distribution.
Table VI and VII report details of all experimental results on VOC2007 and VOC2012, respectively. It can be seen that IF-DLDL outperforms IF-VGG- by 1.1% for VOC2007 and 1.3% for VOC2012, which indicates that the KL loss function is more suitable than loss for measuring the similarity of two label distributions. Furthermore, IF-DLDL improves IF-VGG-KL for about 0.2–0.3 points in mAP, which suggests that learning a label distribution is beneficial. More importantly, PF-DLDL can achieve 93.4% for VOC2007 and 92.4% for VOC2012 in mAP when we average fuse output scores of four PF-DLDL models.
Our framework shows good performance especially for scene categories such as “chair”, ‘table” and “sofa”. Although PF-DLDL significantly outperforms IF-DLDL in mAP, PF-DLDL has higher computational cost than IF-DLDL on both training and testing stages. Since IF-DLDL does not need region proposals or bounding box information, it may be effectively and efficiently implemented for practical multi-label application such as multi-label image retrieval. It is also possible that by adopting new techniques (such as the region proposal method using gated unit in , which has higher accuracy that ours on VOC tasks), the accuracy of our DLDL methods can be further improved.
Iv-D Semantic segmentation
Datasets. We employ the PASCAL VOC2011 segmentation dataset and the Semantic Boundaries Dataset (SBD) for training the proposed DLDL. There are 2,224 images (1,112 for training and 1,112 for testing) with pixel labels for 20 semantic categories in VOC2011. SBD contains 11,355 annotated images (8,984 for training and 2,371 for testing) from Hariharan et al. . Following FCN , we train DLDL using the union set (8,825 images) of SBD and VOC2011 training images. We evaluate the proposed approach on VOC2011 (1,112) and VOC2012 (1,456) test images.
Evaluation criteria. The performance is measured in terms of mean IU (intersection over union), which is the most widely used metric in semantic segmentation.
We keep the same settings as FCN including training images and model structure. The main change is that we employ KL divergence as the loss function based on label distribution (Eq. 15). Note that although we transform the ground-truth to label distribution in the training process, our evaluation rely only on ground-truth label.
Recently, Conditional Random Field (CRF) has been broadly used in many state-of-the-art semantic segmentation systems. We optionally employ a fully connected CRF  to refine the predicted category score maps using the default parameters of .
Results. Table VIII gives the performance of DLDL-8s and DLDL-8s-CRF on the test images of VOC2011 and VOC2012 and compares it to the well-known FCN-8s. DLDL-8s improves the mean IU of FCN-8s form 62.7% to 64.9% on VOC2011. On VOC2012, DLDL-8s leads to an improvement of 2.3 points in mean IU. DLDL achieves better results than FCN, which suggests it is important to improve the segmentation performance using label ambiguity. In addition, the CRF further improve performance of DLDL-8s, offering a 2.6% absolute increase in mean IU both on VOC2011 and VOC2012.
|Methods||mean IU||mean IU|
|VOC2011 test||VOC2012 test|
Fig. 7 shows four semantic segmentation examples from the VOC2011 validation images using FCN-8s, DLDL-8s and DLDL-8s-CRF. We can see that DLDL-8s can successfully segment some small objects (e.g., car and bicycle) and particularly improve the segmentation of object boundaries (e.g., horse’s leg and plant’s leaves), but FCN-8s does not. DLDL-8s may fail, e.g., it sees a flowerpot as a potted plant in the fourth row in Fig. 7. Furthermore, compared to DLDL-8s, DLDL-8s-CRF is able to refine coarse pixel-level label predictions to produce sharp boundaries and fine-grained segmentations (e.g., plant’s leaves).
In this section, we try to understand the generalization performance of DLDL through feature visualization, and to analyze why DLDL can achieve high accuracy with limited training data. In addition, a study of the hyper-parameter is also provided.
Feature visualization. We visualize the model features in a low-dimensional space. Early layers learn low-level features (e.g., edge and corner) and latter layers learn high level features (e.g., shapes and objects) in a deep ConvNet . Hence, we extract the penultimate layer features (4,096-dimensional) on Morph, ChaLearn, Pointing’04 and AFLW validation sets. To obtain the 2-dimensional embeddings of the extracted high dimensional features, we employ a popular dimension reduction algorithm t-SNE . The low-dimensional embeddings of validation images from the above four datasets are shown in Fig. 6. The first row shows the 2-dim embeddings of hand-crafted features (BIF for Morph and Chalearn, HOG for Pointing’04 and AFLW) and the second row shows that of the DLDL features. These figures are colored by their semantic category. It can be observed that clear semantic clusterings (old or young for age datasets, left or right, up or down for head pose datasets) appear in deep features but do not in hand-crafted features.
Reduce over-fitting. DLDL can effectively reduce over-fitting when the training set is small. This effect can be explained by the label ambiguity. Considering an input sample with one single label . In traditional deep ConvNet, and for all . In DLDL, the label distribution contains many non zeros elements. The diversity of labels helps reduce over-fitting. Moreover, the objective function (Eq. 3) of DLDL can be rewritten as
In Eq. 20, the first term is the tradition ConvNet loss function. The second term maximize the log-likelihood of the ambiguous labels. Unlike existing data augmentation techniques such as random cropping on the images, DLDL augments data on the label side.
In Fig. 8, MAE is shown as a function of the number of epochs on two age datasets (ChaLearn and Morph) and two head pose datasets (BJUT-3D and AFLW). On ChaLearn and AFLW, C-ConveNet (softmax) achieves the lowest training MAE, but produces the highest validation MAE. In particular, the validation MAE increases after the 8th epoch on ChaLearn. Similar phenomenon is observed on AFLW. This fact shows that over-fitting happens in C-ConvNet when the number of training images is small. Although there are 15,561 training images in AFLW, each category contains on averagely 4 training images since there are 3,721 categories.
Accelerate convergence. We further analyze the convergence performance of DLDL, C-ConvNet and R-ConvNet. We can observe that the training MAE is reduced very slowly at the beginning of training using C-ConvNet and R-ConveNet in many cases as shown in Fig. 8. On the contrary, the MAE of DLDL reduces quickly.
Robust performance. One notable observation is that C-ConvNet and R-ConveNet is unstable. Fig. (c)c shows the MAE for pitch+yaw, a complicated estimation of the joint distribution. This is a very sparse label set because the interval of adjacent class (pitch or yaw) is . R-ConvNet has difficulty in estimating this output, yielding errors that are roughly 20 times higher than DLDL and C-ConvNet. On the other hand, C-ConvNet easily fall into over-fitting when there are not enough training data (e.g, Fig. (a)a and Fig. (d)d). The proposed DLDL is more amenable to small datasets or sparse labels than C-ConvNet and R-ConvNet.
Analyze the hyper-parameter. DLDL’s performance may be affected by the label distribution. Here, we take age estimation (Morph) and head pose estimation (Pointing’04) for examples. is a common hyper-parameter in these tasks if it is not provided in the ground-truth. We have empirically set in Morph, and in Pointing’04 in our experiments. In order to study the impact of , we test DLDL with different values, changing from 0 to 3 with 0.5 interval. Fig. 9 shows the MAE performance on Morph and Pointing’04 with different . We can see that a proper is important for low MAE. But generally speaking, a value that is close to the interval between neighboring labels is a good choice. Because the shape of all curves are V-shape like, it is also very convenient to find an optimal value using the cross-validation strategy.
We observe that current deep ConvNets cannot successfully learn good models when there are not enough training data and/or the labels are ambiguous. We propose DLDL, a deep label distribution learning framework to solve this issue by exploiting label ambiguity. In DLDL, each image is labeled by a label distribution, which can utilize label ambiguity in both feature learning and classifier learning. DLDL consistently improves the network training process in our experiments, by preventing it from over-fitting when the training set is small. We empirically showed that DLDL produces robust and competitive performances than traditional classification or regression deep models on several popular visual recognition tasks.
However, constructing a reasonable label distribution is still challenging due to the diversity of label space for different recognition tasks. It is an interesting direction to extend DLDL to more recognition problems by constructing different label distributions.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems, 2012, pp. 1097–1105.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies
for accurate object detection and semantic segmentation,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580–587.
-  J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
-  X. Geng, C. Yin, and Z.-H. Zhou, “Facial age estimation by learning from label distributions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 10, pp. 2401–2412, 2013.
-  S. G. Kong and R. O. Mbouna, “Head pose estimation from a 2D face image using 3D face morphing with depth parameters,” IEEE Transactions on Image Processing, vol. 24, no. 6, pp. 1801–1808, 2015.
-  M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The PASCAL visual object classes (VOC) challenge,” International Journal of Computer Vision, vol. 88, no. 2, pp. 303–338, 2010.
-  X. Geng and Y. Xia, “Head pose estimation based on multivariate label distribution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1837–1842.
-  C. Xing, X. Geng, and H. Xue, “Logistic boosting regression for label distribution learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4489–4497.
-  Z. He, X. Li, Z. Zhang, F. Wu, X. Geng, Y. Zhang, M.-H. Yang, and Y. Zhuang, “Data-dependent label distribution learning for age estimation,” IEEE Transactions on Image Processing, 2017, to be published, doi: 10.1109/TIP.2017.2655445.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
-  V. Belagiannis, C. Rupprecht, G. Carneiro, and N. Navab, “Robust optimization for deep regression,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2830–2838.
-  G. Fanelli, J. Gall, and L. Van Gool, “Real time head pose estimation with random regression forests,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 617–624.
-  J. Lu, V. E. Liong, and J. Zhou, “Cost-sensitive local binary feature learning for facial age estimation,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5356–5368, 2015.
-  B. Ahn, J. Park, and I. S. Kweon, “Real-time head orientation from a monocular camera using deep neural network,” in Asian Conference on Computer Vision, 2015, pp. 82–96.
-  Y. Sun, X. Wang, and X. Tang, “Deep convolutional network cascade for facial point detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 3476–3483.
-  K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proceedings of International Conference on Learning Representations, 2015, pp. 1–14.
-  O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,” in Proceedings of the British Machine Vision Conference, 2015, p. 6.
-  C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826.
-  M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European Conference on Computer Vision, 2014, pp. 818–833.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026–1034.
-  A. Vedaldi and K. Lenc, “MatConvNet: Convolutional neural networks for MATLAB,” in Proceedings of the 23rd ACM International Conference on Multimedia, 2015, pp. 689–692.
-  K. Ricanek Jr and T. Tesafaye, “Morph: A longitudinal image database of normal adult age-progression,” in International Conference on Automatic Face and Gesture Recognition, 2006, pp. 341–345.
-  S. Escalera, J. Fabian, P. Pardo, X. Baró, J. Gonzalez, H. J. Escalante, D. Misevic, U. Steiner, and I. Guyon, “Chalearn looking at people 2015: Apparent age and cultural event recognition datasets and results,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2015, pp. 1–9.
-  T. Minka, “Divergence measures and message passing,” Microsoft Research, Tech. Rep. MSR-TR-2005-173, 2005.
-  X. Geng, “Label distribution learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 28, no. 7, pp. 1734–1748, 2016.
-  M. Mathias, R. Benenson, M. Pedersoli, and L. Van Gool, “Face detection without bells and whistles,” in European Conference on Computer Vision, 2014, pp. 720–735.
-  K.-Y. Chang and C.-S. Chen, “A learning framework for age rank estimation based on face images with scattering transform,” IEEE Transactions on Image Processing, vol. 24, no. 3, pp. 785–798, 2015.
-  D. Yi, Z. Lei, and S. Z. Li, “Age estimation by multi-scale convolutional network,” in Asian Conference on Computer Vision, 2015, pp. 144–158.
-  I. Huerta, C. Fernández, C. Segura, J. Hernando, and A. Prati, “A deep analysis on age estimation,” Pattern Recognition Letters, vol. 68, pp. 239–249, 2015.
-  R. Rothe, R. Timofte, and L. Gool, “DEX: Deep EXpectation of apparent age from a single image,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2015, pp. 252–257.
-  R. Rothe, R. Timofte, and L. Van Gool, “Deep expectation of real and apparent age from a single image without facial landmarks,” International Journal of Computer Vision, pp. 1–14, 2016, doi:10.1007/s11263-016-0940-36.
-  N. Gourier, D. Hall, and J. L. Crowley, “Estimating face orientation from robust detection of salient facial structures,” in FG Net Workshop on Visual Observation of Deictic Gestures, 2004, pp. 1–9.
-  B. Yin, Y. Sun, C. Wang, and Y. Ge, “BJUT-3D large scale 3D face database and information processing,” Journal of Computer Research and Development, vol. 46, no. 6, pp. 1009–1018, 2009.
-  M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof, “Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2011, pp. 2144–2151.
-  D. F. Dementhon and L. S. Davis, “Model-based object pose in 25 lines of code,” International Journal of Computer Vision, vol. 15, no. 1–2, pp. 123–141, 1995.
-  K. Sundararajan and D. Woodard, “Head pose estimation in the wild using approximate view manifolds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, pp. 50–58.
-  Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “CNN: single-label to multi-label,” CoRR, abs:1406.5726, 2014.
M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. Torr, “BING: binarized normed gradients for objectness estimation at 300fps,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3286–3293.
-  C. L. Zitnick and P. Dollár, “Edge boxes: Locating object proposals from edges,” in European Conference on Computer Vision, 2014, pp. 391–405.
-  H. Yang, J. T. Zhou, Y. Zhang, B.-B. Gao, J. Wu, and J. Cai, “Exploit bounding box annotations for multi-label object recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 280–288.
-  Y. Gong, Y. Jia, T. Leung, A. Toshev, and S. Ioffe, “Deep convolutional ranking for multilabel image annotation,” CoRR, abs:1312.4894, 2013.
-  Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan, “HCP: A flexible CNN framework for multi-label image classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 9, pp. 1901–1907, 2015.
-  J. Dong, W. Xia, Q. Chen, J. Feng, Z. Huang, and S. Yan, “Subcategory-aware object classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 827–834.
-  Z. Song, Q. Chen, Z. Huang, Y. Hua, and S. Yan, “Contextualizing object detection and classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 1585–1592.
-  M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1717–1724.
-  J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888–905, 2000.
-  H. Lai, P. Yan, X. Shu, Y. Wei, and S. Yan, “Instance-Aware hashing for multi-label image retrieval,” IEEE Transactions on Image Processing, vol. 25, no. 6, pp. 2469–2479, 2016.
R.-W. Zhao, J. Li, Y. Chen, J.-M. Liu, Y.-G. Jiang, and X. Xue, “Regional gating neural networks for multi-label image classification,” inProceedings of the British Machine Vision Conference, vol. 6, 2016.
-  B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, “Semantic contours from inverse detectors,” in Proceedings of the IEEE International Conference on Computer Vision, 2011, pp. 991–998.
-  P. Krähenbühl and V. Koltun, “Efficient inference in fully connected CRFs with gaussian edge potentials,” in Advances in Neural Information Processing Systems, 2011, pp. 109–117.
-  L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected CRFs,” in Proceedings of International Conference on Learning Representations, 2015.
-  L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.