1 Introduction
Deep learning methods, although achieving excellent accuracy on a variety of vision tasks, are also notorious for their dependency on large training sets with precise annotations. The reality is that, however, this is a difficulttosatisfy requirement. Instead, a small training set with precise annotations or a large training set with noisy labels are easy to obtain (, through manual labeling or image search engines, respectively). Hence, learning from data with noisy labels has attracted increasing attention recently.
In spite of the many progresses made in recent years, the area of learning from noisy labels suffers from two fundamental drawbacks: improper dataset and improper metric for the evaluation of algorithms. In order to ablate various aspects of new algorithms, noisy CIFAR datasets [28] were created by adding noise to the classic CIFAR [9] datasets. A realworld data, Clothing1M [33], was used to evaluate realworld performance. And, the accuracy was used in both settings as the performance metric.
Both drawbacks are connected to two improper assumptions implicitly assumed in most previous research: an incorrect closedworld label noise assumption and an incorrect balanced (clean) training set and test set assumption.
One implicit assumption in existing algorithms is that the label noise is in the closedworld (or closed noise). For example, the noisy CIFAR datasets [28] only contain closed noise; and DivideMix [13]
treated examples that the model were not confident about as unlabeled ones, and subsequently used a semisupervised learning method
[2]—which mostly assumed that these unlabeled examples come from only these categories that we are interested in (, in the closedworld). But this assumption does not fit the realworld. In the realworld Clothing1M [33] dataset, the ‘jacket’ category contains lots of images of jeans (more than 50%!), which is clearly a label noise in the openworld (or open noise, not in the 14 categories of Clothing1M). In fact, collecting images with noisy labels from search engines will inevitably contain open noise, and it is natural to deduce that ignoring open noise will surely lead to inferior results.Another implicit previous assumption is that the training set (or its subset of those examples with clean labels) is balanced. The noisy CIFAR datasets [28]
are balanced (modulo slight sampling variances) in terms of those training examples with clean labels. But, because of the very existence of noisy labels, the clean labels will
inevitably be imbalanced or even longtailed. The realworld Clothing1M [33] dataset even has an imbalanced test set! As is widely known in both vision and learning, accuracy is a poor metric when the test set is imbalanced or longtailed, which will bias against underrepresented categories with few examples and make evaluation results less trustworthy [21].The drawbacks and assumptions are foes for developing, evaluating, comparing and understanding algorithms for learning from noisy labeled examples, and have implicitly affected most (if not all) previous efforts in this area. Hence, in this paper, we defend against these enemies by convening friends for learning from noisy labels:

We create noisy CIFAR datasets that contain both open and closed noise, whose construction process is more scientific than that for existing ones;

We show that mean average precision (mAP) is more suitable for evaluation, and advocate its adoption in this area;

We show that selfsupervised learning and warmup are effective in this task, but both require some special handling;

We propose an improved PENCIL [34], where our instance filtering process is effective when noise exists and the revised PENCIL is good at correcting closed noisy labels.
In short, by separating friends and foes, this paper contributes new datasets which are more suitable than existing ones, adopts and advocates a new evaluation metric as the replacement of the current accuracy metric, and proposes a method (called “F&F”) that achieves superior results than existing methods.
2 Related Works
First, we briefly review related works on learning from noisy labeled examples. Semi and selfsupervised learning are also discussed, as they are important tools in this area.
2.1 Learning with noisy labels
Most existing methods for training DNNs with noisy labels adopted at least one of the three approaches: to devise a loss function that is robust to noisy labels, to overcome overfitting via regularization, or to develop a process to relabel / correct the samples so that the data fed into a model can be less noisy.
PENCIL [34] and SCE [31] demonstrated that reversed KLdivergence, as the loss function, prevents a model from fitting corrupted labels by giving a small gradient to an example when the model and the label strongly disagree. [18]
showed that a simple variant of gradient clipping can achieve the same goal. In contrast, MWNet
[27] fitted a mapping from loss value to sample weight used in the loss function instead of relying on a predefined scheme. Semisupervised learning methods like DivideMix [13] can also be deemed a variant of reweighting.AIR [1] designed a regularizer that encourages the model to learn from samples with similar representations to the established. [23] used negative entropy regularization to prevent the model from being too confident and overfitting the samples with corrupted labels. ELR [16] proposed a simpler regularizer from an insight similar to AIR [1] and used it in the earlylearning stage.
Previous works tried to model the noise with the help of a few clean samples to perform label correction [29, 15, 11]. [28] proposed a framework to jointly optimize a model and pseudolabels for each sample without access to any clean samples. On top of the framework, PENCIL [34]
adopted a robust loss and learned the pseudolabels with stochastic gradient descent.
2.2 Semisupervised learning
Semisupervised learning deals with datasets where only a fraction of the samples are labeled. Most modern methods make use of the unlabeled data via regularization. Two kinds of regularization are the most commonly used: consistencybased regularization [10] and entropy minimization [5]. MixMatch [2] proposed a holistic framework by combining these two approaches with MixUp [35].
2.3 Selfsupervised learning
Selfsupervised learning aims at learning a mapping from raw data to representation by exploiting the intrinsic structure of the data without resorting to labels. Various pretext tasks have been proposed for selfsupervised learning on images, including Denoising Autoencoder, missing patches prediction
[30], solving Jigsaw puzzles [20], angle of rotation prediction [4], and more recently, determining if two augmented views come from the same original image [6, 3].3 F&F: Datasets and Evaluation
As aforementioned, existing noisy label learning datasets and the evaluation metric are both questionable. In this section, we will propose alternative ones for analyzing and validating algorithms.
3.1 Proposing the nCIFAR datasets
Two origins may produce images with noisy labels. First, when we use search engines and keywords to collect a dataset, like Clothing1M [33] and WebVision [14], both closedworld and openworld label noise exist. The datasets will also be imbalanced or longtailed—the returned list of images for a common object will definitely be much longer than that for a rare object. Second, there are domains that are inherently difficult, where even labels manually provided by experts will be noisy, , in medical imaging. In this scenario, open noise, closed noise and imbalance all naturally exist. Furthermore, label noise (both open and closed) often happens among categories that share certain degrees of similarity rather than occurring randomly.
We want the datasets we use to mimic realworld data, such that the proposed algorithms can be validly evaluated. However, the current noisy CIFAR datasets [28] pose opposite properties: there are only closedworld noise (, in noisy CIFAR10, all images come from the original 10 categories in CIFAR10); the subset of images with clean samples in the symmetric versions is balanced; label noise in the symmetric versions is uniform (, regardless of category similarity), and label noise in the asymmetric versions for CIFAR100 is sequential in the category order (, regardless of similarity and ad hoc). There is one work [25] trying to add openworld noise into CIFAR10, however, CIFAR10 is relatively simple and less effective in algorithm evaluation. Furthermore, [25] did not take into account the category similarity and imbalance requirement. Hence, in order to evaluate algorithm effectiveness more reliably, we propose a principled and flexible approach to create new noisy CIFAR datasets (which we call nCIFAR) that satisfy all the above requirements.
To incorporate openworld noise, we use CIFAR100 as the source dataset to inject image into nCIFAR10, and Tiny ImageNet
[12](200 categories) to add openworld noise to nCIFAR100. Note that the categories in both pairs do not overlap. However, there are categories who share conceptual or visual similarity, which can either benefit or confuse the learning algorithms, depending on how they handle these noise images.
We can also find the semantic similarity between two categories by using word embeddings. For CIFAR categories, we extract the GloVe embedding vectors
[22](which are 50 dimensional) as their representation. For a Tiny ImageNet category / synset, we average the GloVe vectors of all the descriptive words in this synset. Because the word embedding encodes semantics, we can effectively use the cosine similarity between two categories’ representation vectors to compute the similarity between them.
In our nCIFAR, a dataset is named in the convention “nCIFAR10” or “nCIFAR100” depending on whether CIFAR10 or CIFAR100 is the base, where and denote the percentage of open and closed noise rates, respectively. By specifying different and , we can generate datasets with different noise levels flexibly. Suppose the base dataset has classes ( or ) and the open noise source has classes ( or ). We use a probabilistic sampling approach to generate them.

First, we generate two similarity matrices. The closedworld similarity matrix is of size , and the openworld matrix is of . One entry or is computed using the cosine similarity.

Second, since we often pick the top images returned by search engines in the real world, we generate ( for nCIFAR10 and for nCIFAR100, ) images in the generated dataset for every category. There will be clean examples in all categories, openworld noise examples, and closedworld noise images in total.

To sample open noise images, we convert the similarity matrix by a softmax transformation with temperature , then followed by a columnwise normalization, ,
Because of the subsequent columnwise normalization processing, we do not need to complete the full softmax operation, and the exponential operation alone is enough. Then, the th column sums up to 1 and forms a valid distribution. Following this distribution, we sample examples from the open noise source categories without replacement, and they form the open noise examples for the th category in the base dataset. We use . Images are resized to .

Similarly, to sample clean and closed noise samples, we transform to by softmax with temperature and a subsequent normalization:
Note that different leads to different proportion of sampled clean examples, and the relationship between them (value of and the number of clean examples) is monotonic, which is easily proved. Hence, we can use a binary search to find the value that leads to clean samples in total. Because the number of closedworld examples (clean plus closed noise) is , the binary search terminates when the average of the diagonal entries of sum up to . Also note that the binary search process has only trivial computational cost. Then, we use to sample examples from all base categories to the th category according to with the proper , which include both clean and closed noise samples.

Both training sets (nCIFAR10 and nCIFAR100) have size . The test sets are the original CIFAR10 / 100 test sets, which are balanced.
Note that the training datasets generated in our approach are imbalanced (different categories have different number of clean and closed noise samples), and categories with higher similarity scores are more likely to generate noisy samples. These properties make our nCIFAR datasets suitable in learning with noisy labeled examples.^{1}^{1}1It is desirable to have different numbers of open noise in different categories. However, since we have no clue on how open set noise distributes in general, we sample open noise in every category.
3.2 Advocating the mAP evaluation metric
Although imbalance is critical in making a good dataset, it causes difficulty in both training and evaluation. Suppose in an imbalanced binary recognition problem, the ratio between number of examples in two categories A and B is 99:1 in both training and test sets. Then, when accuracy is used as the evaluation metric, a model tends to predict all examples as A, and achieving 99% accuracy. The minority class, B, is entirely sacrificed—hence, high accuracy does not necessarily mean good model in an imbalanced task.
A realworld dataset with label noise is imbalanced in the training set, and very likely also imbalanced in the test set, such as in Clothing1M [33]. Fig. 1 compares two models (Aug for AugDesc [19] and F&F for the proposed method) on Clothing1M (row 6 and 8 in Table 3, respectively). They have almost identical accuracy (74.93% for Aug and 74.92% for F&F). Although we do not know the number of clean training examples in each category, it is reasonable to rank the categories by their size in the test set in the ascending order. Let denote the number of correctly predicted examples of Aug minus that of F&F in the th ranked category, and the cumulative sum of (, ).
Fig. 1 plots , and an obvious trend shown in Fig. 1 is that the curve is below the zero line in the left half (categories with fewer examples) and above it in the right half (categories with more examples). That is, Aug sacrificed accuracy of categories with fewer (clean) examples to boost the accuracy of other categories with more examples. Again, a higher accuracy value does not necessarily mean a better model in an imbalanced dataset.
Following practices in evaluating imbalanced datasets, we advocate that we must use mean average precision (mAP) to evaluate models for noisy label learning, especially when the test set is imbalanced. mAP is not affected by the imbalance property [21]. In this example, F&F has an mAP of 75.22%, outperforming Aug (73.25%) by a large margin. As shown in Fig. 1, F&F indeed acted more robustly than Aug across different categories.
4 F&F: Techniques
Unlike earlier methods that often focus on one technical aspect (, loss function), recent methods often build systems that integrate multiple techniques. For example, DivideMix [13] integrates techniques like semisupervised learning, cotraining, etc. Indeed, learning from noisy labels is a complex task, and has a strong systems flavor. In this section, we propose a few techniques and build an F&F system, which is based on DivideMix [13] and PENCIL [34].
4.1 Selfsupervised backbones & freezed warmup
backbone  epoch  accuracy  

warmup policy  random  0 + 5  66.22 
0 + 30  67.94  
0 + 35  67.72  
ssl1b  0 + 5  67.42  
30 + 5  68.22  
ssl2b  0 + 5  68.34  
30 + 0  67.59  
30 + 5  68.38 
Selfsupervised learning has been shown to learn good representation without labels, a property that is conceptually particularly useful in our task (by throwing away all labels). Our first improvement (friend) is to use selfsupervised learning algorithms (SimCLR [3] for nCIFAR and MoCo [6] for Clothing1M) to learn initial backbone networks.
Although selfsupervised learning has been applied to this domain [17], we argue that in complex methods like DivideMix which have 2 (or more) branches, diversity is in fact essential—we have to use two different selfsupervised backbones to initialize them.
Another improvement (friend) is the warmup of backbones. Before starting to train the models, a few warmup epochs are often needed, which updates all parameters in DivideMix. We argue that it is prone to overfitting, and propose to first warmup epochs by freezing the backbone, and then normal epochs that updates all parameters.
As the ablative experiments on nCIFAR1000.30.3 shown in Table 1, both selfsupervised learning (our diverse one in particular) and freezed warmup are friends for learning with noisy labels.^{2}^{2}2The nCIFAR test set is balanced, hence either accuracy or mAP can be used here.
Selfsupervised backbones, freezed and normal warmup, plus a smaller number (compared to the original DivideMix) of epochs of DivideMix training form the stage 1 of F&F.
4.2 Improved PENCIL & instance filtering
The first stage’s output model initializes that of the second stage in F&F. In the first stage, DivideMix treats instances with large loss (, less confident ones) as unlabeled examples and adopts semisupervised learning on them. However, label correction learns or relabels some closedworld noise instances, and can significantly improve the accuracy [34, 36, 17]. Hence, label correction is another friend.
We adopt the PENCIL framework [34]
in stage 2. However, it is difficult to tune the hyperparameters in PENCIL and it is rather complex. We simplify it and improve it.
Consider a class classification problem. One example has a hard label
in the onehot encoding:
and , where is a vector of all 1’s. Note that the label is noisy, and hence can be wrong.In PENCIL, a soft label is defined, where
is the estimated probability of
in the th class. That is, forms a valid distribution. Note that is not constant, but updated by SGD (, being corrected). For , the model predicts (after softmax) another distribution, and the KullbackLeibler divergence
is PENCIL’s first loss term.Its second loss term measures the compatibility between hard and soft labels, as .
The third PENCIL loss term is the entropy of the prediction, .
Hence, the overall PENCIL loss is
(1) 
where and are hyperparameters.
PENCIL is sensitive to the hyperparameters: , , and in particular the learning rate to update . When the learning rate for the backbone is 0.01, the learning rate for updating varies in different datasets (which can be as large as 10000), and it is difficult to find a proper value for it. When these hyperparameter values are not suitable, the PENCIL training process often breaks down.
In our F&F, we first remove the compatibility (second) loss term (, ), and fix . Then, after removing these two hyperparameters, the loss is greatly simplified after simple algebraic manipulation: It becomes a reverse crossentropy loss:
(2) 
In the original PENCIL, the gradient for the soft labels () involves , which is very unstable. The gradient in F&F becomes simple: , which greatly stabilizes its updating, and we do not need to tune the learning rate for updating .
The final friend that F&F counts on is instance filtering, which has been shown to be effective [26, 24, 32, 17] in the presence of open noise. In PENCIL (and our improved version), closed noisy instances are in fact useful when the model is confident on them. Hence, we believe that removing lowconfidence examples in both open and closed noise will be helpful.
When the prediction
has a large entropy, its confidence is low, and vice versa. In stage 2, we compute the prediction’s entropy of all training examples, and fit a Gaussian mixture model (GMM) with 2 components—the first component is considered as high and the second lowconfidence. We can easily compute the posterior probability of an example belonging to the confident mode, and keep this example if the probability is larger than a threshold
. The rest examples are not used. Note that this filtering is similar to that in DivideMix. However, examples with lowconfidence are treated as unlabeled and still used in DivideMix, while we ignore them in F&F. This instance filtering process is repeated for each of the 2 branches independently in every epoch. That is, different subsets are filtered per epoch.Note that when , there is no instance filtering at all. We ablated on nCIFAR1000.30.3. From Table 2, it is clear that removing examples that are not confident () always outperforms using all examples (). That is, instance filtering is an effective approach.
clean  open  closed  accuracy  

✓  ✓  ✓  0  69.04 0.25 
✓  ✓  ✓  0.5  69.69 0.22 
✓  ✓  0  69.31 0.24  
✓  ✓  0.5  69.80 0.28  
✓  ✓  0  69.35 0.27  
✓  ✓  0.5  69.90 0.08  
✓  n/a  71.65 0.06  
✓  ✓  ✓  n/a  66.61 0.26 
Because we have groundtruth clean labels in nCIFAR for the entire dataset, we can manually remove either open noise or closed noise from the training set in stage 2 of F&F for ablation. When open noise exists, instance filtering improves roughly 0.5–0.6% in the first two groups in Table 2
and the standard deviations are all
. In the third group, closed noise exists but open noise does not, where instance filtering improves 0.65% and the standard deviation is much smaller (only 0.08), , more stable. And, the sixth row has the highest accuracy, where open noise has been removed and instances have been filtered.In the last group, we also trained networks using only the classic cross entropy loss (, without noise handling). When only the clean samples are used, its accuracy is 71.65%. When all examples are used (including both types of noisy examples), its accuracy is 66.61%.
These observations indicate that

Closed noise is more friendly than open noise, because many closed noisy labels will be corrected by our improved PENCIL (, becoming clean).

When we use all samples (like in DivideMix), , without instance filtering (), both closed and open noise will be harmful.

When we only keep confident samples (), even open noise will not significantly impoverish the model.

Both instance filtering and label correction are effective. They bring close to 1 percentage point improvement when combined in F&F (69.04% vs. 69.90%).

F&F has significantly bridged the accuracy gap caused by noisy labels: 66.64% when without noise handling, 69.90% for F&F, and 71.65% for clean data.
Finally, the two branches in stage 2 are independent of each other. We are in fact training two different networks separately and then average their predictions.
4.3 The F&F framework
Pseudocode for the overall F&F framework is shown in Algorithm 1. For more details, we will publish the code of F&F.
It is also worth noting that the second stage runs much faster than the first stage. We recorded the timing statistics on the nCIFAR1000.20.2 dataset, in which the first stage took 9.53 hours, while stage 2 only took 0.27 hours.
That is, the combination of instance filtering and label correction in stage 2, although only used 2.9% running time of that of stage 1, can lead to considerable improvements in accuracy.
5 Experimental Settings and Results
In this section, we first describe the settings in our experiments, then present experimental results on both the proposed nCIFAR datasets and the realworld Clothing1M dataset. Although the noisy CIFAR datasets from [28] are inappropriate, we also report F&F’s results on them, in order to facilitate comparisons with existing methods. We will publish the code to generate the proposed nCIFAR datasets, and the exact versions used in our experiments will be published, too. We will also publish code of the F&F system.
5.1 Detailed data and algorithmic settings
Datasets. In this paper, we use 5 versions of the proposed nCIFAR datasets: nCIFAR100.10.1, nCIFAR100.20.2, nCIFAR1000.10.1, nCIFAR1000.20.2 and nCIFAR1000.30.3. The last dataset has only 40% clean samples. nCIFAR10 are relatively easy, which are suitable for sanitycheck of algorithms. nCIFAR100 are more difficult, and are useful for evaluating, comparing, analyzing and understanding algorithms. Note that although we set the percentage of open noise the same as that of the closed noise, it is trivial to generate nCIFAR datasets with different ratios for them.
Baseline methods. In terms of baseline methods, we mainly compare with 2 existing methods: DivideMix [13] and AugDesc [19]. DivideMix is a strong method and many recent advances in learning from noisy labels are developed based on it. AugDesc is the current top performer.^{3}^{3}3https://paperswithcode.com/sota/imageclassificationonclothing1m
For both methods, we used the codes published by their respective authors. We use their hyperparameter settings without modifications.
F&F: shared. We update network parameters and soft labels using SGD with a momentum of 0.9. We set weight decay of network parameters to for CIFAR based datasets and for Clothing1M. We do not apply weight decay for updating soft labels.
F&F stage 2: Improved PENCIL. As aforementioned, the hyperparameter setting in F&F’s stage 2 is much simpler than that in the original PENCIL [34]. We used the same setting in all our experiments in this part. In the improved PENCIL training, we always used 0.5 as the initial learning rate to update soft labels . For other network parameters, the learning rate was for all CIFAR based datasets and for Clothing1M. Improved PENCIL always ran 20 epochs, and both learning rates were reduced by a factor of 10 midway (, after 10 epochs).
F&F stage 2: Instance filtering. Since we have shown the effectiveness of instance filtering in ablative studies (cf. Table 2), we always used in instance filtering. For all CIFAR based datasets, label correction with instance filtering was conducted on all the training samples. For Clothing1M, we sampled a subset containing about 260k images to apply label correction, following the original PENCIL [34]. Note that this subset is balanced when considering the noisy labels. However, if we have access to clean labels, this subset will not be balanced.
F&F stage 1: Selfsupervised learning. For noisy CIFAR from [28], we used SimCLR [3] to train PreActResNet18 [8] on CIFAR for 1000 epochs. Because these datasets share the same unlabeled images for CIFAR10 / 100 regardless of the noise type or noise ratio, we only need to train a pair of backbones with the original CIFAR10 / 100 datasets, and they can be shared among experiments on different noisy CIFAR datasets.
On our nCIFAR datasets, SimCLR (1000 epochs) was still used. However, because different nCIFAR10 or nCIFAR100 datasets have different sets of unlabeled training images, it is essential to train a pair of backbones for each nCIFAR dataset separately.
On Clothing1M, we used MoCo [6] to train two different backbones of ResNet50 [7], initialized with the official pretrained weights trained on ImageNet (800 epochs), then trained on Clothing1M for 200 epochs.
F&F stage 1: Warmup. For noisy CIFAR in [28], in warmup we trained the classifiers for 30 epochs with the backbones frozen. The learning rate was , and the batch size was 128. Because we only experimented with very high noise rate here, there was no normal warmup training in order to avoid overfitting.
For nCIFAR, we first train the classifiers for 30 epochs with the backbones frozen and then warmup the whole network for 5 epochs. The learning rate was , and the batch size was 128.
On Clothing1M, we sampled a (noisy) labelbalanced subset containing about 260k images, and trained the classifiers for 10 epochs with the backbones frozen. We used a learning rate of , and a batch size of 128.
F&F stage 1: DivideMix. On all datasets, the temperature of label sharpening and the threshold of smallloss sample filtering were both fixed to 0.5 in all experiments.
For noisy CIFAR in [28], the DivideMix training part in stage 1 used the same parameters as those in the original DivideMix.
For nCIFAR, the learning rate and batch size were the same as in warmup. , strength of the consistency loss in DivideMix was set to 100 for nCIFAR100, and 0 for nCIFAR10, following the original DivideMix. in mixup was set to 0.5. The networks were trained for 200 epochs with the learning rate reduced by a factor of 10 after half of the epochs.
On Clothing1M, we set to 0.2, and to 0 (which follows the original DivideMix). We trained the networks for 30 epochs. In each epoch we sampled 2000 (noisy) labelbalanced minibatches, and the batch size was 64. The learning rate was set to , and was reduced by a factor of 10 after 15 epochs.
For all these methods, we first sum the logits of the two branches, then apply a softmax transformation to predict the final results. When standard deviations are reported, we ran the experiments 5 times and report the mean accuracy or mAP, plus its standard deviation.
Selection criterion  Method  mAP  accuracy 

last  DivideMix  72.27  74.03 
AugDesc  72.92  75.18  
F&F stage1  72.39  73.83  
F&F stage2  75.31  74.89  
best accuracy  DivideMix  72.22  74.19 
AugDesc  73.25  74.93  
F&F stage1  73.75  74.54  
F&F stage2  75.22  74.92  
best mAP  DivideMix  73.15  74.33 
AugDesc  73.42  74.87  
F&F stage1  74.78  75.00  
F&F stage2  75.42  75.10 
Dataset  nCIFAR100.10.1  nCIFAR100.20.2  nCIFAR1000.10.1  nCIFAR1000.20.2  nCIFAR1000.30.3 
DivideMix  93.88 0.22  92.00 0.32  75.76 0.21  73.55 0.49  67.66 0.32 
F&F stage1  94.16 0.07  92.15 0.34  76.04 0.21  73.68 0.28  68.29 0.12 
F&F stage2  94.73 0.02  92.84 0.26  77.44 0.18  75.29 0.21  69.69 0.22 
CIFAR10  CIFAR100  

Method  Selection criterion  asym40%  sym90%  sym80%  sym90% 
best  91.16  61.21      
PENCIL [34]  last  91.01  60.80     
best  93.4  76.0  60.2  31.5  
DivideMix [13]  last  92.1  75.4  59.6  31.0 
best  94.6  91.9  66.4  41.2  
AugDesc [19]  last  94.3  91.8  66.1  40.9 
best  93.95 0.23  93.24 0.47  66.99 0.48  54.48 1.25  
F&F (Ours)  last  93.58 0.17  93.24 0.51  67.08 0.51  54.54 1.36 
5.2 Results on Clothing1M
We start reporting results from the realworld Clothing1M dataset, whose test set is imbalanced. Different selection criteria were used to select the model: the one at the last training epoch, and the one with the best accuracy or the best mAP on the validation set. Note that no validation data is required for the ‘last’ criterion. Results are in Table 3.
It is clear that F&F is the consistent winner in terms of mAP, the evaluation metric which we advocate because it does not sacrifice underrepresented categories. But, when accuracy is used, F&F is also a strong method. Even an incomplete F&F (stopping at stage 1) consistently outperforms DivideMix. These results show that our warmup and selfsupervised learning improvements in stage 1, and instance filtering and label correction in stage 2 are all effective.
Another interesting observation is that the ‘best mAP’ criterion acts betters than the ‘best accuracy’ one, which shows that accuracy is not an appropriate metric in imbalanced tasks from a different perspective.
5.3 Results on nCIFAR
On the proposed nCIFAR datasets, as shown in Table 4, the accuracy is used because its test set is balanced. We argue that a validation set is difficult to obtain in real world settings, and advocate using the last epoch’s model. Table 4 shows five time average of the accuracy of the last training epoch.
From Table 4, the stage 1 of F&F consistently outperforms DivideMix, and the complete F&F (stage 2) consistently beats its own stage 1.
Another interesting observation is that the complete F&F has smallest variance in general, while F&F stage 1 has smaller variance than DivideMix in most cases—meaning F&F is more robust than DivideMix.
One of the reasons that methods which learn from noisy labels require a validation set is that they are sometimes not robust, hence a validation set is needed to pick the optimal model from the many training epochs. A robust method, like F&F, suggests that we better fend off our dependency on the validation set, and directly use the last epoch’s model.
5.4 Results on noisy CIFAR from [28]
We have discussed the drawbacks of the noisy CIFAR dataset previously used in the community, and have advocated the adoption of our nCIFAR datasets. However, to provide a thorough comparison to previous methods, we also conducted experiments on them, with results presented in Table 5. We only experimented with the most difficult ones: 40% asymmetric and 90% symmetric noise on noisy CIFAR10, plus 80% and 90% symmetric noise on noisy CIFAR100 from [28].
First, F&F is still very robust. In fact, in three out of four cases, F&F’s ‘last’ outperforms ‘best’. Second, except on noisy CIFAR10 with 40% asymmetric noise, F&F is the clear winner. Third, in the two CIFAR100 symmetric noise cases, the winning margin of F&F is significant, even more than 10 percentage points with 90% symmetric noise.
For the last observation, our conjecture is that two factors lead to this large gap: selfsupervised learning (especially our diverse initialization strategy), and label correction.
6 Conclusions and Future Work
In this paper, we first advocated two changes to the area of learning from examples with noisy labels: the dataset and the evaluation metric. We analyzed the drawbacks of the current datasets and metric, then proposed replacements: our nCIFAR datasets and the mAP metric (at least for imbalanced test sets). Then, we proposed an F&F framework, which highlights four ‘friends’ with our improved techniques: diverse selfsupervised learning, freezed warmup, label correction and instance filtering. F&F has not only outperformed existing methods, but also has higher robustness—its last epoch model’s prediction has smaller variances.
The filed of learning from noisy labels shares similarity with learning from webly supervised data. In the future, we will migrate F&F to such datasets as WebVision [14]. Furthermore, learning from noisy labels in a multilabel setting is not only more challenging but also has wider utility in applications. We will investigate into this area by relaxing constraints on the soft labels in F&F.
References
 [1] Samaneh Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. Auxiliary image regularization for deep CNNs with noisy labels. In International Conference on Learning Representations, 2016.
 [2] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semisupervised learning. In Advances in Neural Information Processing Systems 32, pages 5050–5060, 2019.

[3]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.
A simple framework for contrastive learning of visual
representations.
In
International Conference on Machine Learning
, pages 1597–1607, 2020.  [4] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
 [5] Yves Grandvalet and Yoshua Bengio. Semisupervised learning by entropy minimization. In Conférence francophone sur l’apprentissage automatique, pages 281–296, 2005.

[6]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.
Momentum contrast for unsupervised visual representation learning.
In
IEEE/CVF Conference on Computer Vision and Pattern Recognition
, pages 9726–9735, 2020.  [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
 [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, volume 9908 of Lecture Notes in Computer Science, pages 630–645, 2016.
 [9] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.
 [10] Samuli Laine and Timo Aila. Temporal ensembling for semisupervised learning. In International Conference on Learning Representations, 2017.

[11]
KuangHuei Lee, Xiaodong He, Lei Zhang, and Linjun Yang.
Cleannet: Transfer learning for scalable image classifier training with label noise.
In IEEE Conference on Computer Vision and Pattern Recognition, pages 5447–5456, 2018.  [12] FeiFei Li, Andrej Karpathy, and Justin Johnson. Tiny ImageNet challenge, 2017.
 [13] Junnan Li, Richard Socher, and Steven C. H. Hoi. DivideMix: Learning with noisy labels as semisupervised learning. In International Conference on Learning Representations, 2020.
 [14] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. WebVision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017.
 [15] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and LiJia Li. Learning from noisy labels with distillation. In IEEE International Conference on Computer Vision, pages 1928–1936, 2017.
 [16] Sheng Liu, Jonathan NilesWeed, Narges Razavian, and Carlos FernandezGranda. Earlylearning regularization prevents memorization of noisy labels. In Advances in Neural Information Processing Systems 33, 2020.
 [17] Devraj Mandal, Shrisha Bharadwaj, and Soma Biswas. A novel selfsupervised relabeling approach for training with noisy labels. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2020.
 [18] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2020.
 [19] Kento Nishi, Yi Ding, Alex Rich, and Tobias Höllerer. Augmentation strategies for learning with noisy labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
 [20] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving Jigsaw puzzles. In European Conference on Computer Vision, volume 9910 of Lecture Notes in Computer Science, pages 69–84, 2016.
 [21] Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with longtail distribution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 864–873, 2016.

[22]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning.
GloVe: Global vectors for word representation.
In
Empirical Methods in Natural Language Processing
, pages 1532–1543, 2014. 
[23]
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E.
Hinton.
Regularizing neural networks by penalizing confident output distributions.
In International Conference on Learning Representations, 2017.  [24] Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Advances in Neural Information Processing Systems 33, 2020.
 [25] Ragav Sachdeva, Filipe R. Cordeiro, Vasileios Belagiannis, Ian D. Reid, and Gustavo Carneiro. EvidentialMix: Learning with combined openset and closedset noisy labels. In IEEE Winter Conference on Applications of Computer Vision, 2021.
 [26] Karishma Sharma, Pinar Donmez, Enming Luo, Yan Liu, and I. Zeki Yalniz. NoiseRank: Unsupervised label noise reduction with dependence models. In European Conference on Computer Vision, volume 12372 of Lecture Notes in Computer Science, pages 737–753, 2020.
 [27] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. MetaWeightNet: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems 32, pages 1917–1928, 2019.
 [28] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5552–5560, 2018.
 [29] Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Advances in Neural Information Processing Systems 30, pages 5596–5605, 2017.

[30]
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and PierreAntoine Manzagol.
Extracting and composing robust features with denoising autoencoders.
In International Conference on Machine Learning, pages 1096–1103, 2008.  [31] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE/CVF International Conference on Computer Vision, pages 322–330, 2019.
 [32] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N. Metaxas, and Chao Chen. A topological filter for learning with label noise. In Advances in Neural Information Processing Systems 33, 2020.
 [33] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2691–2699, 2015.
 [34] Kun Yi and Jianxin Wu. Probabilistic endtoend noise correction for learning with noisy labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7017–7025, 2019.
 [35] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David LopezPaz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.

[36]
Guoqing Zheng, Ahmed H. Awadallah, and Susan Dumais.
Meta label correction for noisy label learning.
In
AAAI Conference on Artificial Intelligence
, 2021.
Appendix A Proof of the monotonicity of the closed noise rate with respect to
As is mentioned in the paper, one entry in the closedworld similarity matrix is calculated using the cosine similarity between the representation vectors of the th and th categories for . The cosine similarity between one vector and itself equals to 1, , for any . Different categories always have different representation vectors, and hence if . Thus, we can safely assume that holds for all .
Now consider the set of closedworld examples generated in the proposed nCIFAR dataset with hyperparameter , which includes both instances with clean and closedworld noisy labels. We denote the percentage of closedworld noisy labels as (, the number of closed noise divided by the number of all closedworld examples). According to the approach that generates examples in nCIFAR, we have
(3) 
where
(4) 
for . Note that we want in nCIFAR.
Then,
(5)  
(6)  
(7)  
(8)  
(9) 
That is, the closed noise rate is monotonically increasing with respect to the temperature . Therefore, it is reasonable to apply binary search to efficiently find the exact value that leads to .
Appendix B Details of the loss function in our improved PENCIL
It is known that PENCIL suffers from a seemingly unreasonable hyperparameter value, which is the learning rate used to update the soft labels . In F&F, the key technique to solve this issue is: to update soft labels with a modified loss function, and to update soft labels and network parameters separately.
As is shown in the paper, the loss related to one sample is defined as
(10) 
Both network parameters and soft labels are updated using SGD. Let us assume that the size of a minibatch is denoted by . For the th example in this minibatch, the example, the network’s prediction and its soft label are denoted by , and , respectively. Note that the th dimension of the vector is denoted as .
We follow the conventional approach, and use the average gradient from a minibatch to update network parameters, because the same set of network parameters are shared among all the samples. Hence, the loss term responsible for updating network parameters is
(11) 
When it turns to update the soft labels, the situation is completely different: one sample has its own soft label ; and, is only corresponding to and it is not related to other or when . Hence, the gradient must not be averaged within a minibatch.
In PENCIL, the learning rate to update soft labels has to be enlarged to offset the effect of the batch size, namely the coefficient . In other words, this hyperparameter is in fact dependent on the minibatch size in PENCIL. This explains why this hyperparameter has unusual values like 3000 or 10000 in PENCIL.
We use the following loss term to update soft labels:
(12) 
By removing the term in the loss function, we can use the same learning rate to update soft labels in all our F&F experiments, , now it becomes very robust.
Technically, we detach the gradient to network parameters when updating soft labels, and vice versa.
Comments
There are no comments yet.