Friends and Foes in Learning from Noisy Labels

by   Yifan Zhou, et al.
Nanjing University

Learning from examples with noisy labels has attracted increasing attention recently. But, this paper will show that the commonly used CIFAR-based datasets and the accuracy evaluation metric used in the literature are both inappropriate in this context. An alternative valid evaluation metric and new datasets are proposed in this paper to promote proper research and evaluation in this area. Then, friends and foes are identified from existing methods as technical components that are either beneficial or detrimental to deep learning from noisy labeled examples, respectively, and this paper improves and combines technical components from the friends category, including self-supervised learning, new warmup strategy, instance filtering and label correction. The resulting F F method significantly outperforms existing methods on the proposed nCIFAR datasets and the real-world Clothing1M dataset.



There are no comments yet.


page 1

page 2

page 3

page 4


Robust Long-Tailed Learning under Label Noise

Long-tailed learning has attracted much attention recently, with the goa...

Skeptical Deep Learning with Distribution Correction

Recently deep neural networks have been successfully used for various cl...

Learning from Noisy Labels with Deep Neural Networks: A Survey

Deep learning has achieved remarkable success in numerous domains with h...

Continual Learning on Noisy Data Streams via Self-Purified Replay

Continually learning in the real world must overcome many challenges, am...

Learning From Long-Tailed Data With Noisy Labels

Class imbalance and noisy labels are the norm rather than the exception ...

Noise-Resistant Deep Metric Learning with Probabilistic Instance Filtering

Noisy labels are commonly found in real-world data, which cause performa...

Data Shapley Value for Handling Noisy Labels: An application in Screening COVID-19 Pneumonia from Chest CT Scans

A long-standing challenge of deep learning models involves how to handle...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Deep learning methods, although achieving excellent accuracy on a variety of vision tasks, are also notorious for their dependency on large training sets with precise annotations. The reality is that, however, this is a difficult-to-satisfy requirement. Instead, a small training set with precise annotations or a large training set with noisy labels are easy to obtain (, through manual labeling or image search engines, respectively). Hence, learning from data with noisy labels has attracted increasing attention recently.

In spite of the many progresses made in recent years, the area of learning from noisy labels suffers from two fundamental drawbacks: improper dataset and improper metric for the evaluation of algorithms. In order to ablate various aspects of new algorithms, noisy CIFAR datasets [28] were created by adding noise to the classic CIFAR [9] datasets. A real-world data, Clothing1M [33], was used to evaluate real-world performance. And, the accuracy was used in both settings as the performance metric.

Both drawbacks are connected to two improper assumptions implicitly assumed in most previous research: an incorrect closed-world label noise assumption and an incorrect balanced (clean) training set and test set assumption.

One implicit assumption in existing algorithms is that the label noise is in the closed-world (or closed noise). For example, the noisy CIFAR datasets [28] only contain closed noise; and DivideMix [13]

treated examples that the model were not confident about as unlabeled ones, and subsequently used a semi-supervised learning method 

[2]—which mostly assumed that these unlabeled examples come from only these categories that we are interested in (, in the closed-world). But this assumption does not fit the real-world. In the real-world Clothing1M [33] dataset, the ‘jacket’ category contains lots of images of jeans (more than 50%!), which is clearly a label noise in the open-world (or open noise, not in the 14 categories of Clothing1M). In fact, collecting images with noisy labels from search engines will inevitably contain open noise, and it is natural to deduce that ignoring open noise will surely lead to inferior results.

Another implicit previous assumption is that the training set (or its subset of those examples with clean labels) is balanced. The noisy CIFAR datasets [28]

are balanced (modulo slight sampling variances) in terms of those training examples with clean labels. But, because of the very existence of noisy labels, the clean labels will

inevitably be imbalanced or even long-tailed. The real-world Clothing1M [33] dataset even has an imbalanced test set! As is widely known in both vision and learning, accuracy is a poor metric when the test set is imbalanced or long-tailed, which will bias against under-represented categories with few examples and make evaluation results less trustworthy [21].

The drawbacks and assumptions are foes for developing, evaluating, comparing and understanding algorithms for learning from noisy labeled examples, and have implicitly affected most (if not all) previous efforts in this area. Hence, in this paper, we defend against these enemies by convening friends for learning from noisy labels:

  • We create noisy CIFAR datasets that contain both open and closed noise, whose construction process is more scientific than that for existing ones;

  • We show that mean average precision (mAP) is more suitable for evaluation, and advocate its adoption in this area;

  • We show that self-supervised learning and warmup are effective in this task, but both require some special handling;

  • We propose an improved PENCIL [34], where our instance filtering process is effective when noise exists and the revised PENCIL is good at correcting closed noisy labels.

In short, by separating friends and foes, this paper contributes new datasets which are more suitable than existing ones, adopts and advocates a new evaluation metric as the replacement of the current accuracy metric, and proposes a method (called “F&F”) that achieves superior results than existing methods.

2 Related Works

First, we briefly review related works on learning from noisy labeled examples. Semi- and self-supervised learning are also discussed, as they are important tools in this area.

2.1 Learning with noisy labels

Most existing methods for training DNNs with noisy labels adopted at least one of the three approaches: to devise a loss function that is robust to noisy labels, to overcome overfitting via regularization, or to develop a process to relabel / correct the samples so that the data fed into a model can be less noisy.

PENCIL [34] and SCE [31] demonstrated that reversed KL-divergence, as the loss function, prevents a model from fitting corrupted labels by giving a small gradient to an example when the model and the label strongly disagree. [18]

showed that a simple variant of gradient clipping can achieve the same goal. In contrast, MWNet 

[27] fitted a mapping from loss value to sample weight used in the loss function instead of relying on a predefined scheme. Semi-supervised learning methods like DivideMix [13] can also be deemed a variant of reweighting.

AIR [1] designed a regularizer that encourages the model to learn from samples with similar representations to the established. [23] used negative entropy regularization to prevent the model from being too confident and overfitting the samples with corrupted labels. ELR [16] proposed a simpler regularizer from an insight similar to AIR [1] and used it in the early-learning stage.

Previous works tried to model the noise with the help of a few clean samples to perform label correction [29, 15, 11]. [28] proposed a framework to jointly optimize a model and pseudo-labels for each sample without access to any clean samples. On top of the framework, PENCIL [34]

adopted a robust loss and learned the pseudo-labels with stochastic gradient descent.

2.2 Semi-supervised learning

Semi-supervised learning deals with datasets where only a fraction of the samples are labeled. Most modern methods make use of the unlabeled data via regularization. Two kinds of regularization are the most commonly used: consistency-based regularization [10] and entropy minimization [5]. MixMatch [2] proposed a holistic framework by combining these two approaches with MixUp [35].

2.3 Self-supervised learning

Self-supervised learning aims at learning a mapping from raw data to representation by exploiting the intrinsic structure of the data without resorting to labels. Various pretext tasks have been proposed for self-supervised learning on images, including Denoising Autoencoder, missing patches prediction 

[30], solving Jigsaw puzzles [20], angle of rotation prediction [4], and more recently, determining if two augmented views come from the same original image [6, 3].

3 F&F: Datasets and Evaluation

As aforementioned, existing noisy label learning datasets and the evaluation metric are both questionable. In this section, we will propose alternative ones for analyzing and validating algorithms.

3.1 Proposing the nCIFAR datasets

Two origins may produce images with noisy labels. First, when we use search engines and keywords to collect a dataset, like Clothing1M [33] and WebVision [14], both closed-world and open-world label noise exist. The datasets will also be imbalanced or long-tailed—the returned list of images for a common object will definitely be much longer than that for a rare object. Second, there are domains that are inherently difficult, where even labels manually provided by experts will be noisy, , in medical imaging. In this scenario, open noise, closed noise and imbalance all naturally exist. Furthermore, label noise (both open and closed) often happens among categories that share certain degrees of similarity rather than occurring randomly.

We want the datasets we use to mimic real-world data, such that the proposed algorithms can be validly evaluated. However, the current noisy CIFAR datasets [28] pose opposite properties: there are only closed-world noise (, in noisy CIFAR10, all images come from the original 10 categories in CIFAR10); the subset of images with clean samples in the symmetric versions is balanced; label noise in the symmetric versions is uniform (, regardless of category similarity), and label noise in the asymmetric versions for CIFAR100 is sequential in the category order (, regardless of similarity and ad hoc). There is one work [25] trying to add open-world noise into CIFAR10, however, CIFAR10 is relatively simple and less effective in algorithm evaluation. Furthermore, [25] did not take into account the category similarity and imbalance requirement. Hence, in order to evaluate algorithm effectiveness more reliably, we propose a principled and flexible approach to create new noisy CIFAR datasets (which we call nCIFAR) that satisfy all the above requirements.

To incorporate open-world noise, we use CIFAR100 as the source dataset to inject image into nCIFAR10, and Tiny ImageNet 


(200 categories) to add open-world noise to nCIFAR100. Note that the categories in both pairs do not overlap. However, there are categories who share conceptual or visual similarity, which can either benefit or confuse the learning algorithms, depending on how they handle these noise images.

We can also find the semantic similarity between two categories by using word embeddings. For CIFAR categories, we extract the GloVe embedding vectors 


(which are 50 dimensional) as their representation. For a Tiny ImageNet category / synset, we average the GloVe vectors of all the descriptive words in this synset. Because the word embedding encodes semantics, we can effectively use the cosine similarity between two categories’ representation vectors to compute the similarity between them.

In our nCIFAR, a dataset is named in the convention “nCIFAR10--” or “nCIFAR100--” depending on whether CIFAR10 or CIFAR100 is the base, where and denote the percentage of open and closed noise rates, respectively. By specifying different and , we can generate datasets with different noise levels flexibly. Suppose the base dataset has classes ( or ) and the open noise source has classes ( or ). We use a probabilistic sampling approach to generate them.

  • First, we generate two similarity matrices. The closed-world similarity matrix is of size , and the open-world matrix is of . One entry or is computed using the cosine similarity.

  • Second, since we often pick the top images returned by search engines in the real world, we generate ( for nCIFAR10 and for nCIFAR100, ) images in the generated dataset for every category. There will be clean examples in all categories, open-world noise examples, and closed-world noise images in total.

  • To sample open noise images, we convert the similarity matrix by a softmax transformation with temperature , then followed by a column-wise normalization, ,

    Because of the subsequent column-wise normalization processing, we do not need to complete the full softmax operation, and the exponential operation alone is enough. Then, the -th column sums up to 1 and forms a valid distribution. Following this distribution, we sample examples from the open noise source categories without replacement, and they form the open noise examples for the -th category in the base dataset. We use . Images are resized to .

  • Similarly, to sample clean and closed noise samples, we transform to by softmax with temperature and a subsequent normalization:

    Note that different leads to different proportion of sampled clean examples, and the relationship between them (value of and the number of clean examples) is monotonic, which is easily proved. Hence, we can use a binary search to find the value that leads to clean samples in total. Because the number of closed-world examples (clean plus closed noise) is , the binary search terminates when the average of the diagonal entries of sum up to . Also note that the binary search process has only trivial computational cost. Then, we use to sample examples from all base categories to the -th category according to with the proper , which include both clean and closed noise samples.

  • Both training sets (nCIFAR10 and nCIFAR100) have size . The test sets are the original CIFAR10 / 100 test sets, which are balanced.

Note that the training datasets generated in our approach are imbalanced (different categories have different number of clean and closed noise samples), and categories with higher similarity scores are more likely to generate noisy samples. These properties make our nCIFAR datasets suitable in learning with noisy labeled examples.111It is desirable to have different numbers of open noise in different categories. However, since we have no clue on how open set noise distributes in general, we sample open noise in every category.

3.2 Advocating the mAP evaluation metric

Figure 1:

Comparing two classifiers in the imbalanced Clothing1M dataset across different categories.

Although imbalance is critical in making a good dataset, it causes difficulty in both training and evaluation. Suppose in an imbalanced binary recognition problem, the ratio between number of examples in two categories A and B is 99:1 in both training and test sets. Then, when accuracy is used as the evaluation metric, a model tends to predict all examples as A, and achieving 99% accuracy. The minority class, B, is entirely sacrificed—hence, high accuracy does not necessarily mean good model in an imbalanced task.

A real-world dataset with label noise is imbalanced in the training set, and very likely also imbalanced in the test set, such as in Clothing1M [33]. Fig. 1 compares two models (Aug for AugDesc [19] and F&F for the proposed method) on Clothing1M (row 6 and 8 in Table 3, respectively). They have almost identical accuracy (74.93% for Aug and 74.92% for F&F). Although we do not know the number of clean training examples in each category, it is reasonable to rank the categories by their size in the test set in the ascending order. Let denote the number of correctly predicted examples of Aug minus that of F&F in the -th ranked category, and the cumulative sum of (, ).

Fig. 1 plots , and an obvious trend shown in Fig. 1 is that the curve is below the zero line in the left half (categories with fewer examples) and above it in the right half (categories with more examples). That is, Aug sacrificed accuracy of categories with fewer (clean) examples to boost the accuracy of other categories with more examples. Again, a higher accuracy value does not necessarily mean a better model in an imbalanced dataset.

Following practices in evaluating imbalanced datasets, we advocate that we must use mean average precision (mAP) to evaluate models for noisy label learning, especially when the test set is imbalanced. mAP is not affected by the imbalance property [21]. In this example, F&F has an mAP of 75.22%, outperforming Aug (73.25%) by a large margin. As shown in Fig. 1, F&F indeed acted more robustly than Aug across different categories.

4 F&F: Techniques

Unlike earlier methods that often focus on one technical aspect (, loss function), recent methods often build systems that integrate multiple techniques. For example, DivideMix [13] integrates techniques like semi-supervised learning, co-training, etc. Indeed, learning from noisy labels is a complex task, and has a strong systems flavor. In this section, we propose a few techniques and build an F&F system, which is based on DivideMix [13] and PENCIL [34].

4.1 Self-supervised backbones & freezed warmup

backbone epoch accuracy
warm-up policy random 00 + 05 66.22
00 + 30 67.94
00 + 35 67.72
ssl-1b 00 + 05 67.42
30 + 05 68.22
ssl-2b 00 + 05 68.34
30 + 00 67.59
30 + 05 68.38
Table 1: DivideMix on nCIFAR100-0.3-0.3. ‘random’: randomly initialized backbones, ‘ssl-1b’: one self-supervised backbone (ssl) in both branches, ‘ssl-2b’: two different ssl backbones. ‘+’: freezed plus normal warmup epochs.

Self-supervised learning has been shown to learn good representation without labels, a property that is conceptually particularly useful in our task (by throwing away all labels). Our first improvement (friend) is to use self-supervised learning algorithms (SimCLR [3] for nCIFAR and MoCo [6] for Clothing1M) to learn initial backbone networks.

Although self-supervised learning has been applied to this domain [17], we argue that in complex methods like DivideMix which have 2 (or more) branches, diversity is in fact essential—we have to use two different self-supervised backbones to initialize them.

Another improvement (friend) is the warmup of backbones. Before starting to train the models, a few warmup epochs are often needed, which updates all parameters in DivideMix. We argue that it is prone to overfitting, and propose to first warmup epochs by freezing the backbone, and then normal epochs that updates all parameters.

As the ablative experiments on nCIFAR100-0.3-0.3 shown in Table 1, both self-supervised learning (our diverse one in particular) and freezed warmup are friends for learning with noisy labels.222The nCIFAR test set is balanced, hence either accuracy or mAP can be used here.

Self-supervised backbones, freezed and normal warmup, plus a smaller number (compared to the original DivideMix) of epochs of DivideMix training form the stage 1 of F&F.

4.2 Improved PENCIL & instance filtering

The first stage’s output model initializes that of the second stage in F&F. In the first stage, DivideMix treats instances with large loss (, less confident ones) as unlabeled examples and adopts semi-supervised learning on them. However, label correction learns or relabels some closed-world noise instances, and can significantly improve the accuracy [34, 36, 17]. Hence, label correction is another friend.

We adopt the PENCIL framework [34]

in stage 2. However, it is difficult to tune the hyperparameters in PENCIL and it is rather complex. We simplify it and improve it.

Consider a -class classification problem. One example has a hard label

in the one-hot encoding:

and , where is a vector of all 1’s. Note that the label is noisy, and hence can be wrong.

In PENCIL, a soft label is defined, where

is the estimated probability of

in the -th class. That is, forms a valid distribution. Note that is not constant, but updated by SGD (, being corrected). For , the model predicts (after softmax) another distribution

, and the Kullback-Leibler divergence

is PENCIL’s first loss term.

Its second loss term measures the compatibility between hard and soft labels, as .

The third PENCIL loss term is the entropy of the prediction, .

Hence, the overall PENCIL loss is


where and are hyperparameters.

PENCIL is sensitive to the hyperparameters: , , and in particular the learning rate to update . When the learning rate for the backbone is 0.01, the learning rate for updating varies in different datasets (which can be as large as 10000), and it is difficult to find a proper value for it. When these hyperparameter values are not suitable, the PENCIL training process often breaks down.

In our F&F, we first remove the compatibility (second) loss term (, ), and fix . Then, after removing these two hyperparameters, the loss is greatly simplified after simple algebraic manipulation: It becomes a reverse cross-entropy loss:


In the original PENCIL, the gradient for the soft labels () involves , which is very unstable. The gradient in F&F becomes simple: , which greatly stabilizes its updating, and we do not need to tune the learning rate for updating .

The final friend that F&F counts on is instance filtering, which has been shown to be effective [26, 24, 32, 17] in the presence of open noise. In PENCIL (and our improved version), closed noisy instances are in fact useful when the model is confident on them. Hence, we believe that removing low-confidence examples in both open and closed noise will be helpful.

When the prediction

has a large entropy, its confidence is low, and vice versa. In stage 2, we compute the prediction’s entropy of all training examples, and fit a Gaussian mixture model (GMM) with 2 components—the first component is considered as high- and the second low-confidence. We can easily compute the posterior probability of an example belonging to the confident mode, and keep this example if the probability is larger than a threshold

. The rest examples are not used. Note that this filtering is similar to that in DivideMix. However, examples with low-confidence are treated as unlabeled and still used in DivideMix, while we ignore them in F&F. This instance filtering process is repeated for each of the 2 branches independently in every epoch. That is, different subsets are filtered per epoch.

Note that when , there is no instance filtering at all. We ablated on nCIFAR100-0.3-0.3. From Table 2, it is clear that removing examples that are not confident () always outperforms using all examples (). That is, instance filtering is an effective approach.

clean open closed accuracy
0.0 69.04 0.25
0.5 69.69 0.22
0.0 69.31 0.24
0.5 69.80 0.28
0.0 69.35 0.27
0.5 69.90 0.08
n/a 71.65 0.06
n/a 66.61 0.26
Table 2: Effects of different types of noise and instance filtering on nCIFAR100-0.3-0.3. 5-time average reported. In the last two rows, models were trained with cross entropy without considering label noise, hence is not applicable.

Because we have groundtruth clean labels in nCIFAR for the entire dataset, we can manually remove either open noise or closed noise from the training set in stage 2 of F&F for ablation. When open noise exists, instance filtering improves roughly 0.5–0.6% in the first two groups in Table 2

and the standard deviations are all

. In the third group, closed noise exists but open noise does not, where instance filtering improves 0.65% and the standard deviation is much smaller (only 0.08), , more stable. And, the sixth row has the highest accuracy, where open noise has been removed and instances have been filtered.

In the last group, we also trained networks using only the classic cross entropy loss (, without noise handling). When only the clean samples are used, its accuracy is 71.65%. When all examples are used (including both types of noisy examples), its accuracy is 66.61%.

1: Stage 1
2:Learn two backbone networks using self-supervised learning, and use them to initialize the two branches
3:For each backbone, perform freezed and normal warmup
4:Run DivideMix for a few epochs
5: Stage 2
6:for  to the number of training epochs do
7:     for each of the two branches do
8:         Compute entropy values for all training examples
9:         Fit a GMM with 2 components to model the entropy distribution
10:         Compute the posterior probability for all training examples
11:         Filter instances using the posterior probability, where is the threshold
12:         Run improved PENCIL on only examples not filtered away in this epoch
13:     end for
14:end for
Algorithm 1 The proposed F&F framework

These observations indicate that

  • Closed noise is more friendly than open noise, because many closed noisy labels will be corrected by our improved PENCIL (, becoming clean).

  • When we use all samples (like in DivideMix), , without instance filtering (), both closed and open noise will be harmful.

  • When we only keep confident samples (), even open noise will not significantly impoverish the model.

  • Both instance filtering and label correction are effective. They bring close to 1 percentage point improvement when combined in F&F (69.04% vs. 69.90%).

  • F&F has significantly bridged the accuracy gap caused by noisy labels: 66.64% when without noise handling, 69.90% for F&F, and 71.65% for clean data.

Finally, the two branches in stage 2 are independent of each other. We are in fact training two different networks separately and then average their predictions.

4.3 The F&F framework

Pseudo-code for the overall F&F framework is shown in Algorithm 1. For more details, we will publish the code of F&F.

It is also worth noting that the second stage runs much faster than the first stage. We recorded the timing statistics on the nCIFAR100-0.2-0.2 dataset, in which the first stage took 9.53 hours, while stage 2 only took 0.27 hours.

That is, the combination of instance filtering and label correction in stage 2, although only used 2.9% running time of that of stage 1, can lead to considerable improvements in accuracy.

5 Experimental Settings and Results

In this section, we first describe the settings in our experiments, then present experimental results on both the proposed nCIFAR datasets and the real-world Clothing1M dataset. Although the noisy CIFAR datasets from [28] are inappropriate, we also report F&F’s results on them, in order to facilitate comparisons with existing methods. We will publish the code to generate the proposed nCIFAR datasets, and the exact versions used in our experiments will be published, too. We will also publish code of the F&F system.

5.1 Detailed data and algorithmic settings

Datasets. In this paper, we use 5 versions of the proposed nCIFAR datasets: nCIFAR10-0.1-0.1, nCIFAR10-0.2-0.2, nCIFAR100-0.1-0.1, nCIFAR100-0.2-0.2 and nCIFAR100-0.3-0.3. The last dataset has only 40% clean samples. nCIFAR10 are relatively easy, which are suitable for sanity-check of algorithms. nCIFAR100 are more difficult, and are useful for evaluating, comparing, analyzing and understanding algorithms. Note that although we set the percentage of open noise the same as that of the closed noise, it is trivial to generate nCIFAR datasets with different ratios for them.

Baseline methods. In terms of baseline methods, we mainly compare with 2 existing methods: DivideMix [13] and AugDesc [19]. DivideMix is a strong method and many recent advances in learning from noisy labels are developed based on it. AugDesc is the current top performer.333

For both methods, we used the codes published by their respective authors. We use their hyperparameter settings without modifications.

F&F: shared. We update network parameters and soft labels using SGD with a momentum of 0.9. We set weight decay of network parameters to for CIFAR based datasets and for Clothing1M. We do not apply weight decay for updating soft labels.

F&F stage 2: Improved PENCIL. As aforementioned, the hyperparameter setting in F&F’s stage 2 is much simpler than that in the original PENCIL [34]. We used the same setting in all our experiments in this part. In the improved PENCIL training, we always used 0.5 as the initial learning rate to update soft labels . For other network parameters, the learning rate was for all CIFAR based datasets and for Clothing1M. Improved PENCIL always ran 20 epochs, and both learning rates were reduced by a factor of 10 midway (, after 10 epochs).

F&F stage 2: Instance filtering. Since we have shown the effectiveness of instance filtering in ablative studies (cf. Table 2), we always used in instance filtering. For all CIFAR based datasets, label correction with instance filtering was conducted on all the training samples. For Clothing1M, we sampled a subset containing about 260k images to apply label correction, following the original PENCIL [34]. Note that this subset is balanced when considering the noisy labels. However, if we have access to clean labels, this subset will not be balanced.

F&F stage 1: Self-supervised learning. For noisy CIFAR from [28], we used SimCLR [3] to train PreActResNet-18 [8] on CIFAR for 1000 epochs. Because these datasets share the same unlabeled images for CIFAR10 / 100 regardless of the noise type or noise ratio, we only need to train a pair of backbones with the original CIFAR10 / 100 datasets, and they can be shared among experiments on different noisy CIFAR datasets.

On our nCIFAR datasets, SimCLR (1000 epochs) was still used. However, because different nCIFAR10 or nCIFAR100 datasets have different sets of unlabeled training images, it is essential to train a pair of backbones for each nCIFAR dataset separately.

On Clothing1M, we used MoCo [6] to train two different backbones of ResNet-50 [7], initialized with the official pretrained weights trained on ImageNet (800 epochs), then trained on Clothing1M for 200 epochs.

F&F stage 1: Warmup. For noisy CIFAR in [28], in warmup we trained the classifiers for 30 epochs with the backbones frozen. The learning rate was , and the batch size was 128. Because we only experimented with very high noise rate here, there was no normal warmup training in order to avoid overfitting.

For nCIFAR, we first train the classifiers for 30 epochs with the backbones frozen and then warm-up the whole network for 5 epochs. The learning rate was , and the batch size was 128.

On Clothing1M, we sampled a (noisy) label-balanced subset containing about 260k images, and trained the classifiers for 10 epochs with the backbones frozen. We used a learning rate of , and a batch size of 128.

F&F stage 1: DivideMix. On all datasets, the temperature of label sharpening and the threshold of small-loss sample filtering were both fixed to 0.5 in all experiments.

For noisy CIFAR in [28], the DivideMix training part in stage 1 used the same parameters as those in the original DivideMix.

For nCIFAR, the learning rate and batch size were the same as in warmup. , strength of the consistency loss in DivideMix was set to 100 for nCIFAR100, and 0 for nCIFAR10, following the original DivideMix. in mixup was set to 0.5. The networks were trained for 200 epochs with the learning rate reduced by a factor of 10 after half of the epochs.

On Clothing1M, we set to 0.2, and to 0 (which follows the original DivideMix). We trained the networks for 30 epochs. In each epoch we sampled 2000 (noisy) label-balanced mini-batches, and the batch size was 64. The learning rate was set to , and was reduced by a factor of 10 after 15 epochs.

For all these methods, we first sum the logits of the two branches, then apply a softmax transformation to predict the final results. When standard deviations are reported, we ran the experiments 5 times and report the mean accuracy or mAP, plus its standard deviation.

Selection criterion Method mAP accuracy
last DivideMix 72.27 74.03
AugDesc 72.92 75.18
F&F stage1 72.39 73.83
F&F stage2 75.31 74.89
best accuracy DivideMix 72.22 74.19
AugDesc 73.25 74.93
F&F stage1 73.75 74.54
F&F stage2 75.22 74.92
best mAP DivideMix 73.15 74.33
AugDesc 73.42 74.87
F&F stage1 74.78 75.00
F&F stage2 75.42 75.10
Table 3: Comparison with state-of-the-art methods on Clothing1M. We ran author-provided codes to generate DivideMix and AugDesc results.
Dataset nCIFAR10-0.1-0.1 nCIFAR10-0.2-0.2 nCIFAR100-0.1-0.1 nCIFAR100-0.2-0.2 nCIFAR100-0.3-0.3
DivideMix 93.88 0.22 92.00 0.32 75.76 0.21 73.55 0.49 67.66 0.32
F&F stage1 94.16 0.07 92.15 0.34 76.04 0.21 73.68 0.28 68.29 0.12
F&F stage2 94.73 0.02 92.84 0.26 77.44 0.18 75.29 0.21 69.69 0.22
Table 4: Comparison between F&F and DivideMix in terms of accuracy (%) on the proposed nCIFAR datasets.
Method Selection criterion asym-40% sym-90% sym-80% sym-90%
best 91.16 61.21 - -
PENCIL [34] last 91.01 60.80 - -
best 93.4 76.0 60.2 31.5
DivideMix [13] last 92.1 75.4 59.6 31.0
best 94.6 91.9 66.4 41.2
AugDesc [19] last 94.3 91.8 66.1 40.9
best 93.95 0.23 93.24 0.47 66.99 0.48 54.48 1.25
F&F (Ours) last 93.58 0.17 93.24 0.51 67.08 0.51 54.54 1.36
Table 5: Comparison between F&F and previous state-of-the-art methods in terms of accuracy (%) on noisy CIFAR datasets from [28]. Results of PENCIL, DivideMix and AugDesc are copied from respectively papers. ‘asym’ and ‘sym’ are abbreviations for asymmetric and symmetric noise, respectively.

5.2 Results on Clothing1M

We start reporting results from the real-world Clothing1M dataset, whose test set is imbalanced. Different selection criteria were used to select the model: the one at the last training epoch, and the one with the best accuracy or the best mAP on the validation set. Note that no validation data is required for the ‘last’ criterion. Results are in Table 3.

It is clear that F&F is the consistent winner in terms of mAP, the evaluation metric which we advocate because it does not sacrifice under-represented categories. But, when accuracy is used, F&F is also a strong method. Even an incomplete F&F (stopping at stage 1) consistently outperforms DivideMix. These results show that our warmup and self-supervised learning improvements in stage 1, and instance filtering and label correction in stage 2 are all effective.

Another interesting observation is that the ‘best mAP’ criterion acts betters than the ‘best accuracy’ one, which shows that accuracy is not an appropriate metric in imbalanced tasks from a different perspective.

5.3 Results on nCIFAR

On the proposed nCIFAR datasets, as shown in Table 4, the accuracy is used because its test set is balanced. We argue that a validation set is difficult to obtain in real world settings, and advocate using the last epoch’s model. Table 4 shows five time average of the accuracy of the last training epoch.

From Table 4, the stage 1 of F&F consistently outperforms DivideMix, and the complete F&F (stage 2) consistently beats its own stage 1.

Another interesting observation is that the complete F&F has smallest variance in general, while F&F stage 1 has smaller variance than DivideMix in most cases—meaning F&F is more robust than DivideMix.

One of the reasons that methods which learn from noisy labels require a validation set is that they are sometimes not robust, hence a validation set is needed to pick the optimal model from the many training epochs. A robust method, like F&F, suggests that we better fend off our dependency on the validation set, and directly use the last epoch’s model.

5.4 Results on noisy CIFAR from [28]

We have discussed the drawbacks of the noisy CIFAR dataset previously used in the community, and have advocated the adoption of our nCIFAR datasets. However, to provide a thorough comparison to previous methods, we also conducted experiments on them, with results presented in Table 5. We only experimented with the most difficult ones: 40% asymmetric and 90% symmetric noise on noisy CIFAR10, plus 80% and 90% symmetric noise on noisy CIFAR100 from [28].

First, F&F is still very robust. In fact, in three out of four cases, F&F’s ‘last’ outperforms ‘best’. Second, except on noisy CIFAR10 with 40% asymmetric noise, F&F is the clear winner. Third, in the two CIFAR100 symmetric noise cases, the winning margin of F&F is significant, even more than 10 percentage points with 90% symmetric noise.

For the last observation, our conjecture is that two factors lead to this large gap: self-supervised learning (especially our diverse initialization strategy), and label correction.

6 Conclusions and Future Work

In this paper, we first advocated two changes to the area of learning from examples with noisy labels: the dataset and the evaluation metric. We analyzed the drawbacks of the current datasets and metric, then proposed replacements: our nCIFAR datasets and the mAP metric (at least for imbalanced test sets). Then, we proposed an F&F framework, which highlights four ‘friends’ with our improved techniques: diverse self-supervised learning, freezed warmup, label correction and instance filtering. F&F has not only outperformed existing methods, but also has higher robustness—its last epoch model’s prediction has smaller variances.

The filed of learning from noisy labels shares similarity with learning from webly supervised data. In the future, we will migrate F&F to such datasets as WebVision [14]. Furthermore, learning from noisy labels in a multi-label setting is not only more challenging but also has wider utility in applications. We will investigate into this area by relaxing constraints on the soft labels in F&F.


  • [1] Samaneh Azadi, Jiashi Feng, Stefanie Jegelka, and Trevor Darrell. Auxiliary image regularization for deep CNNs with noisy labels. In International Conference on Learning Representations, 2016.
  • [2] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32, pages 5050–5060, 2019.
  • [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In

    International Conference on Machine Learning

    , pages 1597–1607, 2020.
  • [4] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
  • [5] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Conférence francophone sur l’apprentissage automatique, pages 281–296, 2005.
  • [6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In

    IEEE/CVF Conference on Computer Vision and Pattern Recognition

    , pages 9726–9735, 2020.
  • [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  • [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, volume 9908 of Lecture Notes in Computer Science, pages 630–645, 2016.
  • [9] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  • [10] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2017.
  • [11] Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang.

    Cleannet: Transfer learning for scalable image classifier training with label noise.

    In IEEE Conference on Computer Vision and Pattern Recognition, pages 5447–5456, 2018.
  • [12] Fei-Fei Li, Andrej Karpathy, and Justin Johnson. Tiny ImageNet challenge, 2017.
  • [13] Junnan Li, Richard Socher, and Steven C. H. Hoi. DivideMix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
  • [14] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. WebVision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862, 2017.
  • [15] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In IEEE International Conference on Computer Vision, pages 1928–1936, 2017.
  • [16] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. In Advances in Neural Information Processing Systems 33, 2020.
  • [17] Devraj Mandal, Shrisha Bharadwaj, and Soma Biswas. A novel self-supervised re-labeling approach for training with noisy labels. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2020.
  • [18] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2020.
  • [19] Kento Nishi, Yi Ding, Alex Rich, and Tobias Höllerer. Augmentation strategies for learning with noisy labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  • [20] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving Jigsaw puzzles. In European Conference on Computer Vision, volume 9910 of Lecture Notes in Computer Science, pages 69–84, 2016.
  • [21] Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In IEEE Conference on Computer Vision and Pattern Recognition, pages 864–873, 2016.
  • [22] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In

    Empirical Methods in Natural Language Processing

    , pages 1532–1543, 2014.
  • [23] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton.

    Regularizing neural networks by penalizing confident output distributions.

    In International Conference on Learning Representations, 2017.
  • [24] Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking. In Advances in Neural Information Processing Systems 33, 2020.
  • [25] Ragav Sachdeva, Filipe R. Cordeiro, Vasileios Belagiannis, Ian D. Reid, and Gustavo Carneiro. EvidentialMix: Learning with combined open-set and closed-set noisy labels. In IEEE Winter Conference on Applications of Computer Vision, 2021.
  • [26] Karishma Sharma, Pinar Donmez, Enming Luo, Yan Liu, and I. Zeki Yalniz. NoiseRank: Unsupervised label noise reduction with dependence models. In European Conference on Computer Vision, volume 12372 of Lecture Notes in Computer Science, pages 737–753, 2020.
  • [27] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-Weight-Net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems 32, pages 1917–1928, 2019.
  • [28] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5552–5560, 2018.
  • [29] Arash Vahdat. Toward robustness against label noise in training deep discriminative neural networks. In Advances in Neural Information Processing Systems 30, pages 5596–5605, 2017.
  • [30] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.

    Extracting and composing robust features with denoising autoencoders.

    In International Conference on Machine Learning, pages 1096–1103, 2008.
  • [31] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE/CVF International Conference on Computer Vision, pages 322–330, 2019.
  • [32] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N. Metaxas, and Chao Chen. A topological filter for learning with label noise. In Advances in Neural Information Processing Systems 33, 2020.
  • [33] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2691–2699, 2015.
  • [34] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7017–7025, 2019.
  • [35] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
  • [36] Guoqing Zheng, Ahmed H. Awadallah, and Susan Dumais. Meta label correction for noisy label learning. In

    AAAI Conference on Artificial Intelligence

    , 2021.

Appendix A Proof of the monotonicity of the closed noise rate with respect to

As is mentioned in the paper, one entry in the closed-world similarity matrix is calculated using the cosine similarity between the representation vectors of the -th and -th categories for . The cosine similarity between one vector and itself equals to 1, , for any . Different categories always have different representation vectors, and hence if . Thus, we can safely assume that holds for all .

Now consider the set of closed-world examples generated in the proposed nCIFAR dataset with hyperparameter , which includes both instances with clean and closed-world noisy labels. We denote the percentage of closed-world noisy labels as (, the number of closed noise divided by the number of all closed-world examples). According to the approach that generates examples in nCIFAR, we have




for . Note that we want in nCIFAR.



That is, the closed noise rate is monotonically increasing with respect to the temperature . Therefore, it is reasonable to apply binary search to efficiently find the exact value that leads to .

Appendix B Details of the loss function in our improved PENCIL

It is known that PENCIL suffers from a seemingly unreasonable hyperparameter value, which is the learning rate used to update the soft labels . In F&F, the key technique to solve this issue is: to update soft labels with a modified loss function, and to update soft labels and network parameters separately.

As is shown in the paper, the loss related to one sample is defined as


Both network parameters and soft labels are updated using SGD. Let us assume that the size of a mini-batch is denoted by . For the -th example in this mini-batch, the example, the network’s prediction and its soft label are denoted by , and , respectively. Note that the -th dimension of the vector is denoted as .

We follow the conventional approach, and use the average gradient from a mini-batch to update network parameters, because the same set of network parameters are shared among all the samples. Hence, the loss term responsible for updating network parameters is


When it turns to update the soft labels, the situation is completely different: one sample has its own soft label ; and, is only corresponding to and it is not related to other or when . Hence, the gradient must not be averaged within a mini-batch.

In PENCIL, the learning rate to update soft labels has to be enlarged to offset the effect of the batch size, namely the coefficient . In other words, this hyperparameter is in fact dependent on the mini-batch size in PENCIL. This explains why this hyperparameter has unusual values like 3000 or 10000 in PENCIL.

We use the following loss term to update soft labels:


By removing the term in the loss function, we can use the same learning rate to update soft labels in all our F&F experiments, , now it becomes very robust.

Technically, we detach the gradient to network parameters when updating soft labels, and vice versa.