L_DMI: An Information-theoretic Noise-robust Loss Function

1 Introduction

Deep neural networks, together with large scale accurately annotated datasets, have achieved remarkable performance in a great many classification tasks in recent years (e.g., Krizhevsky et al. (2012); He et al. (2016)). However, it is usually money- and time- consuming to find experts to annotate labels for large scale datasets. While collecting labels from crowdsourcing platforms like Amazon Mechanical Turk is a potential way to get annotations cheaper and faster, the collected labels are usually very noisy. The noisy labels hampers the performance of deep neural networks since the commonly used cross entropy loss is not noise-robust. This raises an urgent demand on designing noise-robust loss functions.

Some previous works have proposed several loss functions for training deep neural networks with noisy labels. However, they either use auxiliary information Patrini et al. (2017); Hendrycks et al. (2018), like an additional set of clean data or the noise transition matrix, which cannot be easily obtained in practice, or make assumptions on the noise Ghosh et al. (2017); Zhang and Sabuncu (2018) and thus can only handle limited kinds of the noise patterns (see perliminaries for definition of different noise patterns), like uniform noise only or diagonally dominant noise only.

One reason that the loss functions used in previous works are not robust to a certain noise pattern, say diagonally non-dominant noise, is that they are distance-based, i.e

., the loss is the distance between the classifier’s outputs and the labels (

e.g. 0-1 loss, cross entropy loss). When datapoints are labeled by a careless annotator who tends to label the a priori popular class (e.g. For medical images, given the prior knowledge is

10 %

malignant and

90 %

benign, a careless annotator labels “benign” when the underline true label is “benign” and labels “benign” with 90% probability when the underline true label is “malignant”.), the collected noisy labels have a diagonally non-dominant noise pattern and are extremely biased to one class (“benign”). In this situation, the distanced-based losses will prefer the meaningless classifier who always outputs the a priori popular class (“benign”) than the classifier who outputs the true labels.

To address this issue, instead of using distance-based losses, we try the information-theoretic loss such that the classifier, whose outputs have the highest mutual information with the labels, has the lowest loss. The key observation is that the meaningless classifier has no information about anything and will be naturally eliminated by the information-theoretic loss. Moreover, the information-monotonicity of the mutual information guarantees that adding noises to a classifier’s output will make this classifier less preferred by the information-theoretic loss.

However, the key observation is not sufficient. In fact, we want a information measure I to satisfy

		$I (classifier 1's output; noisy labels) > % I (classifier 2's output; noisy labels)$
	$\Leftrightarrow$	$I (classifier 1's output; clean labels) > % I (classifier 2's output; clean labels) .$

Unfortunately, the traditional Shannon mutual information (MI) does not satisfy the above formula. Therefore, we generalize it to a new information measure, DMI (Determinant based Mutual Information). Like MI, DMI measures the correlation between two random variables. It is defined as the determinant of the matrix that describes the joint distribution over the two variables. Intuitively, when two random variables are independent, their joint distribution matrix has low rank and zero determinant. Moreover, DMI is not only information-monotone like MI, but also relatively invariant because of the multiplication property of the determinant. The relative invariance of DMI makes it satisfy the above formula.

Based on DMI, we propose a noise-robust loss function $L_{DMI}$ which is simply

L_{DMI} (data; classifier) := - log [DMI (classifier's output; labels)] .

With $L_{DMI}$ , the following equation holds:

L_{DMI} (noisy data; classifier) = L_{DMI} (clean data; classifier) + % noise amount,

and the noise amount is a constant given the dataset. The equation reveals that with $L_{DMI}$ , training with the noisy labels is theoretically equivalent with training with the clean labels in the dataset, regardless of the noise patterns, including the noise amounts.

In a nutshell, our main contribution can be summarized into the following three points:

We propose a new information measure, DMI, which is information-monotone and relatively invariant. Under the performance measure based on DMI, the measurement based on noisy labels is consistent with the measurement based on clean labels.
We design a novel information theoretic noise-robust loss function $L_{DMI}$ based on the above information measure. It is not sensitive to noise patterns and it can be easily applied to any existing classification neural networks straightforwardly without any auxiliary information.
Extensive experiments on Fashion-MNIST, CIFAR-10, Dogs vs. Cats datasets with a variety of synthesized noise patterns and noise amounts as well as a real-world dataset Clothing1M demonstrate the superior performance of $L_{DMI}$ .

2 Related Work

A series of works have attempted to design noise-robust loss functions. In the context of binary classification, some loss functions (e.g., 0-1 lossManwani and Sastry (2013), ramp lossBrooks (2011), unhinged lossVan Rooyen et al. (2015), savage lossMasnadi-Shirazi and Vasconcelos (2009)) have been proved to be robust to uniform or symmetric noise and Natarajan et al. Natarajan et al. (2013) presented a general way to modify any given surrogate loss function. Ghosh et al. Ghosh et al. (2017) generalized the existing results for binary classification problem to multi-class classification problem and proved that MAE (Mean Absolute Error) is robust to diagonally dominant noise. Zhang et al. Zhang and Sabuncu (2018) showed MAE performs poorly with deep neural network and they combined MAE and cross entropy loss to obtain a new loss function. Patrini et al. Patrini et al. (2017) provided two kinds of loss correction methods with the noise transition matrix. Hendrycks et al. Hendrycks et al. (2018) proposed another loss correction technique with an additional set of clean data. To the best of our knowledge, we are the first to provide a loss function that is provably not sensitive to noise patterns and noise amounts.

Instead of designing an inherently noise-robust function, several works used special architectures to deal with the problem of training deep neural networks with noisy labels. Some of them focused on estimating the noise transition matrix to handle the label noise and proposed a variety of ways to constrain the optimization

Sukhbaatar et al. (2014); Xiao et al. (2015); Goldberger and Ben-Reuven (2016); Vahdat (2017); Han et al. (2018a); Yao et al. (2019). Some of them focused on finding ways to distinguish noisy labels from clean labels and used example re-weighting strategies to give the noisy labels less weights Reed et al. (2014); Ren et al. (2018); Ma et al. (2018). While these methods seem to perform well in practice, they cannot guarantee the robustness to label noise theoretically and are also outperformed by our method empirically.

On the other hand, Zhang et al. Zhang et al. (2016)

have shown that deep neural networks can easily memorize completely random labels, thus several works propose frameworks to prevent this overfitting issue empirically in the setting of deep learning from noisy labels. For example, teacher-student curriculum learning framework

Jiang et al. (2017) and co-teaching framework Han et al. (2018b) have been shown to be helpful. Multi-task frameworks that jointly estimates true labels and learns to classify images are also introduced Veit et al. (2017); Lee et al. (2018); Tanaka et al. (2018); Yi and Wu (2019). We consider a different perspective from them and focus on designing an inherently noise-robust function.

3 Preliminaries

3.1 Problem settings

We denote the set of classes by $C$ and the size of $C$ by $C$ . We also denote the domain of datapoints by $X$ . A classifier is denoted by $h : X \mapsto Δ_{C}$ , where $Δ_{C}$ is the set of all possible distributions over $C$ . $h$ represents a randomized classifier such that given $x \in X$ , $h (x)_{c}$ is the probability that $h$ maps $x$ into class $c$ . Note that fixing the input $x$ , the randomness of a classifier is independent of everything else.

There are $N$ datapoints ${x_{i}}_{i = 1}^{N}$ . For each datapoint $x_{i}$ , there is an unknown ground truth $y_{i} \in C$ . We assume that there is an unknown prior distribution $Q_{X, Y}$ over $X \times C$ such that ${(x_{i}, y_{i})}_{i = 1}^{N}$ are i.i.d. samples drawn from $Q_{X, Y}$ and

Q_{X, Y} (x, y) = Pr [X = x, Y = y] .

Note that here we allow the datapoints to be “imperfect” instances, i.e., there still exists uncertainty for $Y$ conditioning on fully knowing $X$ .

Traditional supervised learning aims to train a classifier

h^{*}

that is able to classify new datapoints into their ground truth categories with access to

{(x_{i}, y_{i})}_{i = 1}^{N}

. However, in the setting of learning with noisy labels, instead, we only have access to

{(x_{i}, {~ y}_{i})}_{i = 1}^{N}

where

{~ y}_{i}

is a noisy version of

y_{i}

We use a random variable $~ Y$ to denote the noisy version of $Y$ and $T_{Y \to ~ Y}$ to denote the transition distribution between $Y$ and $Y$ , i.e.

T_{Y \to ~ Y} (y, ~ y) = Pr [~ Y = ~ y | Y = y] .

We use $T_{Y \to ~ Y}$ to represent the $C \times C$ matrix format of $T_{Y \to ~ Y}$ .

Generally speaking Patrini et al. (2017); Ghosh et al. (2017); Zhang and Sabuncu (2018), label noise can be divided into several kinds according to the noise transition matrix $T_{Y \to ~ Y}$ . It is defined as class-independent (or uniform) if a label is substituted by a uniformly random label regardless of the classes, i.e. $Pr [~ Y = ~ c | Y = c] = Pr [~ Y = {~ c}^{'} | Y = c], \forall ~ c, {~ c}^{'} \neq c$ (e.g. $T_{Y \to ~ Y} = [\begin{matrix} 0.7 & 0.3 0.3 & 0.7 \end{matrix}]$ ). It is defined as diagonally dominant if for every row of $T_{Y \to ~ Y}$ , the magnitude of the diagonal entry is larger than any non-diagonal entry, i.e. $Pr [~ Y = c | Y = c] > Pr [~ Y = ~ c | Y = c], \forall ~ c \neq c$ (e.g. $T_{Y \to ~ Y} = [\begin{matrix} 0.7 & 0.3 0.2 & 0.8 \end{matrix}]$ ). It is defined as diagonally non-dominant if it is not diagonally dominant (e.g. the example mentioned in introduction, $T_{Y \to ~ Y} = [\begin{matrix} 1 & 0 0.9 & 0.1 \end{matrix}]$ ).

We assume that the noise is independent of the datapoints conditioning on the ground truth, which is commonly assumed in the literature Patrini et al. (2017); Ghosh et al. (2017); Zhang and Sabuncu (2018), i.e.,

Assumption 3.1 (Independent noise).

$X$ is independent of $~ Y$ conditioning on $Y$ .

We also need that the noisy version $~ Y$ is still informative.

Assumption 3.2 (Informative noisy label).

$T_{Y \to ~ Y}$ is invertible, i.e., $det (T_{Y \to ~ Y}) \neq 0$ .

3.2 Information theory concepts

Since Shannon’s seminal work Shannon (1948), information theory has shown its powerful impact in various of fields, including several recent deep learning works Hjelm et al. (2018); Cao et al. (2018). Our work is also inspired by information theory. This section introduces several basic information theory concepts.

Information theory is commonly related to random variables. For every random variable $W_{1}$ , Shannon’s entropy $H (W_{1}) := \sum_{w_{1}} Pr [W = w_{1}] log Pr [W = w_{1}]$ measures the uncertainty of $W_{1}$ . For example, deterministic $W_{1}$ has lowest entropy. For every two random variables $W_{1}$ and $W_{2}$ , Shannon mutual information $MI (W_{1}, W_{2}) := \sum_{w_{1}, w_{2}} Pr [W_{1} = w_{1}, W_{2} = w_{2}] log \frac{Pr [W = w_{1}, W = w_{2}]}{Pr [W_{1} = w_{1}] Pr [W_{2} = w_{2}]}$ measures the amount of relevance between $W_{1}$ and $W_{2}$ . For example, when $W_{1}$ and $W_{2}$ are independent, they have the lowest Shannon mutual information, zero.

Shannon mutual information is non-negative, symmetric, i.e., $MI (W_{1}, W_{2}) = MI (W_{2}, W_{1})$ , and also satisfies a desired property, information-monotonicity, i.e., the mutual information between $W_{1}$ and $W_{2}$ will always decrease if either $W_{1}$ or $W_{2}$ has been “processed”.

Fact 3.3 (Information-monotonicity Csiszár et al. (2004)).

For all random variables $W_{1}, W_{2}, W_{3}$ , when $W_{3}$ is less informative for $W_{2}$ than $W_{1}$ , i.e., $W_{3}$ is independent of $W_{2}$ conditioning $W_{1}$ ,

MI (W_{3}, W_{2}) \leq MI (W_{1}, W_{2}) .

This property naturally induces that for all random variables $W_{1}, W_{2}$ ,

MI (W_{1}, W_{2}) \leq MI (W_{2}, W_{2}) = H (W_{2})

since $W_{2}$ is always the most informative random variable for itself.

Based on Shannon mutual information, a performance measure for a classifier $h$ can be naturally defined. High quality classifier’s output $h (X)$ should have high mutual information with the ground truth category $Y$ . Thus, a classifier $h$ ’s performance can be measured by $MI (h (X), Y)$ .

However, in our setting, we only have access to the i.i.d. samples of $h (X)$ and $~ Y$ . A natural attempt is to measure a classifier $h$ ’s performance by $MI (h (X), ~ Y)$ . Unfortunately, under this performance measure, the measurement based on noisy labels $MI (h (X), ~ Y)$ may not be consistent with the measurement based on true labels $MI (h (X), Y)$ . (See a counterexample in Appendix B.) That is,

Thus, we cannot use Shannon mutual information as the performance measure for classifiers. Here we give a new version of mutual information, Determinant based Mutual Information (DMI), such that under the performance measure based on DMI, the measurement based on noisy labels is consistent with the measurement based on true labels.

Definition 3.4 (Determinant based Mutual Information).

Given two discrete random variables

W_{1}, W_{2}

, we define the Determinant based Mutual Information between

W_{1}

and

W_{2}

DMI (W_{1}, W_{2}) = | det (Q_{W_{1}, W_{2}}) |

where $Q_{W_{1}, W_{2}}$ is the matrix format of the joint distribution over $W_{1}$ and $W_{2}$ .

DMI is a generalized version of Shannon’s mutual information: it preserves all properties of Shannon mutual information, including non-negativity, symmetry and information-monotonicity and it is additionally relatively invariant.

Theorem 3.5 (Properties of DMI).

DMI is non-negative, symmetric and information-monotone. Moreover, it is relatively invariant: for all random variables $W_{1}, W_{2}, W_{3}$ , when $W_{3}$ is less informative for $W_{2}$ than $W_{1}$ , i.e., $W_{3}$ is independent of $W_{2}$ conditioning $W_{1}$ ,

where $T_{W_{1} \to W_{3}}$ is the matrix format of

T_{W_{1} \to W_{3}} (w_{1}, w_{3}) = Pr [W_{3} = w_{3} | W_{1} = W_{1}] .

Proof.

The non-negativity and symmetry follow directly from the definition, so we only need to prove the relatively invariance. Note that

Pr Q_{W_{2}, W_{3}} [W_{2} = w_{2},, W_{3} = w_{3}] = \sum w_{1} Pr Q_{W_{1}, W_{2}} [W_{2} = w_{2}, W_{1} = w_{1}] Pr [W_{3} = w_{3} | W_{1} = w_{1}] .

as $W_{3}$ is independent of $W_{2}$ conditioning on $W_{1}$ . Thus,

Q_{W_{2}, W_{3}} = Q_{W_{2}, W_{1}} T_{W_{1} \to W_{3}}

where $Q_{W_{2}, W_{3}}$ , $Q_{W_{2}, W_{1}}$ , $T_{W_{1} \to W_{3}}$ are the matrix formats of $Q_{W_{2}, W_{3}}$ , $Q_{W_{2}, W_{1}}$ , $T_{W_{1} \to W_{3}}$ , respectively. We have

det (Q_{W_{2}, W_{3}}) = det (Q_{W_{2}, W_{1}}) det (T_{W_{1} \to W_{3}})

because of the multiplication property of the determinant (i.e. $det (A B) = det (A) det (B)$ for every two matrices $A, B$ ). Therefore, .

The relative invariance and the symmetry imply the information-monotonicity of DMI. When $W_{3}$ is less informative for $W_{2}$ than $W_{1}$ , i.e., $W_{3}$ is independent of $W_{2}$ conditioning on $W_{1}$ ,

	$DMI (W_{3}, W_{2}) = DMI (W_{2}, W_{3})$	$= DMI (W_{2}, W_{1}) \| det (T_{W_{1} \to W_{3}}) \|$
		$\leq DMI (W_{2}, W_{1}) = DMI (W_{1}, W_{2})$

because of the fact that for every square transition matrix $T$ , $det (T) \leq 1$ Seneta (2006). ∎

Based on DMI, an information-theoretic performance measure for each classifier $h$ is naturally defined as $DMI (h (X), ~ Y)$ . Under this performance measure, the measurement based on noisy labels $DMI (h (X), ~ Y)$ is consistent with the measurement based on clean labels $DMI (h (X), Y)$ , i.e., for every two classifiers $h$ and $h^{'}$ ,

DMI (h (X), Y) > DMI (h^{'} (X), Y) \Leftrightarrow DMI (h (X), ~ Y) > DMI (h^{'} (X), ~ Y) .

4 $L_{DMI}$ : An Information-theoretic Noise-robust Loss Function

4.1 Method overview

Our loss function is defined as

L_{DMI} (Q_{h (X), ~ Y}) := - log (D M I (h (X), ~ Y)) = - log (| det (Q_{h (X), ~ Y}) |)

where $Q_{h (X), ~ Y}$ is the joint distribution over $h (X), ~ Y$ and $Q_{h (X), ~ Y}$ is the $C \times C$ matrix format of $Q_{h (X), ~ Y}$ . The randomness $h (X)$ comes from both the randomness of $h$ and the randomness of $X$ . The $log$ function here resolves many scaling issues¹¹1 $\frac{\partial (c | det (A) |)}{\partial A} = c | det (A) | (A^{- 1})^{T}$ while $\frac{\partial log (c | det (A) |)}{\partial A} = (A^{- 1})^{T}$ , $\forall$ matrix $A$ and $\forall$ constant $c$ ..

Figure 1: The computation of $L_{DMI}$ in each step of iteration

Figure 1 shows the computation of $L_{DMI}$ . In each step of iteration, we sample a batch of datapoints and their noisy labels ${(x_{i}, {~ y}_{i})}_{i = 1}^{N}$ . We denote the outputs of the classifier by a matrix $O$ . Each column of $O$ is a distribution over $C$ , representing for an output of the classifier. We denote the noisy labels by a 0-1 matrix $L$ . Each row of $L$

is an one-hot vector, representing for a label. i.e.

O_{c i} = h (x_{i})_{c}, L_{i ~ c} = 1 [{~ y}_{i} = ~ c],

We define $U := \frac{1}{N} O L$ , i.e.,

U_{c ~ c} := \frac{1}{N} N \sum i = 1 O_{c i} L_{i ~ c} = \frac{1}{N} N \sum i = 1 h (x_{i})_{c} 1 [{~ y}_{i} = ~ c] .

We have $E U_{c ~ c} = Pr [h (X) = c, ~ Y = ~ c] = Q_{h (X), ~ Y} (c, ~ c)$ ( $E$ means expectation, see proof in Appendix B). Thus, $U$ is an empirical estimation of $Q_{h (X), ~ Y}$ . By abusing notation a little bit, we define

L_{DMI} ({(x_{i},_{i})}_{i = 1}^{N}; h) = - log (| det (U) |)

as the empirical loss function. Our formal training process is shown in Appendix A.

4.2 Theoretical justification

Theorem 4.1 (Main Theorem).

With Assumption 3.1 and Assumption 3.2, $L_{DMI}$ is

legal

if there exists a ground truth classifier $h^{*}$ such that $h^{*} (X) = Y$ , then it must have the lowest loss, i.e., for all classifier $h$ ,

L_{DMI} (Q_{h^{*} (X), ~ Y}) \leq L_{DMI} (Q_{h (X), ~ Y})

and the inequality is strict when $h (X)$ is not a permutation of $h^{*} (X)$ , i.e., there does not exist a permutation $π : C \mapsto C$ s.t. $h (x) = π (h^{*} (x)), \forall x \in X$ ;

noise-robust

for the set of all possible classifiers $H$ ,

a r g m i n h \in H L_{DMI} (Q_{h (X), ~ Y}) = a r g m i n h \in H L_{DMI} (Q_{h (X), Y})

and in fact, training using noisy labels is the same as training using clean labels in the dataset except a constant shift,

L_{DMI} (Q_{h (X), ~ Y}) = L_{DMI} (Q_{h (X), Y}) + α;

information-monotone

for every two classifiers $h, h^{'}$ , if $h^{'} (X)$ is less informative for $Y$ than $h (X)$ , i.e. $h^{'} (X)$ is independent of $Y$ conditioning on $h (X)$ , then

L_{DMI} (Q_{h (X), ~ Y}) \leq L_{DMI} (Q_{h (X), Y}) .

Proof.

The relatively invariance of $DMI$ (Theorem 3.5) implies

DMI (h (X), ~ Y) = DMI (h (X), Y) | det (T_{Y \to ~ Y}) | .

Therefore,

L_{DMI} (Q_{h^{*} (X), ~ Y}) = L_{DMI} (Q_{h^{*} (X), Y}) + log (| det (T_{Y \to ~ Y}) |) .

Thus, the information-monotonicity and the noise-robustness of $L_{DMI}$ follows and the constant $α = log (| det (T_{Y \to ~ Y}) |) \leq 0$ .

The legal property follows from the information-monotonicity of $L_{DMI}$ as $h^{*} (X) = Y$ is the most informative random variable for $Y$ itself and the fact that for every square transition matrix $T$ , $det (T) = 1$ if and only if $T$ is a permutation matrix Seneta (2006). ∎

5 Experiments

We evaluate our method on both synthesized and real-world noisy datasets with different deep neural networks to demonstrate that our method is independent of both architecture and data domain. We call our method DMI and compare it with: CE (the cross entropy loss), FW (the forward loss Patrini et al. (2017)), GCE (the generalized cross entropy loss Zhang and Sabuncu (2018)) and LCCN (the latent class-conditional noise model Yao et al. (2019)). For the synthesized data, noises are added to the training and validation sets, and test accuracy is computed with respect to true labels. For our method, we pick the best learning rate from ${1.0 \times 10^{- 4}, 1.0 \times 10^{- 5}, 1.0 \times 10^{- 6}}$

based on the minimum validation loss. For other methods, we use the best hyperparameters they provided in similar settings. The classifiers are pretrained with cross entropy loss first. All reported experiments were repeated five times. We implement all networks and training procedures in Pytorch

Paszke et al. (2017) and conduct all experiments on NVIDIA TITAN Xp GPUs. The explicit noise transition matrix and the experimental data for each synthesized case are shown in Appendix C and Appendix D.

5.1 Experiments on Fashion-MNIST

Fashion-MNIST Xiao et al. (2017) consists of 70,000 $28 \times 28$ grayscale fashion product image from $10$ classes, which is split into a $50, 000$ -image training set, a $10, 000$ -image valiadation set and a $10, 000$ -image test set. We convert the label to two classes, bags and clothes, to synthesize a highly imbalanced dataset ( $10 %$ bags, $90 %$

clothes). We use a simple two-layer convolutional neural network as the classifier. Adam with default parameters and a learning rate of

1.0 \times 10^{- 4}

is used as the optimizer during training. Batch size is set to

128

We synthesize three cases of noise patterns: (1) class-independent: with probability $r$ , a true label is substituted by a random label through uniform sampling. (2) class-dependent (a): with probability $r$ , bags $\to$ clothes, that is, a true label of the a priori less popular class, “bags”, is flipped to the popular one, “clothes”. This happens in real world when the annotators are lazy. (e.g., a careless medical image annotator may be more likely to label “benign” since most images are in the “benign” category.) (3) class-dependent (b): with probability $r$ , clothes $\to$ bags, that is, the a priori more popular class, “clothes”, is flipped to the other one, “bags”. This happens in real world when the annotators are risk-avoid and there will be smaller adverse effects if the annotators label the image to a certain class. (e.g. a risk-avoid medical image annotator may be more likely to label “malignant” since it is usually safer when the annotator is not confident, even if it is less likely a priori.) Note that the parameter $0 \leq r \leq 1$ in the above three cases also represents the amount of noise. When $r = 0$ , the labels are clean and when $r = 1$ , the labels are totally uninformative. Moreover, in case (2) and (3), as $r$ increases, the noise pattern changes from diagonally dominant to diagonally non-dominant.

Figure 2: Test accuracy (mean and std. dev.) on Fashion-MNIST.

Note that CE is distance-based and DMI is information-theoretic. As we mentioned in introduction, CE will perform badly when the noise is non-diagonally dominant and the labels are biased to one class, as distance-based losses prefer the meaningless classifier $h_{0}$ who always outputs the class who is the majority in the labels. ( $\forall x, h_{0} (x) =$ “clothes” and has accuracy $90 %$ in case (2) and $\forall x, h_{0} (x) =$ “bags” and has accuracy $10 %$ in case (3)). The experiment results match our expectation. CE performs similarly with our DMI for diagonally dominant noises. For non-diagonally dominant noises, however, CE only obtains the meaningless classifier $h_{0}$ while DMI still performs pretty well.

5.2 Experiments on CIFAR-10 and Dogs vs. Cats

CIFAR-10 3 consists of 60,000 $32 \times 32$ color images from $10$ classes, which is split into a $40, 000$ training set, a $10, 000$ valiadation set and a $10, 000$ test set. Dogs vs. Cats 5 consists of $25, 000$ images from $2$ classes, dogs and cats, which is split into a $12, 500$ -image training set, a $6, 250$ -image validation set and a $6, 250$ -image test set. We use ResNet-34 as the classifier for CIFAR-10 and VGG-16 as the classifier for Dogs vs. Cats. SGD with a momentum of 0.9, a weight decay of $1.0 \times 10^{- 4}$ and a learning rate of $1.0 \times 10^{- 5}$ is used as the optimizer during training. Batch size is set to $128$ . We use per-pixel normalization, horizontal random flip and $32 \times 32$

random crops after padding with

4

pixels on each side as data augmentation. Following Yao et al. (2019), the noise for CIFAR-10 is added between the similar classes, i.e. truck

\to

automobile, bird

\to

airplane, deer

\to

horse, cat

\to

dog, with probability

r

. The noise for Dogs vs. Cats is added as cat

\to

dog with probability

r

As shown in Figure 3, our method DMI almost outperforms all other methods in every experiment and its accuracy drops slowly as the noise amount increases. GCE has great performance in diagonally dominant noises but it fails in diagonally non-dominant noises. This phenomenon matches its theory: it assumes that the label noise is diagonally dominant. FW needs to pre-estimate a noise transition matrix before training and LCCN uses the output of the model to estimate the true labels. These tasks become harder as the noise amount grows larger, so their performance also drop quickly as the noise amount increases.

Figure 3: Test accuracy (mean and std. dev.) on CIFAR-10 and Dogs vs. Cats.

5.3 Experiments on Clothing1M

Clothing1M Xiao et al. (2015) is a large-scale real world dataset, which consists of 1 million images of clothes collected from shopping websites with noisy labels from 14 classes assigned by the surrounding text provided by the sellers. It has additional 14k and 10k clean data respectively for validation and test. We use ResNet-50 as the classifier and apply random crop of $224 \times 224$ , random flip, brightness and saturation as data augmentation. SGD with a momentum of $0.9$ , a weight decay of $1.0 \times 10^{- 3}$ is used as the optimizer during training. We train the classifier with learning rates of $1.0 \times 10^{- 6}$ in the first $5$ epochs and $0.5 \times 10^{- 6}$ in the second $5$ epochs. Batch size is set to $256$ .

Method	CE	FW	GCE	LCCN	DMI
Accuracy	$68.94$	$70.83$	$69.09$	$71.63$	$72.46$

Table 1: Test accuracy (mean) on Clothing1M

As shown in Table 4, DMI also outperforms other methods in the real-world setting.

6 Conclusion

We propose a simple yet powerful loss function, $L_{DMI}$ , for training deep neural networks robust to label noise. It is based on a generalized version of mutual information, DMI. We provide theoretical validation to our approach and compare our approach experimentally with previous methods on both synthesized and real-world datasets. To the best of our knowledge, $L_{DMI}$ is the first loss function that is provably not sensitive to noise patterns and noise amounts, and can be applied to any existing neutral networks architecture straightforwardly without any auxiliary information.

References

J. P. Brooks (2011) Support vector machines with the ramp loss and the hard margin loss. Operations research 59 (2), pp. 467–479. Cited by: §2.
P. Cao, Y. Xu, Y. Kong, and Y. Wang (2018) Max-mig: an information theoretic approach for joint learning from crowds. Cited by: §3.2.
[3] CIFAR-10 and CIFAR-100 datasets. Note: https://www.cs.toronto.edu/~kriz/cifar.html2009 Cited by: §5.2.
I. Csiszár, P. C. Shields, et al. (2004) Information theory and statistics: a tutorial. Foundations and Trends® in Communications and Information Theory 1 (4), pp. 417–528. Cited by: Fact 3.3.
[5] Dogs vs. Cats competition. Note: https://www.kaggle.com/c/dogs-vs-cats2013 Cited by: §5.2.
A. Ghosh, H. Kumar, and P. Sastry (2017) Robust loss functions under label noise for deep neural networks. In
Thirty-First AAAI Conference on Artificial Intelligence
, Cited by: §1, §2, §3.1, §3.1.
J. Goldberger and E. Ben-Reuven (2016) Training deep neural-networks using a noise adaptation layer. Cited by: §2.
B. Han, J. Yao, G. Niu, M. Zhou, I. Tsang, Y. Zhang, and M. Sugiyama (2018a) Masking: a new perspective of noisy supervision. In Advances in Neural Information Processing Systems, pp. 5836–5846. Cited by: §2.
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama (2018b) Co-teaching: robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, pp. 8527–8537. Cited by: §2.
K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition
, pp. 770–778. Cited by: §1.
D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel (2018) Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems, pp. 10456–10465. Cited by: §1, §2.
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, A. Trischler, and Y. Bengio (2018) Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670. Cited by: §3.2.
L. Jiang, Z. Zhou, T. Leung, L. Li, and L. Fei-Fei (2017) Mentornet: regularizing very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055 4. Cited by: §2.
A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §1.
K. Lee, X. He, L. Zhang, and L. Yang (2018)
Cleannet: transfer learning for scalable image classifier training with label noise
. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5447–5456. Cited by: §2.
X. Ma, Y. Wang, M. E. Houle, S. Zhou, S. M. Erfani, S. Xia, S. Wijewickrema, and J. Bailey (2018) Dimensionality-driven learning with noisy labels. arXiv preprint arXiv:1806.02612. Cited by: §2.
N. Manwani and P. Sastry (2013) Noise tolerance under risk minimization. IEEE transactions on cybernetics 43 (3), pp. 1146–1151. Cited by: §2.
H. Masnadi-Shirazi and N. Vasconcelos (2009)
On the design of loss functions for classification: theory, robustness to outliers, and savageboost
. In Advances in neural information processing systems, pp. 1049–1056. Cited by: §2.
N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari (2013) Learning with noisy labels. In Advances in neural information processing systems, pp. 1196–1204. Cited by: §2.
A. Paszke, S. Gross, S. Chintala, and G. Chanan (2017) Tensors and dynamic neural networks in python with strong gpu acceleration. Pytorch. Cited by: §5.
G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu (2017) Making deep neural networks robust to label noise: a loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944–1952. Cited by: §1, §2, §3.1, §3.1, §5.
S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich (2014) Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596. Cited by: §2.
M. Ren, W. Zeng, B. Yang, and R. Urtasun (2018) Learning to reweight examples for robust deep learning. arXiv preprint arXiv:1803.09050. Cited by: §2.
E. Seneta (2006)
Non-negative matrices and markov chains
. Springer Science & Business Media. Cited by: §3.2, §4.2.
C. E. Shannon (1948) A mathematical theory of communication. Bell System Technical Journal 27 (3), pp. 379–423. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.1538-7305.1948.tb01338.x Cited by: §3.2.
S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, and R. Fergus (2014) Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080. Cited by: §2.
D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa (2018) Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552–5560. Cited by: §2.
A. Vahdat (2017) Toward robustness against label noise in training deep discriminative neural networks. In Advances in Neural Information Processing Systems, pp. 5596–5605. Cited by: §2.
B. Van Rooyen, A. Menon, and R. C. Williamson (2015) Learning with symmetric label noise: the importance of being unhinged. In Advances in Neural Information Processing Systems, pp. 10–18. Cited by: §2.
A. Veit, N. Alldrin, G. Chechik, I. Krasin, A. Gupta, and S. Belongie (2017) Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 839–847. Cited by: §2.
H. Xiao, K. Rasul, and R. Vollgraf (2017) External Links: cs.LG/1708.07747 Cited by: §5.1.
T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang (2015) Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691–2699. Cited by: §2, §5.3.
J. Yao, H. Wu, Y. Zhang, I. W. Tsang, and J. Sun (2019) Safeguarded dynamic label regression for noisy supervision. Cited by: §2, §5.2, §5.
K. Yi and J. Wu (2019) Probabilistic end-to-end noise correction for learning with noisy labels. arXiv preprint arXiv:1903.07788. Cited by: §2.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals (2016) Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530. Cited by: §2.
Z. Zhang and M. Sabuncu (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778–8788. Cited by: §1, §2, §3.1, §3.1, §5.

Appendix A Training Process

0: A training dataset

D = {(x_{i}, {~ y}_{i})}_{i = 1}^{D}

, a validation dataset

V = {(x_{i}, {~ y}_{i})}_{i = 1}^{V}

, a classifier modeled by deep neural network

h_{Θ}

, the running epoch number

T

, the learning rate

γ

and the batch size

N

1: Pretrain the classifier

h_{Θ}

on the dataset

D

with cross entropy loss

2: Initialize the best classifier:

h_{Θ^{*}} \leftarrow h_{Θ}

3: Randomly sample a batch of samples

B_{v} = {(x_{i}, {~ y}_{i})}_{i = 1}^{N}

from the validation dataset

4: Initialize the minimum validation loss:

L^{*} \leftarrow L_{DMI} (B_{v}; h_{Θ})

5: for epoch

t = 1 \to T

6: for batch

b = 1 \to ⌈ D / B ⌉

7: Randomly sample a batch of samples

B_{t} = {(x_{i}, {~ y}_{i})}_{i = 1}^{N}

from the training dataset

8: Compute the training loss:

L \leftarrow L_{DMI} (B_{t}; h_{Θ})

9: Update

Θ

Θ \leftarrow Θ - γ \frac{\partial L}{\partial Θ}

10: end for

11: Randomly sample a batch of samples

B_{v} = {(x_{i}, {~ y}_{i})}_{i = 1}^{N}

from the validation dataset

12: Compute the validation loss:

L \leftarrow L_{DMI} (B_{v}; h_{Θ})

13: if

L < L^{*}

then

14: Update the minimum validation loss:

L^{*} \leftarrow L

15: Update the best classifier:

h_{Θ^{*}} \leftarrow h_{Θ}

16: end if

17: end for

18: return the best classifier

h_{Θ^{*}}

Algorithm 1 The training process with

L_{DMI}

Appendix B Other Proofs

Claim B.1.

E U_{c ~ c} = Pr [h (X) = c, ~ Y = ~ c]

where

U_{c ~ c} := \frac{1}{N} N \sum i = 1 O_{c i} L_{i ~ c} = \frac{1}{N} N \sum i = 1 h (x_{i})_{c} 1 [{~ y}_{i} = ~ c] .

Proof.

Recall that the randomness of $h (X)$ comes from both $h$ and $X$ and the randomness of $h$ is independent of everything else.

$E U_{c ~ c}$	$= E \frac{1}{N} N \sum i = 1 h (x_{i})_{c} 1 [{~ y}_{i} = ~ c]$
	$= E_{X, ~ Y} h (X)_{c} 1 [~ Y = ~ c]$	(i.i.d. samples)
	$= \sum x, ~ y Pr [X = x, ~ Y = ~ y] h (x)_{c} 1 [~ y = ~ c]$
	$= \sum x Pr [X = x, ~ Y = ~ c] h (x)_{c}$
	$= \sum x Pr [X = x, ~ Y = ~ c] Pr [h (X) = c \| X = x]$	(definition of randomized classifier)
	$= \sum x Pr [X = x, ~ Y = ~ c] Pr [h (X) = c \| X = x, ~ Y = ~ c]$	(fixing $x$ , the randomness of $h$ is independent of everything else)
	$= Pr [h (X) = c, ~ Y = ~ c] .$

∎

Claim B.2.

Under the the performance measure based on Shannon mutual information, the measurement based on noisy labels $MI (h (X), ~ Y)$ is not consistent with the measurement based on true labels $MI (h (X), Y)$ . i.e., for every two classifiers $h$ and $h^{'}$ ,

Proof.

See a counterexample:

The matrix format of the joint distribution $Q_{h (X), Y}$ is $Q_{h (X), Y} = [\begin{matrix} 0.1 & 0.4 0.2 & 0.3 \end{matrix}]$ , the matrix format of the joint distribution $Q_{h^{'} (X), Y}$ is $Q_{h^{'} (X), Y} = [\begin{matrix} 0.2 & 0.6 0.1 & 0.1 \end{matrix}]$ and the noise transition matrix is $T_{Y \to ~ Y} = [\begin{matrix} 0.8 & 0.2 0.4 & 0.6 \end{matrix}]$ .

Given these conditions, $Q_{h (X), ~ Y} = [\begin{matrix} 0.24 & 0.26 0.28 & 0.22 \end{matrix}]$ and $Q_{h^{'} (X), ~ Y} = [\begin{matrix} 0.40 & 0.40 0.12 & 0.08 \end{matrix}]$ .

If we use Shannon mutual information as the performance measure,

MI (h (X), Y) = 2.4157 \times 10^{- 2},

MI (h^{'} (X), Y) = 2.2367 \times 10^{- 2},

MI (h (X), ~ Y) = 3.2085 \times 10^{- 3},

MI (h^{'} (X), ~ Y) = 3.2268 \times 10^{- 3} .

Thus we have $MI (h (X), Y) > MI (h^{'} (X), Y)$ but $MI (h (X), ~ Y) < MI (h^{'} (X), ~ Y)$ .

Therefore,

∎

Appendix C Noise Transition Matrices

On Fashion-MNIST, case (1): $T_{Y \to ~ Y} = [\begin{matrix} 1 - \frac{r}{2} & \frac{r}{2} \frac{r}{2} & 1 - \frac{r}{2} \end{matrix}]$ ;

On Fashion-MNIST, case (2): $T_{Y \to ~ Y} = [\begin{matrix} 1 - r & r 0 & 1 \end{matrix}]$ ;

On Fashion-MNIST, case (3): $T_{Y \to ~ Y} = [\begin{matrix} 1 & 0 r & 1 - r \end{matrix}]$ ;

On CIFAR-10, $T_{Y \to ~ Y} = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 r & 0 & 1 - r & 0 & 0 & 0 & 0 & 0 & 0 & 0 0 & 0 & 0 & 1 - r & 0 & r & 0 & 0 & 0 & 0 0 & 0 & 0 & 0 & 1 - r & 0 & 0 & r & 0 & 0 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 0 & r & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 - r \end{matrix} ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦$ ;

On Dogs vs. Cats, $T_{Y \to ~ Y} = [\begin{matrix} 1 & 0 r & 1 - r \end{matrix}]$ .

For Fashion-MNIST, case (1), $r = 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0, 9$ are diagonally dominant noises.

For other cases, $r = 0.0, 0.1, 0.2, 0.3, 0.4$ are diagonally dominant noises and $r = 0.5, 0.6, 0.7, 0.8, 0, 9$ are diagonally non-dominant noises.

Appendix D Experimental Data

$r$	(1), CE	(1), DMI	(2), CE	(2), DMI	(3), CE	(3), DMI
$0.0$	$99.58 \pm 0.08$	$99.57 \pm 0.08$	$99.58 \pm 0.08$	$99.57 \pm 0.08$	$99.58 \pm 0.08$	$99.57 \pm 0.08$
$0.1$	$99.39 \pm 0.07$	$99.37 \pm 0.06$	$99.47 \pm 0.03$	$99.27 \pm 0.09$	$99.39 \pm 0.07$	$99.17 \pm 0.07$
$0.2$	$99.35 \pm 0.09$	$99.28 \pm 0.14$	$99.13 \pm 0.13$	$99.23 \pm 0.16$	$99.27 \pm 0.10$	$99.07 \pm 0.10$
$0.3$	$99.19 \pm 0.07$	$99.12 \pm 0.17$	$99.06 \pm 0.18$	$99.15 \pm 0.07$	$98.97 \pm 0.19$	$98.84 \pm 0.15$
$0.4$	$99.07 \pm 0.09$	$99.00 \pm 0.19$	$98.23 \pm 0.83$	$99.05 \pm 0.13$	$98.23 \pm 0.35$	$98.69 \pm 0.26$
$0.5$	$98.97 \pm 0.09$	$99.01 \pm 0.08$	$94.33 \pm 2.14$	$98.75 \pm 0.42$	$73.43 \pm 14.09$	$98.58 \pm 0.16$
$0.6$	$98.79 \pm 0.06$	$98.83 \pm 0.05$	$90.09 \pm 0.10$	$98.65 \pm 0.14$	$10.08 \pm 0.07$	$98.42 \pm 0.30$
$0.7$	$98.42 \pm 0.27$	$98.44 \pm 0.26$	$90.04 \pm 0.04$	$98.24 \pm 0.57$	$10.00 \pm 0.00$	$98.18 \pm 0.24$
$0.8$	$97.95 \pm 0.34$	$97.97 \pm 0.33$	$90.01 \pm 0.01$	$98.37 \pm 0.43$	$10.00 \pm 0.00$	$98.08 \pm 0.50$
$0.9$	$96.47 \pm 1.27$	$96.45 \pm 1.28$	$90.00 \pm 0.00$	$97.69 \pm 0.51$	$10.00 \pm 0.00$	$97.32 \pm 0.82$

Table 2: Test accuracy on Fashion-MNIST (mean

\pm

std. dev.)

$r$	CE	FW	GCE	LCCN	DMI
$0.0$	$93.29 \pm 0.18$	$93.12 \pm 0.16$	$93.43 \pm 0.24$	$92.47 \pm 0.36$	$93.37 \pm 0.20$
$0.1$	$91.63 \pm 0.32$	$91.54 \pm 0.15$	$91.96 \pm 0.09$	$91.88 \pm 0.23$	$92.08 \pm 0.08$
$0.2$	$90.36 \pm 0.24$	$90.10 \pm 0.22$	$90.87 \pm 0.16$	$91.05 \pm 0.43$	$91.51 \pm 0.17$
$0.3$	$88.79 \pm 0.40$	$88.77 \pm 0.36$	$89.67 \pm 0.21$	$89.88 \pm 0.40$	$91.12 \pm 0.30$
$0.4$	$84.76 \pm 0.98$	$84.78 \pm 1.53$	$86.6 \pm 0.47$	$89.33 \pm 0.58$	$90.41 \pm 0.32$
$0.5$	$74.81 \pm 3.37$	$77.2 \pm 4.19$	$78.53 \pm 1.93$	$88.30 \pm 0.38$	$89.45 \pm 0.99$
$0.6$	$64.61 \pm 0.72$	$71.98 \pm 1.83$	$71.14 \pm 0.78$	$86.89 \pm 0.51$	$89.03 \pm 0.69$
$0.7$	$59.15 \pm 0.64$	$67.59 \pm 1.64$	$67.10 \pm 0.82$	$77.50 \pm 0.60$	$88.82 \pm 0.89$
$0.8$	$57.65 \pm 0.28$	$62.22 \pm 1.80$	$62.56 \pm 0.72$	$74.62 \pm 1.16$	$87.46 \pm 0.79$
$0.9$	$57.46 \pm 0.08$	$58.23 \pm 0.25$	$58.91 \pm 0.46$	$61.32 \pm 1.87$	$85.94 \pm 0.74$

Table 3: Test accuracy on CIFAR-10 (mean

\pm

std. dev.)

$r$	CE	FW	GCE	LCCN	DMI
$0.0$	$88.50 \pm 0.60$	$89.66 \pm 0.63$	$94.06 \pm 0.41$	$90.41 \pm 0.38$	$90.21 \pm 0.27$
$0.1$	$85.87 \pm 0.79$	$85.87 \pm 0.54$	$92.75 \pm 0.50$	$87.72 \pm 0.46$	$87.99 \pm 0.41$
$0.2$	$82.50 \pm 0.96$	$83.20 \pm 0.83$	$88.94 \pm 0.70$	$84.80 \pm 0.93$	$85.88 \pm 0.83$
$0.3$	$79.11 \pm 1.08$	$78.71 \pm 1.97$	$81.34 \pm 3.23$	$83.16 \pm 1.18$	$84.61 \pm 0.98$
$0.4$	$73.05 \pm 0.20$	$72.13 \pm 2.42$	$70.13 \pm 3.59$	$81.06 \pm 1.05$	$82.52 \pm 1.01$
$0.5$	$57.46 \pm 3.71$	$67.50 \pm 3.99$	$58.31 \pm 1.19$	$76.88 \pm 2.97$	$81.50 \pm 1.19$
$0.6$	$49.98 \pm 0.15$	$64.58 \pm 5.21$	$50.39 \pm 0.47$	$68.50 \pm 3.40$	$80.00 \pm 0.72$
$0.7$	$49.83 \pm 0.09$	$62.87 \pm 6.82$	$49.76 \pm 0.00$	$66.10 \pm 2.45$	$77.01 \pm 1.07$
$0.8$	$49.80 \pm 0.03$	$52.44 \pm 1.52$	$49.76 \pm 0.00$	$65.93 \pm 2.76$	$75.01 \pm 0.88$
$0.9$	$49.77 \pm 0.01$	$50.56 \pm 1.32$	$49.76 \pm 0.00$	$64.29 \pm 1.46$	$67.96 \pm 1.45$

Table 4: Test accuracy on Dogs vs. Cats (mean

\pm

std. dev.)

L_DMI: An Information-theoretic Noise-robust Loss Function

Related Research

Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systems

Improving Generalization by Controlling Label-Noise Information in Neural Network Weights

Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

The Fourier Loss Function

A Committee of Convolutional Neural Networks for Image Classication in the Concurrent Presence of Feature and Label Noise

Mutual Information for Explainable Deep Learning of Multiscale Systems

Code Repositories

L_DMI

PeerLoss-Keras

1 Introduction

2 Related Work

3 Preliminaries

3.1 Problem settings

Assumption 3.1 (Independent noise).

Assumption 3.2 (Informative noisy label).

3.2 Information theory concepts

Fact 3.3 (Information-monotonicity Csiszár et al. (2004)).

Definition 3.4 (Determinant based Mutual Information).

Theorem 3.5 (Properties of DMI).

Proof.

4 LDMI: An Information-theoretic Noise-robust Loss Function

4.1 Method overview

4.2 Theoretical justification

Theorem 4.1 (Main Theorem).

Proof.

5 Experiments

5.1 Experiments on Fashion-MNIST

5.2 Experiments on CIFAR-10 and Dogs vs. Cats

5.3 Experiments on Clothing1M

6 Conclusion

References

Appendix A Training Process

Appendix B Other Proofs

Claim B.1.

Proof.

Claim B.2.

Proof.

Appendix C Noise Transition Matrices

Appendix D Experimental Data

4 $L_{DMI}$ : An Information-theoretic Noise-robust Loss Function