Recently, a plethora of approaches to Mixed Sample Data Augmentation (MSDA) have been proposed which obtain state-of-the-art results, particularly in classification tasks (chawla2002smote; zhang2017mixup; tokozume2017learning; tokozume2018between; inoue2018data; yun2019cutmix; takahashi2019data; summers2019improved). MSDA involves combining data samples according to some policy to create an augmented data set (purely in the sense that it has a greater number of examples) on which to train the model. Explanations of the performance of MSDA methods have thus far failed to reach a consensus, either presenting opposing views, as is the case with liang2018understanding, zhang2017mixup, and he2019data, or justifying the effect of a specific MSDA from a perspective that is not sufficiently broad to provide insight about other methods (devries2017improved; tokozume2018between; guo2019mixup).
Traditionally, augmentation is viewed through the framework of statistical learning as Vicinal Risk Minimisation (VRM) (vapnik2013nature; chapelle2001vicinal). Given some notion of the vicinity of a data point, VRM trains with vicinal samples in addition to the data points themselves. This is the original motivation for MixUp (zhang2017mixup)
; to provide a new notion of vicinity based on mixing data samples. There are two key limitations of an analysis based purely on VRM and statistical learning. Firstly, although VRM provides a helpful basis for MSDA, it fails to characterise the effect of a particular approach on trained models. Secondly, VRM does not endow us with a good sense of what the right vicinal distribution is, despite the fact that this is undoubtedly the key factor which determines success. A theory that may help to counteract the former limitation is the information bottleneck theory of deep learning(tishby2015deep). This theory uses the data processing inequality, summarised as ‘post-processing cannot increase information’, to characterise the functions learned by deep networks. Specifically, tishby2015deep suggest that deep networks progressively discard information about the input whilst preserving information about the targets. An information theoretic viewpoint may also help to ameliorate the unsatisfactory conception of a good vicinal distribution. For example, one might argue that the best notion of vicinity is one which leads to the most compressed, general representations. Alternatively, a better notion of vicinity might be one for which functions learned through VRM capture the same information as those learned when minimising the empirical risk (training on the original data).
We expect that an information theoretic analysis will help to explain how MSDA approaches such as MixUp (zhang2017mixup) and CutMix (yun2019cutmix)
are both able to provide good regularisation despite stark qualitative differences; MixUp interpolates between samples whereas CutMix uses a binary mask to insert a square region from one data point into the other. We posit that MixUp inhibits the ability to learn about example specific features in the data, inducing more compressed representations. In contrast, we suppose that CutMix causes learned models to retain a good knowledge of the real data, since observed features generally only derive from one data point. At the same time CutMix limits the ability of the model to over-fit by dramatically increasing the number of observable data points, in keeping with the original intent of VRM. However, by restricting to only masking a square region, CutMix imposes an unnecessary limitation. Indeed, it should be possible to construct an MSDA which uses masking similar to CutMix whilst increasing the data space much more dramatically.
In this paper we build on the above basis to introduce FMix, a masking MSDA which allows masks of arbitrary shapes whilst retaining the desirable properties of CutMix. We demonstrate performance of FMix for a range of models and tasks against a series of baselines and other MSDA approaches. FMix obtains a new state-of-the-art performance on CIFAR-10 (krizhevsky2009learning) without external data and Fashion MNIST (xiao2017fashion) and improves the performance of several state-of-the-art models (ResNet, DenseNet, WideResNet and PyramidNet) on a range of problems and modalities. We subsequently analyse MixUp, CutMix and FMix under the lens of information theory to provide insight on precisely how they give rise to improved generalisation performance. In particular, we introduce a quantity which captures the extent to which an unsupervised model learns to encode the same information from the augmented data as from the real data. This analysis gives us cause to present a unified account of MSDA which suggests that interpolating approaches such as MixUp differ fundamentally from masking approaches such as FMix in their action on learning models, and ultimately in how they yield better generalisation. We find that interpolation causes early compression, biasing models to more general features, and that masking preserves the distribution of semantic constructs in the data, more appropriately fitting the classical definition of an augmentation.
2 Related Work: MSDA With a Binary Mask
In this section, we review the fundamentals of masking MSDAs that will form the basis of our motivation. Let denote the input data distribution. In general, we can define MSDA for a given mixing function, , where and
are independent random variables on the data domain andis the mixing coefficient.
Synthetic minority over-sampling (chawla2002smote), a predecessor to modern MSDA approaches, can be seen as a special case of the above where and are dependent, jointly sampled as nearest neighbours in feature space. These synthetic samples are drawn only from the minority class to be used in conjunction with the original data, addressing the problem of imbalanced data. The mixing function is linear interpolation, , and . More recently, zhang2017mixup, tokozume2017learning, tokozume2018between and inoue2018data concurrently proposed using this formulation (as MixUp, Between-Class (BC) learning, BC+ and sample pairing respectively) on the whole data set, although the choice of distribution for the mixing coefficients varies for each approach. We refer to this as interpolative MSDA. However, following zhang2017mixup
we can obtain sufficient flexibility with the symmetric Beta distribution, that is.
Recent variants adopt a binary masking approach (yun2019cutmix; summers2019improved; takahashi2019data). Let be a random variable with and , that is, the average value of a generated mask should be equal to the mixing coefficient. The mask mixing function is
where denotes point-wise multiplication. A notable variant of this which motivates our approach is CutMix (yun2019cutmix). This approach is specifically designed for two dimensional data, , which uses
where yields a binary mask with a shaded rectangular region of size at a uniform random coordinate. CutMix improves upon the performance of MixUp on a range of experiments.
In all MSDA approaches the targets are mixed in some fashion, typically to reflect the mixing of the inputs. For classification, both interpolative and masking strategies mix the targets according to the interpolation mixing function from above. This is typically used with a cross entropy loss such that the MSDA classification objective is written
where is the distribution learned by a model, and and are the ground truth targets of and respectively. It could be suggested that by mixing the targets differently, one might obtain better results than with the standard formulation. However, there are two key observations from prior art which give us cause to doubt this supposition. First, it can be shown that the above objective is equivalent to using the original target from one of the data points but with a different distribution for . Specifically, as pointed out in huszar2017, the objective can be written
where . Note that
is skewed towards high values, only balancing in the limit. For clarity, we reproduce the derivation from huszar2017 in Section A.1 of the appendix. We also demonstrate that although the two objectives take the same value in expectation, they do not achieve the same result in practice, with the reformulation consistently performing worse than the standard objective. For this reason, we use the standard objective in our experiments. That said, this reformulation undermines any suggestion that we are embedding some information in the targets since they remain unchanged. The second observation derives from the empirical study by liang2018understanding, where a number of experiments on the importance of the mixing ratio of the labels in MixUp were performed. On the one hand, they concluded that when the targets are not mixed in the same proportion as the inputs the model can be regularised to the point of underfitting. However, despite this conclusion their results show only a mild performance change even in the extreme event that targets are mixed randomly, independent of the inputs. In light of these findings, it is appropriate to suggest that the most important element of MSDA is the input mixing function. We provide some additional exposition on our viewpoint in Section A.2 of the appendix. For the remainder of the paper we focus on the inputs and the development of a better mixing function.
3 FMix: Improved Masking
To form the motivation for our approach, it is now important to understand precisely why CutMix is so effective. We view current masking MSDAs, such as CutMix, as equivalent for the purposes of our analysis since they all fundamentally mix rectangular regions. Our contention is that the masking MSDA approach works because it effectively preserves the data distribution in a way that interpolative MSDAs do not, particularly in the perceptual space of a Convolutional Neural Network (CNN). Specifically, each convolutional neuron generally encodes information from only one of the inputs at a time. This could also be viewed as local consistency in the sense that elements that are close to each other in space typically derive from the same data point. To the detriment of CutMix, it would be easy for a model to learn about the augmentation since perfectly horizontal and vertical artefacts are unlikely to be a salient feature of the data. If we can increase the number and complexity of masks then the space of novel features (that is, features which occur due to edges in the mask) would become significantly larger than the space of features native in the data. As a result, it is highly unlikely that a model would be able to ‘fit’ to this information since it would require exponentially greater capacity. This leads to our core motivation: to construct a masking MSDA which maximises the space of edge shapes whilst preserving local consistency.
For local consistency, we require masks that are predominantly made up of a single shape or contiguous region. We might think of this as trying to minimise the number of times the binary mask transitions from ‘’ to ‘’ or vice-versa. For our approach, we begin by sampling a low frequency grey-scale mask from Fourier space which can then be converted to binary with a threshold. We will first detail our approach to obtaining the low frequency image before discussing our approach for choosing the threshold. Let denote a complex random variable with values on the domain , with density and , where and return the real and imaginary parts of their input respectively. Let denote the magnitude of the sample frequency corresponding to the , ’th bin of the
discrete Fourier transform. We can apply a low pass filter toby decaying its high frequency components. Specifically, for a given decay power , we use
Defining as the inverse discrete Fourier transform, we can obtain a grey-scale image with
All that now remains is to convert the grey-scale image to a binary mask such that the mean value is some given . Let return a set containing the top elements of the input . Setting the top elements of some grey-scale image to have value ‘’ and all others to have value ‘’ we obtain a binary mask with mean . Specifically, we have
Example mask and mixed images from ImageNet for FMix withand .
To recap, we first sample a random complex tensor for which both the real and imaginary part are independent and Gaussian. We then scale each component according to its frequency via the parametersuch that higher values of correspond to increased decay of high frequency information. Next, we perform an inverse Fourier transform on the complex tensor and take the real part to obtain a grey-scale image. Finally, we set the top proportion of the image to have value ‘’ and the rest to have value ‘’ to obtain our binary mask. Note that although we have only considered two dimensional data in here it is generally possible to create masks with any number of dimensions via our process. We provide pseudo-code for this algorithm in Listing F and some example two dimensional masks and mixed images (with and ) in Figure 1. From the figure we can see that although the local consistency is marginally reduced for FMix, the space of artefacts is significantly increased, satisfying our aims.
In this section we perform a series of experiments to demonstrate the efficacy of FMix. For each problem setting and data set, we provide exposition on the results and any caveats we feel are of relevance. Throughout, our approach has been to use the hyper-parameter options which yield the best results in the literature for each setting, but fixed for each of the methods on test. This allows us to ensure comparisons are on an equal footing and that baselines provide a good reflection of real life performance. For hyper-parameter choices specific to the different methods, we have aimed for consistency with their respective papers and implementations. The only exception is the distribution of in MSDAs which, unless otherwise stated uses . For FMix, we use since this was found to produce large artefacts with sufficient diversity. In addition to FMix, MixUp, and CutMix, we compare with CutOut (devries2017improved) and Random Erase (Erase) (zhong2017random)
, two popular regularisers from the literature that can be seen as predecessors to CutMix. We perform repeats where possible and report the average performance and standard deviation following the last epoch of training. A complete discussion of the experimental set-up can be found in Section C of the appendix. In all tables, we give the best result and results that are within its margin of error inbold.
We first discuss image classification results on the CIFAR-10/100 (krizhevsky2009learning) and Fashion MNIST (xiao2017fashion) data sets. We train: PreAct ResNet-18 (he2016identity), WideResNet-28-10 (zagoruyko2016wide), DenseNet-BC-190 (huang2017densely) and PyramidNet-272-200 (pyramidnet). For PyramidNet, we additionally apply Fast AutoAugment (lim2019fast) and ShakeDrop (yamada2018shakedrop) following lim2019fast. The results in Table 1 show that FMix consistently offers a significant improvement over the other methods on test. Indeed, there is only one case where FMix does not perform within the margin of error of the best result, and only three other cases where FMix does not provide a significant (outside the margin of error) improvement over the next best result. In the PyramidNet setting, FMix achieves, to the best of our knowledge, the current state of the art result on CIFAR-10 without use of external data and Fashion MNIST.
Next, we obtain classification results on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC2012) and variants (ILSVRC15). We train a ResNet-101 on the full data set (ImageNet), before evaluating on ImageNet-a (hendrycks2019natural), a set of natural adversarial examples to ImageNet models, to determine adversarial robustness. Subsequently, we report results of a PreAct-ResNet18 on Tiny ImageNet (tinyimagenet). The results for these experiments are given in Table 2. FMix again provides improvement across the board. For ImageNet-a, they demonstrate a clear improvement in adversarial robustness. In particular, the improvement on ImageNet-a is much greater than that for ImageNet. On Tiny ImageNet, FMix significantly improves accuracy, where MixUp fails to do so. The MixUp ImageNet result was lower than the baseline, however, we should note our use of a lower batch size than that used in the paper. Overall, these results paint a clear picture of the improved performance that can be obtained using FMix for image classification.
We now evaluate MixUp and FMix on the Google Commands data set, a speech classification task. We perform FMix on a Mel-frequency spectrogram of each utterance, i.e. a spectrogram with each bin transformed into a corresponding bin in the Mel scale. The results for a PreAct ResNet-18 are given in Table 1. We evaluate FMix and MixUp for the standard used for the majority of our experiments and recommended by zhang2017mixup for MixUp. We see in both cases that FMix improves performance over MixUp outside the margin of error, suggesting that this is a significant result.
The Toxic Comments (toxic)
data set was a Kaggle challenge to classify text into one of 6 toxicity classes (including non-toxic, forming by far the majority). The continue to use the ROC-AUC metric of the competition. We embed each sentence using a FastText-300d word embedding before generating 1D FMix masks to mix sentences. We see from Table1 that on average, FMix offers improvement over the baseline, although it is only one standard deviation away. By comparison, note that MixUp actively harms performance in this task, which might be unsurprising given that it is unclear how interpolated words are represented in the embedding.
Point Cloud Classification
We now demonstrate the extension of FMix to 3D through point cloud classification on ModelNet10 (modelnet10). We transform the pointclouds to a voxel representation before applying a 3D FMix mask. Table 1 reports the average median accuracy from the last 5 epochs, due to large variability in the results. It shows that FMix continues to improve results within significance, even in higher dimensions.
Figure 1(a) gives the relationship between validation accuracy and the parameter for three MSDA methods. Validation accuracy is the average over folds with a validation set consisting of of the data. This ablation was performed on the CIFAR-10 data set using the PreAct ResNet18 model from the previous experiments. In the cases of FMix and MixUp there exists an optimal value. In both cases, this point is close to , although for MixUp it is skewed slightly toward 0, as was found for their ImageNet experiments. The choice of decay power is certainly more significant. Figure 1(b) shows that low values of drastically reduce the final accuracy. This is unsurprising since low corresponds to a speckled mask, with no large regions of either data point present in the augmentation. Larger values of correspond to smoother marks with large cohesive regions from each donor image. We note that for there is little improvement to be gained, validating our decision to use .
5 Previous Attempts to Explain MSDA
Attempts to explain the success of MSDAs were not only made when they were introduced, but also through subsequent empirical and theoretical studies. In this section we review these studies to paint a picture of the current theories, and points of contention, on how MSDA works. In addition to their experimentation with the targets discussed in Section 2, liang2018understanding argue that linear interpolation of inputs limits the memorisation ability of the network. A somewhat more mathematical view on MSDA was adopted by guo2019mixup, who argue that MixUp regularises the model by constraining it outside the data manifold. They point out that this could lead to reducing the space of possible hypotheses, but could also lead to generated examples contradicting original ones, degrading quality.
Following zhang2017mixup, he2019data take a statistical learning view of MSDA, basing their study on the observation that MSDA distorts the data distribution and thus does not perform VRM in the traditional sense. They subsequently propose separating features into ‘minor’ and ‘major’, where a feature is referred to as ‘minor’ if it is highly sample-specific. Augmentations that significantly affect the distribution are said to make the model predominantly learn from ‘major’ features. From an information theoretic perspective, ignoring these ‘minor’ features corresponds to increased compression of the input by the model. Although he2019data noted the importance of characterising the effect of data augmentation from an information perspective, they did not explore any measures that do so. Instead, he2019data
analysed the variance in the learned representations. It can be seen that this is analogous to the entropy of the representation since entropy can be estimated via the pairwise distances between samples, with higher distances corresponding to both greater entropy and variance(kolchinsky2017estimating). In proposing Manifold MixUp, pmlr-v97-verma19a
additionally suggest that MixUp works by increasing compression. The authors compute the singular values of the representations in early layers of trained networks, with smaller singular values again corresponding to lower entropy. The issue with these approaches is that the entropy of the representation is only an upper bound on the information that representation has about the input.
An issue with these findings is that they relate purely to interpolative MSDAs. It also the case that there is disagreement in the conclusions of some of these studies. If interpolative MSDA works by preventing the model from learning about so called ‘minor’ features, then that would suggest that the underlying data distribution has been distorted, breaking the core assumption of VRM. Furthermore, yun2019cutmix suggested that masking MSDA approaches work by addressing this distortion. If this is the case then we should expect them to perform worse than interpolative MSDAs since the bias towards compressed representations has been removed. Clearly, there is some contention about the underlying mechanisms driving generalisation in MSDAs. In particular, it is necessary to provide an explanation for masking MSDAs that is complementary to the current explanations of interpolative MSDAs, rather than contradictory to them.
6 Analysis: Towards A Unified Understanding of MSDA
We now analyse both interpolative and masking MSDAs with a view to presenting a unified explanation of their function. To begin, we propose a measure which captures the extent to which learning about the augmented data corresponds to learning about the original data. In particular, we study the mutual information between learned representations. We find that the effect of interpolation is almost entirely different from that of masking. This finding gives us cause to suspect that using both forms as a dual objective would perform better than just using one form on its own. We verify this in a simple setting where we alternate between two MSDA methods for each iteration of training.
6.1 Measuring Distortion
We want to measure if learning about augmented data simulates learning about real data. To achieve this, we propose training models on real data and augmented data, and then comparing the representations they learn. We first require a measure of similarity between learned representations. A good option here is the mutual information, the reduction in uncertainty about one variable given knowledge of another. In essence, this quantity should capture the amount of information encoded in one representation that is also encoded in the other. It is often challenging to compute the mutual information, since it is difficult to tell the extent to which one random variable is an encoding of another. We can think of this in terms of predictability. In our setting, we wish to estimate the mutual information between a learned representation of the original data set, , and a learned representation of some augmented data set, , written
is the Kullback-Leibler divergence. In this form, we can see that our ability to computedepends on our ability to compute . Now observe that we would require a model at least powerful enough to undo the encoding of from and then re-encode this as a in order to obtain the best possible predictor of . In other words, the more powerful our model of , the more precise it can be and the further this prediction will deviate from the marginal distribution of . The result of this is that we will tend to underestimate the mutual information. Furthermore, our estimate will get worse with increasing complexity of the mapping from to .
We can alleviate the above problem through careful choice of the model to be used in our measure. In particular, we propose using Variational Auto-Encoders (VAEs). These comprise of an encoder, , and a decoder, . We impose a standard Normal prior on , train the model to maximise the Evidence Lower BOund (ELBO) objective
There are three key motivations for this choice. First, the representation learned by a VAE gives a rich depiction of the salient or compressible information in the data. Secondly, is somewhat easier to compute when the s are modelled by VAEs. Denoting the outputs of the decoder of the VAE trained on the augmentation as , and by the data processing inequality, we have with equality when the decoder retains all of the information in . Now, we need only observe that we already have a model of , the encoder trained on the augmented data. Estimating the marginal presents a challenge as it is a Gaussian mixture. However, we can measure an alternative form of the mutual information that is equivalent up to an additive constant, and for which the divergence has a closed form solution, with
The above holds for any choice of distribution that does not depend on . Conceptually, this states that we will always lose more information on average if we approximate with any constant distribution other than the marginal . Additionally note that we implicitly minimise during training of the VAE (hoffman2016elbo). In light of this fact, we can write . The third and final advantage of using VAEs is that we can easily obtain a helpful upper bound of such that it is bounded on both sides. Since is just a function of , again by the data processing inequality, we have . This is easy to compute since it is just the relative entropy term from the ELBO objective.
To summarise, we can compute our measure by first training two VAEs, one on the original data and one on the augmented data. We then generate reconstructions of data points in the original data with one VAE and encode them in the other. We now compute the expected value of the relative entropy between the encoded distribution and an estimate of the marginal to obtain an estimate of a lower bound of the mutual information between the representations. We then recompute this using real data points instead of reconstructions to obtain an upper bound. Table 3 gives the above quantities for MixUp, FMix, CutMix, and a baseline. The results show that MixUp consistently reduces the amount of information that is learned about the original data. In contrast, FMix and CutMix both manage to induce greater mutual information with the data than is obtained from training on the real data. The significance of this result is to present concrete evidence that interpolative MSDA performs a fundamentally different function from masking MSDA.
Given this difference in function, it is reasonable to expect that we might obtain better performance through simultaneous action of both types of MSDA. Table 4 contains the results obtained on CIFAR-10 with a PreAct-ResNet18 when alternating between two MSDAs between each batch. A combination of interpolation and masking gives the best results, particularly FMix+MixUp (referred to as FMix+). In contrast, combining FMix and CutMix gives worse results than using either method on its own. These findings support the notion that the effects of the two types of MSDA are complementary and can be exploited simultaneously.
To gain a better understanding of the impact these augmentations have on generalisation, it is necessary to study the learned representations in a classification setting. To this end, we visualise the decisions made by a classifier using Gradient-weighted Class Activation Maps (Grad-CAMs) (selvaraju2017grad). Grad-CAM finds the regions in an image that contribute the most to the network’s prediction by taking the derivative of the model’s output with respect to the activation maps and weighting them according to their contribution. Figure 3 shows the Grad-CAMs of models trained with MixUp, FMix, FMix+, and a baseline for a number of CIFAR-10 images. MixUp causes the model to rely on general or ‘major’ features and achieving a greater compression of the input. In contrast, FMix causes the model to rely on specific contextual or ‘minor’ features, maintaining a high mutual information with the input. Alternating between the two results in a balance between these ‘major’ and ‘minor’ features.
Since interpolation methods bias the network towards configurations that rely more heavily on general features, we would expect their impact to be most notable when the number of observations is limited and it is easier to learn ‘minor’ features. We confirm this empirically by varying the size of the CIFAR-10 training set and training with different MSDAs in Figure 4. The results validate our belief that combining interpolative and masking methods brings the best of both worlds. Significantly, training with FMix+ on only of the data achieves the same performance as the baseline does on , equivalent to a three-fold increase in the number of examples.
7 Conclusions and Future Work
In this paper we have introduced FMix, a masking MSDA that preserves local features while greatly increasing the number of mask shapes. Through a comprehensive set of experiments, we have shown that FMix improves classification performance for a series of models, modalities, and dimensionalities. Through our analysis we present a unified explanation of the performance of both interpolative and masking MSDAs: interpolation causes models to rely on general, compressed features; masking causes models to encode the same information as when trained with the original data whilst eliminating memorisation. We further show that the desirable effects of both can be obtained in a single model through an alternating training procedure. This finding is a direct corollary of our analysis, verifying our approach and findings. Although our work is a first step towards reasoning about MSDA under a unified framework, there is still room for further development of our understanding. Indeed, early experiments in this space resulted in several lines of enquiry that ultimately did not bare fruit, which we discuss further in Section E of the appendix. Future work should look to deepen our understanding of the target space, the impact of MSDA on yet unexplored applications, and the improvement that can be gained from combining MSDAs. In addition to these practical developments, a richer mathematical analysis which captures the effects of augmented training will be necessary to make the best use of MSDA in practice.
Appendix A MSDA for Classification
The standard use case for MSDA is classification. In the paper, we outline our viewpoint that the choice of objective function and target mixing is not of particular importance to the results we can obtain with MSDA. In this section we provide additional exposition and experimentation on this perspective.
a.1 MSDA Objective Reformulation
Here, we briefly reproduce the proof from huszar2017 for the reformulation of the MSDA objective. From the paper, we have
Introducing an auxiliary random variable , with , and by the linearity of expectation we can write
where and . Now, again by the linearity of expectation, we can write
The above derives from the fact that a random variable with density will take the values and precisely half of the time each. For the next step, we assume that the mixing function treats the inputs equivalently, that is , and note that we are free to change variable names to give
Finally, since , we obtain the reformulation
We should note that although the above is a valid reformulation of the objective, it may not produce the same results in practice. This is easy to see from the fact that the Monte-Carlo estimates of each, which we use in practice, are not guaranteed to be the same. To demonstrate this point, in Table A.1 we give the result of a PreAct-ResNet18 trained on CIFAR-10 with FMix for both the standard objective and the reformulation. The results show that the reformulation performs significantly worse in this setting. For this reason, we use the standard objective in our experiments. Note that this does not undermine our analysis of the reformulation, since the results would converge for sufficiently large batch size.
a.2 On the Importance of Targets
Following the reformulation from huszar2017 and the experimental evidence from liang2018understanding
, we take the belief that the target space is not of particular importance to classification performance. However, that doesn’t mean that the target space is always insignificant. For example, we might care about how calibrated the outputs are. Calibration is the extent to which an output ‘probability’ corresponds to theactual probability of being correct. Clearly, this is a challenging property to evaluate since we have no notion of ground truth uncertainty in the data. In peterson2019human, the authors suggest using human uncertainty as a baseline on the CIFAR-10 data set. Specifically, peterson2019human introduce the CIFAR-10H data set consisting of human soft-labels for the CIFAR-10 test set. We evaluate a series of PreAct-ResNet18 models trained on CIFAR-10 for their performance on CIFAR-10H in Table A.2. The metric used is the relative entropy of the model outputs with respect to the soft-labels. The results show that the masking MSDA approaches induce a notion of uncertainty that is most similar to that of human observers. An important weakness of this claim derives from the cross entropy objective used to train models. We note that
In other words, the model is jointly required to match the target distribution and minimise the entropy of each output. The result of this is that trained models naturally output very high confidence predictions as an artefact of their training process. The above claim should therefore be taken with a pinch of salt since it is likely that the improved results derive simply from the lower entropy targets and model outputs. Furthermore, we expect that significant improvement would be gained in this test by training MSDA models with a relative entropy objective rather than the cross entropy.
Appendix B Layer-wise Mutual Information
In this section we discus our attempt to measure the layer-wise mutual information with the input to test the hypothesis that MSDA leads to increased compression in learned models. For the model, we use a VGG-11. We made this choice because of the simplicity of the architecture; there are only ten hidden layers for which we have to compute the mutual information. Our hypothesis is that we will see increased compression in models trained with MSDA by studying the mutual information, , between the features at each layer and the input.
Although mutual information can be computed easily in some cases, such as the VAEs used in the paper, it presents a challenge for arbitrary variables. A potential solution can be found through neural estimation. Let denote a neural network with parameters . The Mutual Information Neural Estimator (MINE) (belghazi2018mine) is defined as
In essence, the above states that the mutual information between and depends on the extent to which a model can learn to differentiate between values of that were generated by a given and values that were not. Using stochastic gradient ascent on the above is unstable when the true mutual information is high. This is because the value of the objective
can be made arbitrarily large if it is always possible to tell whether a particular lead to a particular
. To address this issue, we propose a form of adaptive gradient clipping. Specifically, let, the gradient becomes
As such, by placing a non-linearity on the outputs of , we ensure that
We apply the non-linearity only during the gradient ascent process. This limits the space of achievable functions so that the estimate obtained with clipping is a lower bound on the estimate obtained without it.
Figure B.1 gives the layer-wise mutual information for models trained with a range of MSDAs, computed with the above estimator. We perform epochs of gradient ascent with the Adam optimiser with a learning rate of . Since the optimisation is a supremum, we take the highest average value over the test set for any epoch. That is, if at any point we find a function which obtains a certain value, we know the mutual information is at least that high. In a similar vein, if the result for some layer is higher than at a previous layer , we set since, by the data processing inequality, . Following each of these adaptations, we are able to obtain curves which have a low deviation over three repeats. Unfortunately, the results do not show any significant difference between the methods in terms of the compression in the early layers. That said, the results do appear to show an increase in compression in the later layers for models trained with MSDA, particularly CutMix. We suspect that the first few layers are uninformative because the true mutual information is maximal. If the mutual information between the input and the output from a specific filter in the layer is high, then the mutual information between the input and the output from all filters is at least that high. As such, the mutual information only really tells us about the least compressed feature.
Appendix C Experimental Details
|Experiment||Model||Epochs||Schedule||Batch Size||Learning Rate|
|CIFAR-10 / 100||PreAct ResNet18||200||100, 150||128||0.1|
|DenseNet-BC-190||300||100, 150, 225||32||0.1|
|FashionMNIST||PreAct ResNet18||200||100, 150||128||0.1|
|WideResNet-28-10||300||100, 150, 225||32||0.1|
|DenseNet-BC-190||300||100, 150, 225||32||0.1|
|Google Commands||PreAct ResNet18||90||30, 60, 80||128||0.1|
|ImageNet||ResNet101||90||30, 60, 80||256||0.4|
|TinyImageNet||PreAct ResNet18||200||150, 180||128||0.1|
|ModelNet10||PointNet||50||10, 20, 30, 40||16||0.001|
|Ablations||PreAct ResNet18||200||100, 150||128||0.1|
|Combining MSDAs||PreAct ResNet18||200||100, 150||128||0.1|
|Reformulation||PreAct ResNet18||200||100, 150||128||0.1|
In this section we provide the experimental details for all experiments presented in the main paper. Unless otherwise stated, the following parameters are chosen: , , weight decay of and optimised using SGD with momentum of 0.9. For cross validation experiments, 3 or 5 folds of 10% of the training data are generated and used for a single run each. Test set experiments use the entire training set and give evaluations on the test sets provided. If no test set is provided then a constant validation set of 10% of the available data is used. Table C.1 provides general training details that were present in all experiments.
All experiments were run on a single GTX1080ti or V100, with the exceptions of ImageNet experiments (4 GTX1080ti) and DenseNet/PyramidNet experiments (2 V100). ResNet18 LSTM experiments ran within 2 hours in all instances, PointNet experiments ran within 10 hours, WideResNet/DenseNet experiments ran within 2.5 days and auto-augment experiments ran within 10 days.
Appendix D Grad-CAM
Figure D.1 contains additional examples that support our discussion in the paper, as well as larger-scale versions of the images in Figure 3.
Appendix E Things we Tried That Didn’t Work
This section details a number of experiments and modifications we attempted which did not lead to significant results. Our aim here is to prevent future research effort being devoted to approaches that have already been explored by us. It may also be the case that better versions of these could be constructed which obtain better results.
e.1 Salience Prior
It is clear that we should care about how the mixing coefficient relates to the relative amount of salient information from each data point in the outcome. This presents a challenge because getting of the salient information in the first data point does not imply that we have of the salient information in the second. We could consider making an assumption that the expected distribution of salient information in each data point is the same. In such a case, the above problem no longer exists. For images, a simple assumption would be that the salient information is roughly Gaussian about the centre. To apply a salience prior to our mask generation process, we need to change the binarisation algorithm. Specifically, we iterate over the values in descending order until the mass over the prior is equal to . We experimented with this approach and found no significant performance gain, and so did not pursue it any further. That said, there may still be some value to the above motivation and a more complex, data point specific, salience distribution could work.
e.2 Mask Softening
Following the observation that combining interpolation and masking provides the best results, and particularly the experiments in summers2019improved, we considered a grey-scale version of FMix. Specifically, we explored a method which softened the edges in the mask. To achieve this, after sorting the low frequency image by pixel value, instead of choosing a threshold and setting one side to 1 and the other to 0, we choose an equal distance either side of the threshold and linearly value the mask between 1 and 0 for some number of pixels. The number of grey pixels is chosen to ensure that the mean mask value is retained and that the fraction of the image that is non-binary does not exceed some present value.
We found that softening the masks resulted in no performance gains, and in fact, occasionally hindered training. We considered it again for the toxic comments experiments since we assumed smooth transitions would be very important for text models. It did offer minor improvements over default FMix, however, we judged that the gain was not worth the added complexity and diluting of the core idea of FMix for us to present it in the paper. Furthermore, proposing it for the singular case of toxic comments would have been bad practice, since we only observed an improvement for one model, on one data set. That said, we feel mask softening would be interesting to explore further, certainly in the case of text models. We would need to experiment with softened FMix masks in multiple text data sets and observe improvement in most or all of them over base FMix in order to formally propose softening as an FMix modification.
e.3 Target Distribution
A final alteration that we experimented with relates to the distribution of targets. The idea was that we could change the distribution of the target mixing coefficients to obtain better ‘calibrated’ model outputs. The way this is done is simple, we pass the sampled through its CDF and then through the inverse CDF of the target distribution. This allows us to, for example, encourage confident outputs by choosing a symmetric Beta distribution with . The issue with this approach is two fold. First, changing the distribution of the outputs in this way has no bearing on the ordering, and so no effect on the classification accuracy. Second, any simple transform of this nature can be trivially learned by the model or applied in post. In other words, it is equivalent to training a model normally and then just transforming the outputs. As a result, it is difficult to argue that this approach does anything particularly clever. We trained models with different target distributions at several points and found that the performance was not significantly different.