Context and motivation:
Deep neural networks have revolutionized machine learning and have ubiquitous use in many application domains[31, 28, 19]. In full generality, many key tasks in deep learning reduces to solving the following optimization problem:
where denotes the weights of the neural network, denotes the loss function that is typically non-convex in , each denotes the (instantaneous) loss function that is contributed by the data point , and denotes the total number of data points. Stochastic gradient descent (SGD) is one the most popular approaches for attacking this problem in practice and is based on the following iterative updates:
where denotes the iteration number and denotes the stochastic gradient at iteration , that is defined as follows:
Here, is a random subset that is drawn with or without replacement at iteration , and denotes the number of elements in .
SGD is widely used in deep learning with a great success in its computational efficiency [4, 5]. Beyond efficiency, understanding how SGD performs better than its full batch counterpart in terms of test accuracy remains a major challenge. Even though SGD seems to find zero loss solutions on the training landscape (at least in certain regimes [59, 47, 27, 17]), it appears that the algorithm finds solutions with different properties depending on how it is tuned [50, 27, 25, 21, 37, 49]. Despite the fact that the impact of SGD on generalization has been studied [1, 54, 40], a satisfactory theory that can explain its success in a way that encompasses such peculiar empirical properties is still lacking.
A popular approach for analyzing SGD is based on considering SGD as a discretization of a continuous-time process [36, 25, 33, 22, 61, 9]. This approach mainly requires the following assumption111We note that more sophisticated assumptions than (4 ) have been made in terms of the covariance matrix of the Gaussian distribution (e.g. state dependent, anisotropic). However, in all these cases, the resulting distribution is still a Gaussian, therefore the same criticism holds.
) have been made in terms of the covariance matrix of the Gaussian distribution (e.g. state dependent, anisotropic). However, in all these cases, the resulting distribution is still a Gaussian, therefore the same criticism holds.on the stochastic gradient noise :
denotes the multivariate (Gaussian) normal distribution and
denotes the identity matrix of appropriate size. The rationale behind this assumption is that, if the size of the minibatchis large enough, then we can invoke the Central Limit Theorem (CLT) and assume that the distribution of is approximately Gaussian. Then, under this assumption, (2) can be written as follows:
where denotes a standard normal random variable in . If we further assume that the step-size is small enough, then the continuous-time analogue of the discrete-time process (5) is the following stochastic differential equation (SDE):222 In a recent work with a similar critic taken on the recent theories on the SGD dynamics, some theoretical concerns have been also raised about the SDE approximation of SGD . We believe that the SDE representation is sufficiently accurate for small step-sizes and a good, if not the best, proxy for understanding the behavior of SGD.
where denotes the standard Brownian motion. This SDE is a variant of the well-known Langevin diffusion and under mild regularity assumptions on , one can show that the Markov process is ergodic with its unique invariant measure, whose density is proportional to for any . . From this perspective, the SGD recursion in (5) can be seen as a first-order Euler-Maruyama discretization of the Langevin dynamics (see also [33, 25, 22]), which is often referred to as the Unadjusted Langevin Algorithm (ULA) [45, 30, 14].
Based on this observation,  focused on the relation between this invariant measure and the algorithm parameters, namely the step-size and mini-batch size, as a function of . They concluded that the ratio of learning rate divided by the batch size is the control parameter that determines the width of the minima found by SGD. Furthermore, they revisit the famous wide minima folklore : Among the minima found by SGD, the wider it is, the better it performs on the test set. However, there are several fundamental issues with this approach, which we will explain below.
We first illustrate a typical mismatch between the Gaussianity assumption and the empirical behavior of the stochastic gradient noise. In Figure 1
, we plot the histogram of the norms of the stochastic gradient noise that is computed using a convolutional neural network in a real classification problem and compare it to the histogram of the norms of Gaussian random variables. It can be clearly observed that the shape of the real histogram is very different than the Gaussian and shows aheavy-tailed behavior.
In addition to the empirical observations, the Gaussianity assumption also yields some theoretical issues. The first issue with this assumption is that the current SDE analyses of SGD are based on the invariant measure of the SDE, which implicitly assumes that sufficiently many iterations have been taken to converge to that measure. Recent results on ULA [44, 56] have shown that, the required number of iterations to achieve the invariant measure often grows exponentially with the dimension . This result contradicts with the current practice: considering the large size of the neural networks and limited computational budget, only a limited number of iterations – which is much smaller than – can be taken. This conflict becomes clearer in the light of the recent works that studied the local behavior of ULA [51, 60]. These studies showed that ULA will get close to the nearest local optimum in polynomial time; however, the required amount of time for escaping from that local optimum increases exponentially with the dimension. Therefore, the phenomenon that SGD prefers wide minima within a considerably small number of iterations cannot be explained using the asymptotic distribution of the SDE given in (6).
The second issue is related to the local behavior of the process and becomes clear when we consider the metastability analysis of Brownian motion-driven SDEs. These studies [16, 6, 24] consider the case where is initialized in a quadratic basin and then analyze the minimum time such that is outside that basin. They show that this so-called first exit time depends exponentially on the height of the basin; however, this dependency is only polynomial with the width of the basin. These theoretical results directly contradict with the the wide minima phenomenon: even if the height of a basin is slightly larger, the exit-time from this basin will be dominated by its height, which implies that the process would stay longer in (or in other words, ‘prefer’) deeper minima as opposed to wider minima. The reason why the exit-time is dominated by the height is due to the continuity of the Brownian motion, which is in fact a direct consequence of the Gaussian noise assumption.
A final remark on the issues of this approach is the observation that landscape is flat at the bottom regardless of the batch size used in SGD 
. In particular, the spectrum of the Hessian at a near critical point with close to zero loss value has many near zero eigenvalues. Therefore, local curvature measures that are used as a proxy for measuring the width of a basin correlates with the magnitudes of large eigenvalues of the Hessian which are few. Besides, during the dynamics of SGD it has been observed that the algorithm does not cross barriers except perhaps at the very initial phase[55, 2]. Such dependence of width on an essentially-flat landscape combined with the lack of explicit barrier crossing during the SGD descent forces us to rethink the analysis of basin hopping under a noisy dynamics.
Proposed framework: In this study, we aim at addressing these contradictions and come up with an arguably better-suited hypothesis for the stochastic gradient noise that has more pertinent theoretical implications for the phenomena associated with SGD. In particular, we go back to (3) and (4) and reconsider the application of CLT. This classical CLT assumes that
is a sum of many independent and identically distributed (i.i.d.) random variables, whose variance isfinite, and then it states that the law of converges to a Gaussian distribution, which then paves the way for (5). Even though the finite-variance assumption seems natural and intuitive at the first sight, it turns out that in many domains, such as turbulent motions , oceanic fluid flows , finance , biological evolution , audio signals , the assumption might fail to hold (see  for more examples). In such cases, the classical CLT along with the Gaussian approximation will no longer hold. While this might seem daunting, fortunately, one can prove an extended CLT and show that the law of the sum of these i.i.d. variables with infinite variance still converges to a family of heavy-tailed distributions that is called the -stable distribution . As we will detail in Section 2, these distributions are parametrized by their tail-index and they coincide with the Gaussian distribution when .
In this study, we relax the finite-variance assumption on the stochastic gradient noise and by invoking the extended CLT, we assume that follows an -stable distribution, as hinted in Figure 1(c). By following a similar rationale to (5) and (6), we reformulate SGD with this new assumption and consider its continuous-time limit for small step-sizes. Since the noise might not be Gaussian anymore (i.e. when ), the use of the Brownian motion would not be appropriate in this case and we need to replace it with the -stable Lévy motion, whose increments have an -stable distribution . Due to the heavy-tailed nature of -stable distribution, the Lévy motion might incur large discontinuous jumps and therefore exhibits a fundamentally different behavior than the Brownian motion, whose paths are on the contrary almost surely continuous. As we will describe in detail in Section 2, the discontinuities also reflect in the metastability properties of Lévy-driven SDEs, which indicate that, as soon as , the first exit time from a basin does not depend on its height; on the contrary, it directly depends on its width and the tail-index . Informally, this implies that the process will escape from narrow minima – no matter how deep they are – and stay longer in wide minima. Besides, as
get smaller, the probability for the dynamics to jump in a wide basin will increase. Therefore, if the-stable assumption on the stochastic gradient noise holds, then the existing metastability results automatically provide strong theoretical insights for illuminating the behavior of SGD.
Contributions: The main contributions of this paper are twofold: (i) we perform an extensive empirical analysis of the tail-index of the stochastic gradient noise in deep neural networks and (ii) based on these empirical results, we bring an alternative perspective to the existing approaches for analyzing SGD and shed more light on the folklore that SGD prefers wide minima by establishing a bridge between SGD and the related theoretical results from statistical physics and stochastic analysis.
We conduct experiments on the most common deep learning architectures. In particular, we investigate the tail behavior under fully-connected and convolutional models using negative log likelihood and linear hinge loss functions on MNIST, CIFAR10, and CIFAR100 datasets. For each configuration, we scale the size of the network and batch size used in SGD and monitor the effect of each of these settings on the tail index .
Our experiments reveal several remarkable results:
In all our configurations, the stochastic gradient noise turns out to be highly non-Gaussian and possesses a heavy-tailed behavior.
Increasing the size of the minibatch has a very little impact on the tail-index, and as opposed to the common belief that larger minibatches result in Gaussian gradient noise, the noise is still far from being Gaussian.
There is a strong interaction between the network architecture, network size, dataset, and the tail-index, which ultimately determine the dynamics of SGD on the training surface. This observation supports the view that, the geometry of the problem and the dynamics induced by the algorithm cannot be separated from each other.
In almost all configurations, we observe two distinct phases of SGD throughout iterations. During the first phase, the tail-index rapidly decreases and SGD possesses a clear jump when the tail-index is at its lowest value and causes a sudden jump in the accuracy. This behavior strengthens the view that SGD crosses barriers at the very initial phase.
Our methodology also opens up several interesting future directions and open questions, as we discuss in Section 5.
2 Stable distributions and SGD as a Lévy-Driven SDE
The CLT states that the sum of i.i.d. random variables with a finite second moment converges to a normal distribution if the number of summands grow. However, if the variables have heavy-tail, the second moment may not exist. For instance, if their densityhas a power-law tail decreasing as where ; only -th moment exist with . In this case, generalized central limit theorem (GCLT) says that the sum of such variables will converge to a distribution called the -stable distribution instead as the number of summands grows (see e.g. . In this work, we focus on the centered symmetric -stable () distribution, which is a special case of -stable distributions that are symmetric around the origin.
We can view the distribution as a heavy-tailed generalization of a centered Gaussian distribution. The
distributions are defined through their characteristic function via
. Even though their probability density function does not admit a closed-form formula in general except in special cases, their density decays with a power law tail likewhere is called the tail-index which determines the behavior of the distribution: as gets smaller; the distribution has a heavier tail. In fact, the parameter also determines the moments: if and only if ; implying has infinite variance when . The parameter is known as the scale parameter and controls the spread of around . We recover the Gaussian distribution as a special case of when .
In this study, we make the following assumption on the stochastic gradient noise:
where denotes the
’th component of a vector. Informally, we assume that each coordinate of is distributed with the same and the scale parameter depends on the state . Here, this dependency is not crucial since we are mainly interested in the tail-index
, which can be estimatedindependently from the scale parameter. Therefore, we will simply denote as for clarity.
By using the assumption (7), we can rewrite the SGD recursion as follows:
where is a random vector such that . If the step-size is small enough, then we can consider the continuous-time limit of this discrete-time process, which is expressed in the following SDE driven by an -stable Lévy process:
where denotes the -dimensional -stable Lévy motion with independent components. In other words, each component of is an independent -stable Lévy motion in . For the scalar case it is defined as follows for :
For , the increments are independent ().
The difference and have the same distribution: for .
is continuous in probability (i.e. it has stochastically continuous sample paths): for all and , as .
When , coincides with a scaled version of Brownian motion, . and are illustrated in Figure 2.
The SDE in (9) exhibits a fundamentally different behavior than the one in (6) does. This is mostly due to the stochastic continuity property of , which enables to have a countable number of discontinuities, which are sometimes called ‘jumps’. In the rest of this section, we will recall important theoretical results about this SDE and discuss their implications on SGD.
For clarity of the presentation and notational simplicity we focus on the scalar case and consider the SDE (9) in (i.e. ). Multidimensional generalizations of the metastability results presented in this paper can be found in . We rewrite (9) as follows:
for , started from the initial point , where is the -stable Lévy process, is a parameter and is a non-convex objective with local minima.
When , we recover the gradient descent dynamics in continuous time: , where the local minima are the stable points of this differential equation. However, as soon as , these states become ‘metastable’, meaning that there is a positive probability for to transition from one basin to another. However, the time required for transitioning to another basin strongly depends on the characteristics of the injected noise. The two most important cases are and . When , (i.e. the Gaussianity assumption) the process is continuous, which requires it to ‘climb’ the basin all the way up, in order to be able to transition to another basin. This fact makes the transition-time depend on the height of the basin. On the contrary, when , the process can incur discontinuities and do not need to cross the boundaries of the basin in order to transition to another one since it can directly jump. This property is called the ‘transition phenomenon’  and makes the transition-time mostly depend on the width of the basin. In the rest of the section, we will formalize these explanations.
Under some assumptions on the objective , it is known that the process (10) admits a stationary density . For a general , an explicit formula for the equilibrium distribution is not known, however when the noise level is small enough, finer characterizations of the structure of the equilibrium density in dimension one is known. We next summarize known results in this area, which show that Lévy-driven dynamics spends more time in ‘wide valleys’ in the sense of  when goes to zero.
Assume that is smooth with local minima separated by local maxima , i.e.
Furthermore, assume that the local minima and maxima are not degenerate, i.e. and for every . We also assume the objective gradient has a growth condition for some constant and when is large enough. Each local minima lies in the (interval) valley of (width) length . Consider also a -neighborhood around the local minimum with small enough so that the neighborhood is contained in the valley for every . We are interested in the first exit time from starting from a point and the transition time to a neighborhood of another local minimum, we will remove the dependency to
of the transition time in our discussions as it is clear from the context. The following result shows that the transition times are asymptotically exponentially distributed in the limit of small noise and scales likewith .
Theorem 1 ().
For an initial point , in the limit , the following statements hold regarding the transition time:
If the SDE (10) would be driven by the Brownian motion instead, then an analogous theorem to Theorem 2 holds saying that the transition times are still exponentially distributed but the scaling needs to be replaced by where is the maximal depth of the basins to be traversed between the two local minima [10, 7]. This means that in the small noise limit, Brownian-motion driven gradient descent dynamics need exponential time to transit to another minimum whereas Levy-driven gradient descent dynamics need only polynomial time. We also note from Theorem 2 that the mean transition time between valleys for Lévy SDE does not depend on the depth of the valleys they reside in which is an advantage over Brownian motion driven SDE in the existence of deep valleys. Informally, this difference is due to the fact that Brownian motion driven SDE has to typically climb up a valley to exit it, whereas Lévy-driven SDE could jump out.
The following theorem says that as , up to a normalization in time, the process behaves like a finite state-space Markov process that has support over the set of local minima admitting a stationary density with an infinitesimal generator . The process jumps between the valleys , spending time proportional to probability amount of time in each valley in the equilibrium where the probabilities are given by the solution to the linear system .
Theorem 2 ().
Let , for some . For ,
, as ,
in the sense of finite-dimensional distributions, where is a continuous-time Markov chain on a state space
is a continuous-time Markov chain on a state spacewith the infinitesimal generator with
This process admits a density satisfying .
A consequence of this theorem is that equilibrium probabilities are typically larger for “wide valleys". To see this consider the special case illustrated in Figure 3 with local minima separated by a local maximum at . For this example, , and the second local minimum lies in a wider valley. A simple computation reveals
We see that , that is in the equilibrium the process spends more time on the wider walley. In particular, the ratio grows with an exponent when the ratio of the width of the valleys grows. Consequently, if the gradient noise is indeed -stable distributed, these results directly provide theoretical evidence for the wide-minima behavior of SGD.
3 Experimental Setup and Methodology
Experimental setup: We investigate the tail behavior of the stochastic gradient noise in a variety of scenarios. We first consider a fully-connected network (FCN) on the MNIST and CIFAR10 datasets. For this model, we vary the depth (i.e. the number of layers) in the set
, the width (i.e. the number of neurons per layer) in the set, and the minibatch size ranging from to full batch. We then consider a convolutional neural network (CNN) architecture (AlexNet) on the CIFAR10 and CIFAR100 datasets. We scale the number of filters in each convolutional layer in range . We randomly split the MNIST dataset into train and test parts of sizes K and K, and CIFAR10 and CIFAR100 datasets into train and test parts of sizes K and K, respectively. The order of the total number of parameters range from several thousands to tens of millions.
For both fully connected and convolutional settings, we run each configuration with the negative-log-likelihood (i.e. cross entropy) and with the linear hinge loss, and we repeat each experiment with three different random seeds. The training algorithm is SGD with no explicit modification such as momentum or weight decay. The training runs until 100% training accuracy is achieved or until maximum number of iterations limit is reached (the latter limit is effective in the under-parametrized models). At every
th iteration, we log the full training and test accuracies, and the tail estimate of the gradients that are sampled using the corresponding mini-batch size. The codebase is implemented in python using pytorch and provided it in the supplementary material. Total runtime is3 weeks on 8 relatively modern GPUs.
Method for tail-index estimation: Estimating the tail-index of an extreme-value distribution is a long-standing topic. Some of the well-known estimators for this task are [18, 43, 12, 11]. Despite their popularity, these methods are not specifically developed for -stable distributions and it has been shown that they might fail for estimating the tail-index for -stable distributions [38, 41].
In this study, we use a relatively recent estimator proposed in  for -stable distributions. It is given in the following theorem.
Theorem 3 ().
Let be a collection of random variables with and . Define for . Then, the estimator
converges to almost surely, as .
As shown in Theorem 2.3 of , this estimator admits a provably faster convergence rate and smaller asymptotic variance than all the aforementioned methods.
In order to verify the accuracy of this estimator, we conduct a preliminary experiment, where we first generate many distributed random variables with , for different values of . Then, we estimate by using . We repeat this experiment times for each . As shown in Figure 3, the estimator is very accurate for a large range of . Due to its favorable theoretical properties such as independence of the scale parameter , combined with its empirical stability, we choose this estimator in our experiments.
In order to estimate the tail-index at iteration , we first partition the set of data points into many disjoint sets of size , such that the union of these subsets give all the data points. Formally, for all , , , and for . This approach is similar to sampling without replacement. We then compute the full gradient and the stochastic gradients for each minibatch . We finally compute the stochastic gradient noises , vectorize each and concatenate them to obtain a single vector, and compute the reciprocal of the estimator (15). In this case, we have and we set to the divisor of that is the closest to .
In this section we present the most important and representative results. We have observed that, in all configurations, the choice of the two loss functions and the three different initializations yield no significant difference. Therefore, throughout this section, we will focus on the negative-log-likelihood loss. Unless stated otherwise, we set the minibatch size and the step-size .
Effect of varying network size: In our first set of experiments, we measure the tail-index for varying the widths and depths for the FCN, and varying widths (i.e. the number of filters) for the CNN. For very small sizes, the networks perform poorly, therefore, we only illustrate sufficiently large network sizes, which yield similar accuracies. For these experiments, we compute the average of the tail-index measurements for the last K iterations (i.e. when becomes stationary) to focus on the late stage dynamics.
Figure 4 shows the results for the FCN. The first striking observation is that in all the cases, the estimated tail-index is far from with a very high confidence (the variance of the estimates were around ), meaning that the distribution of the gradient noise is highly non-Gaussian. For the MNIST dataset, we observe that systematically decreases for increasing network size, where this behavior becomes more prominent with the depth. This result shows that, for MNIST, increasing the dimension of the network results in a gradient noise with heavier tails and therefore increases the probability to end up in a wider basin.
For the CIFAR10 dataset, we still observe that is far from ; however, in this case, increasing the network size does not have a clear effect on : in all cases, we observe that is in the range –.
Figure 5 shows the results for the CNN. In this figure, we also depict the train and test accuracy, as well as the tail-index that is estimated on the test set. These results show that, for both CIFAR10 and CIFAR100, the tail-index is extremely low for the under-parametrized regime (e.g. the case when the width is , , or for CIFAR10). As we increase the size of the network the value of increases until the network performs reasonably well and stabilizes in the range –. We also observe that behaves similarly for both train and test sets333We observed a similar behavior in under-parametrized FCN; however, did not plot those results to avoid clutter..
These results show that there is strong interplay between the network architecture, dataset, and the algorithm dynamics: (i) we see that the size of the network can strongly influence , (ii) for the exact same network architecture, the choice of the dataset has a significant impact on not only the landscape of the problem, but also the noise characteristics, hence on the algorithm dynamics.
Effect of the minibatch size: In our second set of experiments, we investigate the effect of the size of the minibatch on . We focus on the FCN and monitor the behavior of for different network and minibatch sizes . Figure 6 illustrates the results. These rather remarkable results show that, as opposed to the common belief that the gradient noise behaves similar to a Gaussian for large , the tail-index does not increase at all with the increasing . We observe that stays almost the same when the depth is and it moves in a small interval when the depth is set to . We note that we obtained the same the train and test accuracies for different minibatch sizes.
Tail behavior throughout iterations: So far, we have focused on the last iterations of SGD, where is in a stationary regime. In our last set of experiments, we shift our focus on the first iterations and report an interesting behavior that we observed in almost all our experiments. As a representative, in Figure 7, we show the temporal evolution of SGD for the FCN with layers and neurons/layer.
The results clearly show that there are two distinct phases of SGD (in this configuration before and after iteration ). In the first phase, the loss decreases very slowly, the accuracy slightly increases, and more interestingly rapidly decreases. When reaches its lowest level, the process possesses a jump, which causes a sudden decrease in the accuracy. After this point the process recovers again and we see a stationary behavior in and an increasing behavior in the accuracy.
The fact that the process has a jump when is at its smallest value provides a strong support to our assumptions and the metastability theory that we discussed in the previous section. Furthermore, these results further strengthen the view that SGD crosses barriers at the very initial phase. On the other hand, our current analysis is not able to determine whether the process jumps in a different basin or a ‘better’ part of the same basin and we leave it as a future work.
5 Conclusion and Open Problems
We investigated the tail behavior of the gradient noise in deep neural networks and empirically showed that the gradient noise is highly non-Gaussian. This outcome enabled us to analyze SGD as an SDE driven by a Lévy motion and establish a bridge between SGD and existing theoretical results, which provides more illumination on the behavior of SGD, especially in terms of choosing wide minima.
This study also brings up interesting open questions and future directions: (i) While the current metastability theory applies for the continuous-time processes, the behavior of the discretized process and its dependence on the algorithm parameters (e.g., the step-size, minibatch size) are not clear and yet to be investigated. (ii) We observe that, especially during the first iterations, the tail-index depends on the current state , which suggests analyzing SGD as a stable-like process  where the tail-index can depend on time. However, the metastability behavior of these processes are not clear at the moment and its theory is still in an early phase . (iii) Furthermore, an extension of the current metastability theory that includes minima with zero modes is also missing and appears to be challenging yet crucial direction of future research.
This work is partly supported by the French National Research Agency (ANR) as a part of the FBIMATRIX (ANR-16-CE23-0014) project. Mert Gürbüzbalaban acknowledges support from the grants NSF DMS-1723085 and NSF CCF-1814888.
-  Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXiv:1710.03667, 2017.
-  Marco Baity-Jesi, Levent Sagun, Mario Geiger, Stefano Spigler, Gerard Ben Arous, Chiara Cammarota, Yann LeCun, Matthieu Wyart, and Giulio Biroli. Comparing dynamics: Deep neural networks versus glassy systems. In International Conference on Machine Learning, volume 80, pages 314–323, Stockholm Sweden, 10–15 Jul 2018.
-  R. F. Bass. Uniqueness in law for pure jump Markov processes. Probability Theory and Related Fields, 79(2):271–287, 1988.
-  Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT’2010, pages 177–186. Physica-Verlag HD, 2010.
-  Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, pages 161–168, 2008.
-  A. Bovier, M. Eckhoff, V. Gayrard, and M. Klein. Metastability in reversible diffusion processes i: Sharp asymptotics for capacities and exit times. Journal of the European Mathematical Society, 6(4):399–424, 2004.
-  Anton Bovier, Véronique Gayrard, and Markus Klein. Metastability in reversible diffusion processes ii: Precise asymptotics for small eigenvalues. Journal of the European Mathematical Society, 7(1):69–99, 2005.
-  P. Chaudhari, Anna Choromanska, S. Soatto, Yann LeCun, C. Baldassi, C. Borgs, J. Chayes, Levent Sagun, and R. Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In International Conference on Learning Representations (ICLR), 2017.
-  P. Chaudhari and S. Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In International Conference on Learning Representations, 2018.
-  Martin V. Day. On the exponential exit law in the small parameter exit problem. Stochastics, 8(4):297–323, 1983.
-  L. De Haan and L. Peng. Comparison of tail index estimators. Statistica Neerlandica, 52(1):60–70, 1998.
-  A. L. M. Dekkers, J. H. J. Einmahl, and L. De Haan. A moment estimator for the index of an extreme-value distribution. The Annals of Statistics, pages 1833–1855, 1989.
-  J. Duan. An Introduction to Stochastic Dynamics. Cambridge University Press, New York, 2015.
-  A. Durmus and E. Moulines. Non-asymptotic convergence analysis for the unadjusted Langevin algorithm. arXiv preprint arXiv:1507.05021, 2015.
-  Hans Fischer. A history of the central limit theorem: From classical to modern probability theory. Springer Science & Business Media, 2010.
-  M. I. Freidlin and A. D. Wentzell. Random perturbations. In Random perturbations of dynamical systems, pages 15–43. Springer, 1998.
-  Mario Geiger, Stefano Spigler, Stéphane d’Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. The jamming transition as a paradigm to understand the loss landscape of deep neural networks. arXiv preprint arXiv:1809.09349, 2018.
-  B. M. Hill. A simple general approach to inference about the tail of a distribution. The Annals of Statistics, pages 1163–1174, 1975.
-  Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82–97, 2012.
-  Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1–42, 1997.
-  Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pages 1731–1741, 2017.
-  W. Hu, C. J. Li, L. Li, and J.-G. Liu. On the diffusion approximation of nonconvex stochastic gradient descent. arXiv preprint arXiv:1705.07562, 2017.
-  P. Imkeller, I. Pavlyukevich, and M. Stauch. First exit times of non-linear dynamical systems in rd perturbed by multifractal Lévy noise. Journal of Statistical Physics, 141(1):94–119, 2010.
-  P. Imkeller, I. Pavlyukevich, and T. Wetzel. The hierarchy of exit times of Lévy-driven Langevin equations. The European Physical Journal Special Topics, 191(1):211–222, Dec 2010.
-  S. Jastrzebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
-  B. Jourdain, S. Méléard, and W. A. Woyczynski. Lévy flights in evolutionary ecology. Journal of Mathematical Biology, 65(4):677–707, 2012.
-  Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097–1105, 2012.
-  I. Kuhwald and I. Pavlyukevich. Bistable behaviour of a jump-diffusion driven by a periodic stable-like additive process. Discrete & Continuous Dynamical Systems-Series B, 21(9), 2016.
-  D. Lamberton and G. Pages. Recursive computation of the invariant distribution of a diffusion: the case of a weakly mean reverting drift. Stochastics and dynamics, 3(04):435–451, 2003.
-  Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 EP –, 05 2015.
-  P. Lévy. Théorie de l’addition des variables aléatoires. Gauthiers-Villars, Paris, 1937.
-  Q. Li, C. Tai, and W. E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Proceedings of the 34th International Conference on Machine Learning, pages 2101–2110, 06–11 Aug 2017.
-  A. Liutkus and R. Badeau. Generalized Wiener filtering with fractional power spectrograms. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 266–270. IEEE, 2015.
-  B. B. Mandelbrot. Fractals and Scaling in Finance: Discontinuity, Concentration, Risk. Selecta Volume E. Springer Science & Business Media, 2013.
-  S. Mandt, M. Hoffman, and D. Blei. A variational analysis of stochastic gradient algorithms. In International Conference on Machine Learning, pages 354–363, 2016.
-  Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks. arXiv preprint arXiv:1804.07612, 2018.
-  S. Mittnik and S. T. Rachev. Tail estimation of the stable index . Applied Mathematics Letters, 9(3):53–56, 1996.
-  M. Mohammadi, A. Mohammadpour, and H. Ogata. On estimating the tail index and the spectral measure of multivariate -stable distributions. Metrika, 78(5):549–561, 2015.
-  Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pages 5947–5956, 2017.
-  V. Paulauskas and M. Vaičiulis. Once more on comparison of tail index estimators. arXiv preprint arXiv:1104.1242, 2011.
-  Ilya Pavlyukevich. Cooling down lévy flights. Journal of Physics A: Mathematical and Theoretical, 40(41):12299, 2007.
-  J. Pickands. Statistical inference using extreme order statistics. The Annals of Statistics, 3(1):119–131, 1975.
-  M. Raginsky, A. Rakhlin, and M. Telgarsky. Non-convex learning via stochastic gradient Langevin dynamics: a nonasymptotic analysis. In Proceedings of the 2017 Conference on Learning Theory, volume 65, pages 1674–1703, 2017.
-  G. O. Roberts and O. Stramer. Langevin Diffusions and Metropolis-Hastings Algorithms. Methodology and Computing in Applied Probability, 4(4):337–357, December 2002.
-  Levent Sagun, Utku Evci, V. Uğur Güney, Yann Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. ICLR 2018 Workshop Contribution, arXiv:1706.04454, 2017.
-  Levent Sagun, V. Uğur Güney, Gérard Ben Arous, and Yann LeCun. Explorations on high dimensional landscapes. International Conference on Learning Representations Workshop Contribution, arXiv:1412.6615, 2015.
-  G. Samorodnitsky and M. Grigoriu. Tails of solutions of certain nonlinear stochastic differential equations driven by heavy tailed Lévy motions. Stochastic Processes and their Applications, 105(1):69 – 97, 2003.
-  Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017.
-  Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147, 2013.
-  B. Tzen, T. Liang, and M. Raginsky. Local optimality and generalization guarantees for the langevin algorithm via empirical metastability. In Proceedings of the 2018 Conference on Learning Theory, 2018.
-  E. R. Weeks, T. H. Solomon, J. S. Urbach, and H. L. Swinney. Observation of anomalous diffusion and Lévy flights. In Lévy flights and related topics in physics, pages 51–71. Springer, 1995.
-  W. A. Woyczyński. Lévy processes in the physical sciences. In Lévy processes, pages 241–266. Springer, 2001.
-  Lei Wu, Chao Ma, and E Weinan. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, pages 8289–8298, 2018.
-  Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. arXiv preprint arXiv:1802.08770, 2018.
-  Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. In Advances in Neural Information Processing Systems, pages 3125–3136, 2018.
-  S. Yaida. Fluctuation-dissipation relations for stochastic gradient descent. In International Conference on Learning Representations, 2019.
-  V. V. Yanovsky, A. V. Chechkin, D. Schertzer, and A. V. Tur. Lévy anomalous diffusion and fractional Fokker–Planck equation. Physica A: Statistical Mechanics and its Applications, 282(1):13–34, 2000.
-  Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. International Conference on Learning Representations, 2017.
-  Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 2017 Conference on Learning Theory, volume 65, pages 1980–2022, 2017.
-  Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects. arXiv preprint arXiv:1803.00195, 2018.