Deep neural networks (NNs) are extremely high dimensional objects. A popular deep NN for image recognition tasks like ResNet-50 (He et al., 2016) has 25 million parameters, and it is common for language models to have more than one billion parameters (Jozefowicz et al., 2016). This overparameterization may be responsible for NNs impressive generalization performance (Novak et al., 2018). Simultaneously, the high dimensional nature of NNs makes them very difficult to reason about.
Over the decades of NN research, the common lore about the geometry of the loss landscape of NNs has changed dramatically. In the early days of NN research it was believed that NNs were difficult to train because they tended to get stuck in suboptimal local minima. Dauphin et al. (2014) and Choromanska et al. (2015) argued that this is unlikely to be a problem for most loss landscapes because local minima will tend not to be much worse than global minima. There are, however, many other plausible properties of the geometry of NN loss landscapes that could pose obstacles to NN optimization. These include: saddle points, vast plateaus where the gradient is very small, cliffs where the loss suddenly increases or decreases, winding canyons, and local maxima that must be navigated around.
Ideally we would like to be able to somehow visualize the loss landscapes of NNs, but this is a difficult, perhaps even futile, task because it involves embedding this extremely high dimensional space into very few dimensions — typically one or two. Goodfellow et al. (2014) introduced a visualization technique that consists of plotting the loss along a straight line from the initial point to the final point of training (the “royal road”). The authors found that the loss often decreased monotonically along this path. They further considered the loss in the space from the residuals between the NN’s trajectory to this royal road. Note that while this is a two-dimensional manifold, it is not a linear subspace. Lorch (2016)
proposed another visualization technique in which principal component analysis (PCA) is performed on the NN trajectory and the trajectory is projected into the subspace spanned by the lowest PCA components. This technique was further explored byLi et al. (2018), who noted that most of the variance is in the first two PCA components.
In this paper we consider the theory behind this visualization technique. We show that PCA projections of random walks in flat space qualitatively have many of the same properties as projections of NN training trajectories. We then generalize these results to a random walk with momentum and a random walk in a quadratic potential, also known as an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930)
. This process is more similar to NN optimization since it consists of a deterministic component (the true gradient) plus a stochastic component. In fact, recent work has suggested that stochastic gradient descent (SGD) approximates a random walk in a quadratic potential(Ahn et al., 2012; Mandt et al., 2016; Smith & Le, 2017). Finally, we perform experiments on linear models and large NNs to show how closely they match this simplified model.
The approach we take to study the properties of the PCA of high dimensional random walks in flat space follows that of Moore & Ahmed (2017), but we correct several errors in their argument, notably in the values of the matrix and the trace of in Eq. 10. We also fill in some critical omissions, particularly the connection between banded Toeplitz matrices and circulant matrices. We extend their contribution by proving that the trajectories of high dimensional random walk in PCA subspaces are Lissajous curves and generalizing to random walks with momentum and Ornstein-Uhlenbeck processes.
2 PCA of random walks in flat space
Let us consider a random walk in -dimensional space consisting of
steps where every step is equal to the previous step plus a sample from an arbitrary probability distribution,, with zero mean and a finite covariance matrix.111The case of a constant non-zero mean corresponds to a random walk with a constant drift term. This is not an especially interesting extension from the perspective of PCA because in the limit of a large number of steps the first PCA component will simply pick out the direction of the drift (i.e., the mean), and the remaining PCA components will behave as a random walk without a drift term. For simplicity we shall assume that the covariance matrix has been normalized so that its trace is 1. This process can be written in the form
where is a
-dimensional vector and. If we collect the s together in an dimensional design matrix X, we can then write this entire process in matrix form as
where the matrix S is an matrix consisting of 1 along the diagonal and -1 along the subdiagonal,
and the matrix R is an matrix where every column is a sample from . Thus .
To perform PCA, we need to compute the eigenvalues and eigenvectors of the covariance matrix , where is the matrix X with the mean of every dimension across all steps subtracted. can be found by applying the centering matrix, C:
We now note that the analysis is simplified considerably by instead finding the eigenvalues and eigenvectors of the matrix . The non-zero eigenvalues of are the same as those of . The eigenvectors are similarly related by , where is a (non-normalized) eigenvector of , and is the corresponding eigenvector of .
We therefore would like to find the eigenvalues and eigenvectors of the matrix
where we note that . Consider the middle term, . In the limit we will have because the off diagonal terms will be , whereas the diagonal terms will be . (Recall that we have assumed that the covariance of the noise distribution is normalized; if the covariance is not normalized, this simply introduces an overall scale factor given by the trace of the covariance.) We therefore have the simplification
2.2 Asymptotic convergence to circulant matrices
Let us consider the new middle term, . The matrix S is a banded Toeplitz matrix. Gray (2006) has shown that banded Toeplitz matrices asymptotically approach circulant matrices as the size of the matrix grows. In particular, Gray (2006) showed that banded Toeplitz matrices have the same inverses, eigenvalues, and eigenvectors as their corresponding circulant matrices in this asymptotic limit (see especially theorem 4.1 and subsequent material from Gray 2006). Thus in our case, if we consider the limit of a large number of steps, S asymptotically approaches a circulant matrix that is equal to S in every entry except the top right, where there appears a instead of a 0.222We note in passing that is the exact representation of a closed random walk.
With the circulant limiting behavior of S in mind, the problem simplifies considerably. We note that C is also a circulant matrix, the product of two circulant matrices is circulant, the transpose of a circulant matrix is circulant, and the inverse of a circulant matrix is circulant. Thus the matrix is asymptotically circulant as . Finding the eigenvectors is trivial because the eigenvectors of all circulant matrices are the Fourier modes. To find the eigenvalues we must explicitly consider the values of . The matrix consists of a 2 along the diagonal, -1 along the subdiagonal and superdiagonal, and 0 elsewhere, with the exception of the bottom right corner where there appears a 1 instead of a 2.
While this matrix is not a banded Toeplitz, it is asymptotically equivalent to a banded Toeplitz matrix because it differs from a banded Toeplitz matrix by a finite amount in a single location. We now note that multiplication of the centering matrix does not change either the eigenvectors or the eigenvalues of this matrix since all vectors with zero mean are eigenvectors of the centering matrix with eigenvalue 1, and all Fourier modes but the first have zero mean. Thus the eigenvalues of can be determined by the inverse of the non-zero eigenvalues of , which is an asymptotic circulant matrix. The eigenvalue of a circulant matrix with entries in the first row is
where is the root of unity. The imaginary parts of the roots of unity cancel out, leaving the eigenvalue of to be
and the eigenvalue of to be
The sum of the eigenvalues is given by the trace of , and is given by a lower triangular matrix with ones everywhere on and below the diagonal. The trace of is therefore given by
and so the explained variance ratio from the PCA component, in the limit is
If we let we can consider only the first term in a Taylor expansion of the cosine term. Requiring that , the explained variance ratio is
We pause here to marvel that the explained variance ratio of a random walk in the limit of infinite dimensions is highlyskewed towards the first few PCA components. Roughly 60% of the variance is explained by the first component, 80% by the first two components, 95% by the first 12 components, and 99% by the first 66 components.
2.3 Projection of the trajectory onto PCA components
Let us now turn to the trajectory of the random walk when projected onto the PCA components. The trajectory projected onto the PCA component is
where is the normalized . We ignore the centering operation from here on because it changes neither the eigenvectors nor the eigenvalues. From above, we then have
By the symmetry of the eigenvalue equations and , it can be shown that
Since is simply the Fourier mode, we therefore have
This implies that the random walk trajectory projected into the subspace spanned by two PCA components will be a Lissajous curve. In Fig. 1 we plot the trajectories of a high dimensional random walk projected to various PCA components and compare to the corresponding Lissajous curves. We perform 1000 steps of a random walk in 10,000 dimensions and find an excellent correspondence between the empirical and analytic trajectories. We additionally show the projection onto the first few PCA components over time in Fig. 6 in the supplementary material.
While our experiments thus far have used an isotropic Gaussian distribution for ease of computation, we emphasize that these results are completely general forany probability distribution with zero mean and a finite covariance matrix with rank much larger than the number of steps. We include the PCA projections and eigenvalue distributions of random walks using non-isotropic multivariate Gaussian distributions in Figs. 7 and 8 in the supplementary material.
3.1 Random walk with momentum
It is a common practice to train neural networks using stochastic gradient descent with momentum. It is therefore interesting to examine the case of a random walk with momentum. In this case, the process is governed by the following set of updates:
It can be seen that this modifies Eq. 2 to instead read
where M is a lower triangular Toeplitz matrix with 1 on the diagonal and on the subdiagonal. The analysis from Section 2 is unchanged, except that now instead of considering the matrix we have the matrix . Although M is not a banded Toeplitz matrix, its terms decay exponentially to zero for terms very far from the main diagonal. It is therefore asymptotically circulant as well, and the eigenvectors remain Fourier modes. To find the eigenvalues consider the product , noting that is a matrix with 1s along the main diagonal and s subdiagonal. With some tedious calculation it can be seen that the matrix is given by
with the exception that , and . As before, this matrix is asymptotically circulant, so the eigenvalues of its inverse are
In the limit of , the distribution of eigenvalues is identical to that of a random walk in flat space, however for finite , it has the effect of shifting the distribution towards the lower PCA components. We empirically test Eq. 21 in Fig. 9 in the supplementary material.
3.2 Discrete Ornstein-Uhlenbeck processes
A useful generalization of the above analysis of random walks in flat space is to consider random walks in a quadratic potential, also known as an AR(1) process or a discrete Ornstein-Uhlenbeck process. For simplicity we will assume that the potential has its minimum at the origin. Now every step consists of a stochastic component and a deterministic component which points toward the origin and is proportional in magnitude to the distance from the origin. In this case the update equation can be written
where measures the strength of the potential. In the limit the potential disappears and we recover a random walk in flat space. In the limit the potential becomes infinitely strong and we recover independent samples from a multivariate Gaussian distribution. For the steps will oscillate across the origin. For outside the updates diverge exponentially.
3.2.1 Analysis of eigenvectors and eigenvalues
This analysis proceeds similarly to the analysis in Section 2 except that instead of S we now have the matrix which has 1s along the diagonal and along the subdiagonal. remains a banded Toeplitz matrix and so the arguments from Sec. 2 that is asymptotically circulant hold. This implies that the eigenvectors of are still Fourier modes. The eigenvalues will differ, however, because we now have that the components of are given by
From Eq. 7 we have that the eigenvalue of is
We show in Fig. 2 a comparison between the eigenvalue distribution predicted from Eq. 24 and the observed distribution from a 3000 step Ornstein-Uhlenbeck process in 30,000 dimensions for several values of . There is generally an extremely tight correspondence between the two. The exception is in the limit of , where there is a catch which we have hitherto neglected. While it is true that the mean eigenvalue of any eigenvector approaches the same constant, there is nevertheless going to be some distribution of eigenvalues for any finite walk. Because PCA sorts the eigenvalues, there will be a characteristic deviation from a flat distribution.
3.2.2 Critical distance and mixing time
While we might be tempted to take the limit as we did in the case of a random walk in flat space, doing so would obscure interesting dynamics early in the walk. (A random walk in flat space is self-similar so we lose no information by taking this limit. This is no longer the case in an Ornstein- Uhlenbeck process because the parameter sets a characteristic scale in the system.) In fact there will be two distinct phases of a high dimensional Ornstein-Uhlenbeck process initialized at the origin. In the first phase the process will behave as a random walk in flat space — the distance from the origin will increase proportionally to and the variance of the PCA component will be proportional to . However, once the distance from the origin reaches a critical value, the gradient toward the origin will become large enough to balance the tendency of the random walk to drift away from the origin.333Assuming we start close to the origin. If we start sufficiently far from the origin the trajectory will exponentially decay to this critical value. At this point the trajectory will wander indefinitely around a sphere centered at the origin with radius given by this critical distance. Thus, while an Ornstein-Uhlenbeck process is mean-reverting in low dimensions, in the limit of infinite dimensions the Ornstein-Uhlenbeck process is no longer mean-reverting — an infinite dimensional Ornstein-Uhlenbeck process will never return to its mean.444Specifically, since the limiting distribution is a -dimensional Gaussian, the probability that the process will return to within of the origin is , where is the regularized gamma function. For small this decays exponentially with .
This critical distance can be calculated by noting that each dimension is independent of every other and it is well known that the asymptotic distribution of an AR(1) process with Gaussian noise is Gaussian with a mean of zero and a standard deviation of, where is the variance of the stochastic component of the process. In high dimensions the asymptotic distribution as is simply a multidimensional isotropic Gaussian. Because we are assuming , the overwhelming majority of points sampled from this distribution will be in a narrow annulus at a distance
from the origin. Since the distance from the origin during the initial random walk phase grows as , the process will start to deviate from a random walk after steps. We show in the right panel of Fig. 2 the distance from the origin over time for 3000 steps of Ornstein-Uhlenbeck processes in 30,000 dimensions with several different choices of . We compare to the prediction of Eq. 25 and find a good match.
3.2.3 Iterate averages converge slowly
We finally note that if the location of the minimum is unknown, then iterate (or Polyak) averaging can be used to provide a better estimate. But the number of steps must be much greater thanbefore iterate averaging will improve the estimate. Only then will the location on the sphere be approximately orthogonal to its original location on the sphere and the variance on the estimate of the minimum will decrease as . We compute the mean of converged Ornstein-Uhlenbeck processes with various choices of in Fig. 10 in the supplementary material.
3.2.4 Random walks in non-isotropic potential are dominated by low curvature directions
While our analysis has been focused on the special case of a quadratic potential with equal curvature in all dimensions, a more realistic quadratic potential will have a distribution of curvatures and the axes of the potential may not be aligned with the coordinate basis. Fortunately these complications do not change the overall picture much. For a general quadratic potential described by a positive semi-definite matrix , we can decompose into its eigenvalues and eigenvectors. We then apply a coordinate transformation to align the parameter space with the eigenvectors of . At this point we have a distribution of curvatures, each one given by an eigenvalue of . However, because we are considering the limit of infinite dimensions, we can assume that there will be a large number of dimensions that fall in any bin . Each of these bins can be treated as an independent high-dimensional Ornstein-Uhlenbeck process with curvature . After steps, PCA will then be dominated by dimensions for which is small enough that . Thus, even if relatively few dimensions have small curvature they will come to dominate the PCA projected trajectory after enough steps.
4 Comparison to linear models and neural networks
While random walks and Ornstein-Uhlenbeck processes are analytically tractable, there are several important differences between these simple processes and optimization of even linear models. In particular, the statistics of the noise will depend on the location in parameter space and so will change over the course of training. Furthermore, there may be finite data or finite trajectory length effects.
To get a sense for the effect of these differences we now compare the distribution of the variances in the PCA components between two models and a random walk. For our first model we train a linear model without biases on CIFAR-10 using a learning rate of
for 10,000 steps. For our second model we train ResNet-50-v2 on Imagenet without batch normalization for 150,000 steps using SGD with momentum and linear learning rate decay. We collect the value of all parameters at every step for the first 1500 steps, the middle 1500 steps, and the last 1500 steps of training, along with collecting the parameters every 100 steps throughout the entirety of training. Further details of both models and the training procedures can be found in the supplementary material. While PCA is tractable on a linear model of CIFAR-10, ResNet-50-v2 has25 million parameters and performing PCA directly on the parameters is infeasible, so we instead perform a random Gaussian projection into a subspace of 30,000 dimensions. We show in Fig. 3 the distribution of the PCA variances at the beginning, middle, and end of training for both models and compare to the distribution of variances from an infinite dimensional random walk. We show tableaux of the PCA projected trajectories from the middle of training for the linear model and ResNet-50-v2 in Fig. 4. Tableaux of the other training trajectories in various PCA subspaces are shown in the supplementary material.
The distribution of eigenvalues of the linear model resembles an OU process, whereas the distribution of eigenvalues of ResNet-50-v2 resembles a random walk with a large drift term. The trajectories appear almost identical to those of random walks shown in Fig. 1, with the exception that there is more variance along the first PCA component than in the random walk case, particularly at the start and end points. This manifests itself in a small outward turn of the edges of the parabola in the PCA2 vs. PCA1 projection. This suggests that ResNet-50-v2 generally moves in a consistent direction over relatively long spans of training, similarly to an Ornstein-Uhlenbeck process initialized beyond .
5 Random walks with decaying step sizes
We finally note that the PCA projected trajectories of the linear model and ResNet-50-v2 over the entire course of training qualitatively resemble those of a high dimensional random walk with exponentially decaying step sizes. To show this we train a linear regression model, where W is a fixed, unknown vector of dimension 10,000. We sample x from a 10,000 dimensional isotropic Gaussian and calculate the loss
where is the correct output. We show in Fig. 14 that the step size decays exponentially. We fit the decay rate to this data and then perform a random walk in 10,000 dimensions but decay the variance of the stochastic term by this rate. We compare in Fig. 15 of the supplementary material the PCA projected trajectories of the linear model trained on synthetic data to the decayed random walk. We note that these trajectories resemble the PCA trajectories over the entire course of training observed in Figs. 12 and 13 for the linear model trained on CIFAR-10 and ResNet-50-v2 trained on Imagenet.
We have derived the distribution of the variances of the PCA components of a random walk both with and without momentum in the limit of infinite dimensions, and proved that the PCA projections of the trajectory are Lissajous curves. We have argued that the PCA projected trajectory of a random walk in a general quadratic potential will be dominated by the dimensions with the smallest curvatures where they will appear similar to a random walk in flat space. Finally, we find that the PCA projections of the training trajectory of a layer in ResNet-50-v2 qualitatively resemble those of a high dimensional random walk despite the many differences between the optimization of a large NN and a high dimensional random walk.
The authors thank Matthew Hoffman, Martin Wattenberg, Jeffrey Pennington, Roy Frostig, and Niru Maheswaranathan for helpful discussions and comments on drafts of the manuscript.
- Ahn et al. (2012) Ahn, Sungjin, Korattikara, Anoop, and Welling, Max. Bayesian posterior sampling via stochastic gradient fisher scoring. arXiv preprint arXiv:1206.6380, 2012.
- Choromanska et al. (2015) Choromanska, Anna, Henaff, Mikael, Mathieu, Michael, Arous, Gérard Ben, and LeCun, Yann. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pp. 192–204, 2015.
- Dauphin et al. (2014) Dauphin, Yann N, Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, Ganguli, Surya, and Bengio, Yoshua. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in neural information processing systems, pp. 2933–2941, 2014.
- Goodfellow et al. (2014) Goodfellow, Ian J, Vinyals, Oriol, and Saxe, Andrew M. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544, 2014.
- Gray (2006) Gray, Robert M. Toeplitz and circulant matrices: A review. Foundations and Trends® in Communications and Information Theory, 2(3):155–239, 2006.
- He et al. (2016) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In
- Jozefowicz et al. (2016) Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016.
- Li et al. (2018) Li, Hao, Xu, Zheng, Taylor, Gavin, Studor, Christoph, and Goldstein, Tom. Visualizing the loss landscape of neural nets. In International Conference on Learning Representations, 2018.
Visualizing deep network training trajectories with pca.
ICML Workshop on Visualization for Deep Learning, 2016.
Mandt et al. (2016)
Mandt, Stephan, Hoffman, Matthew, and Blei, David.
A variational analysis of stochastic gradient algorithms.
International Conference on Machine Learning, pp. 354–363, 2016.
- Moore & Ahmed (2017) Moore, James and Ahmed, Hasan. High dimensional random walks can appear low dimensional: Application to influenza h3n2 evolution. arXiv preprint arXiv:1707.09361, 2017.
- Novak et al. (2018) Novak, Roman, Bahri, Yasaman, Abolafia, Daniel A, Pennington, Jeffrey, and Sohl-Dickstein, Jascha. Sensitivity and generalization in neural networks: an empirical study. International Conference on Learning Representations, 2018.
- Smith & Le (2017) Smith, Samuel L and Le, Quoc V. A bayesian perspective on generalization and stochastic gradient descent. In Proceedings of Second workshop on Bayesian Deep Learning (NIPS 2017), 2017.
- Uhlenbeck & Ornstein (1930) Uhlenbeck, George E and Ornstein, Leonard S. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
7 Further empirical tests
7.1 High dimensional random walks
We test Eq. 12 by computing 1000 steps of a random walk in 10,000 dimensions and performing PCA on the trajectory. We show in Fig. 5 the empirical variance ratio for the various components compared to the prediction from Eq. 12 and find excellent agreement. The empirical variance ratio is slightly higher than the predicted variance ratio for the highest PCA components due to the fact that there are a finite number of dimensions in this experiment, so the contribution from all components greater than the number of steps taken must be redistributed among the other components, which leads to proportionally the largest increase in the largest PCA components.
7.2 Random walk with non-isotropic noise
To demonstrate that our results hold for non-isotropic noise distributions we perform a random walk where the noise is sampled from a multivariate Gaussian distribution with a random covariance matrix, . Because sampling from a multivariate Gaussian with an arbitrary covariance matrix is difficult in high dimensions, we restrict the random walk to 1000 dimensions, keeping the number of steps 1000 as before. To construct the covariance matrix, we sample a
dimensional random matrix,R
, where each element is a sample from a normal distribution and then set. Although R is square. We show the distribution of explained variance ratios with the prediction from Eq. 12 in Fig. 7. There is a tight correspondence between the two up until the largest PCA components where finite dimension effects start to dominate. We also show in Fig. 8 PCA projected trajectories of this random walk along with a random walk where the random variates are sampled from a 1000-dimensional isotropic distribution for comparison to provide a sense for the amount of noise introduced by the relatively small number of dimensions. Although the small dimensionality introduces noise into the PCA projected trajectories, it is clear that the general shapes match the predicted Lissajous curves.
7.3 Random walk with momentum
We test Eq. 21 by computing 1000 steps of a random walk in 10,000 dimensions with various choices of the momentum parameter, . We show in Fig. 9 the observed distribution of PCA variances (not the explained variance ratio) along with the prediction from Eq. 21. There is an extremely tight correspondence between the two, except for the lowest PCA components for . This is expected because the effective step size is set by , and because , the walk does not have sufficient time to settle into its stationary distribution of eigenvalues when .
7.4 Iterate averaging of an Ornstein-Uhlenbeck process
We show in Fig. 10 the mean of all steps of Ornstein-Uhlenbeck processes which have converged to a random walk on a sphere of radius . We show in the dashed line the predicted value of , the number of steps required to reach (i.e., the crossing time of the sphere). The position on the sphere will close to its original location for so iterate averaging will not improve the estimate of the minimum. Only when will iterate averaging improve the estimate of the minimum since the correlation between new points wit the original location will be negligible.
8 Details of models and training
8.1 Linear regression on CIFAR-10
We train linear regression on CIFAR-10 for 10,000 steps using SGD and a batch size of 128 and a learning rate of . The model achieves a validation accuracy of 29.1%.
8.2 ResNet-50-v2 on Imagenet
We train ResNet-50-v2 on Imagenet for 150,000 steps using SGD with momentum and a batch size of 1024. We do not use batch normalization since this could confound our analysis of the training trajectory. We instead add bias terms to every convolutional layer. We decay the learning rate linearly with an initial learning rate of 0.0345769 to a final learning rate a factor of 10 lower by 141,553 steps, at which point we keep the learning rate constant. We set the momentum to 0.9842. The network achieves a validation accuracy of 71.46%.
9 Gallery of PCA projected trajectories
We present here tableaux of the PCA projections of various trajectories. We show in Fig. 11 four tableaux of the PCA projections of the trajectories of high-dimensional Ornstein-Uhlenbeck processes with different values of . For the trajectories are almost identical to a high-dimensional random walk, as they should be since the process was sampled for only 1000 steps. Once we have the trajectories start to visibly deviate from those of a high-dimensional random walk. For larger the deviations continue to grow until they become unrecognizable at because 1000 steps corresponds to many crossing times on the high dimensional sphere on which the process takes place.
In Fig. 12 we present tableaux of the PCA projections of the linear model trained on CIFAR-10. The trajectory of the entire training process somewhat resembles a high-dimensional random walk, though because the model makes larger updates at earlier steps than at later ones there are long tails on the PCA projected trajectories. The model’s trajectory most closely resembles a high-dimensional random walk early in training, but towards the end the higher components become dominated by noise, implying that these components more closely resemble a converged Ornstein-Uhlenbeck process. This corresponds with the flattening of the distribution of eigenvalues in Fig. 3.
In Fig. 13 we present tableaux of the PCA projections of ResNet-50-v2 trained on Imagenet. Perhaps remarkably, these trajectories resemble a high-dimensional random walk much more closely than the linear model. However, as in the case of the linear model, the resemblance deteriorates later in training.