Implicit Regularization Towards Rank Minimization in ReLU Networks

1 Introduction

A central puzzle in the theory of deep learning is how neural networks generalize even when trained without any explicit regularization, and when there are far more learnable parameters than training examples. In such an underdetermined optimization problem, there are many global minima with zero training loss, and gradient descent seems to prefer solutions that generalize well (see

zhang2017understanding). Hence, it is believed that gradient descent induces an implicit regularization (or implicit bias) (neyshabur2015search; neyshabur2017exploring), and characterizing this regularization/bias has been a subject of extensive research.

Several works in recent years studied the relationship between the implicit regularization in linear neural networks and rank minimization. A main focus is on the matrix factorization problem, which corresponds to training a depth-2 linear neural network with multiple outputs w.r.t. the square loss, and is considered a well-studied test-bed for studying implicit regularization in deep learning. gunasekar2018implicit initially conjectured that the implicit regularization in matrix factorization can be characterized by the nuclear norm of the corresponding linear predictor. This conjecture was further studied in a string of works (e.g., belabbas2020implicit; arora2019implicit; razin2020implicit) and was formally refuted by li2020towards. razin2020implicit conjectured that the implicit regularization in matrix factorization can be explained by rank minimization, and also hypothesized that some notion of rank minimization may be key to explaining generalization in deep learning. li2020towards established evidence that the implicit regularization in matrix factorization is a heuristic for rank minimization. razin2021implicit

studied implicit regularization in tensor factorization (a generalization of matrix factorization). They demonstrated, both theoretically and empirically, implicit bias towards low-rank tensors. Going beyond factorization problems,

ji2018gradient; ji2020directional showed that in linear networks of output dimension

1

, gradient flow (GF) w.r.t. exponentially-tailed classification losses converges to networks where the weight matrix of every layer is of rank

1

However, once we move to nonlinear neural networks (which are by far the more common in practice), things are less clear. Empirically, a series of works studying neural network compression (cf. denton2014exploiting; yu2017compressing; alvarez2017compression; arora2018stronger; tukan2020compressed) showed that replacing the weight matrices by low-rank approximations results in only a small drop in accuracy. This suggests that the weight matrices in practice are not too far from being low-rank. However, whether they provably behave this way remains unclear.

In this work we consider fully-connected nonlinear networks employing the popular ReLU activation function, and study whether GF is biased towards networks where the weight matrices have low ranks. On the negative side, we show that already for small (depth and width

2

) ReLU networks, there is no rank-minimization bias in a rather strong sense. On the positive side, for deeper and possibly wider overparameterized networks, we identify reasonable settings where GF is biased towards low-rank solutions. In more details, our contributions are as follows:

[itemsep=3pt,parsep=3pt]
We begin by considering depth- $2$ width- $2$ ReLU networks with multiple outputs, trained with the square loss. li2020towards gave evidence that in linear networks with the same architecture, the implicit bias of GF can be characterized as a heuristic for rank minimization. In contrast, we show that in ReLU networks, the situation is quite different: Specifically, we show that GF does not converge to a low-rank solution, already for the simple case of datasets of size $2$ , ${(x_{1}, y_{1}), (x_{2}, y_{2})} \subseteq S^{1} \times S^{1}$ , whenever the angle between $x_{1}$ and $x_{2}$ is in $(π / 2, π)$ and $y_{1}, y_{2}$
are linearly independent. Thus, rank minimization does not occur even if we just consider “most” datasets of this size. Moreover, we show that with at least constant probability, the solutions that GF converges to are not even close to have low rank, under any reasonable approximate rank metric. We also demonstrate these results empirically.
Next, for ReLU networks that are overparameterized in terms of depth and have width $\geq 2$ , we identify interesting settings in which GF is biased towards low ranks:
- [leftmargin=*,itemsep=3pt,parsep=3pt, topsep=3pt, partopsep=3pt]
- First, we consider ReLU networks trained w.r.t. the square loss. We show that for sufficiently deep networks, if GF converges to a network that attains zero loss and minimizes the $ℓ_{2}$ norm of the parameters, then the average ratio between the spectral and the Frobenius norms of the weight matrices is close to $1$ . Since the squared inverse of this ratio is the stable rank (which is a continuous approximation of the rank, and equals $1$ if and only if the matrix has rank $1$ ), the result implies a bias towards low ranks. While GF in ReLU networks w.r.t. the square loss is not known to be biased towards solutions that minimize the $ℓ_{2}$ norm, in practice it is common to use explicit $ℓ_{2}$ regularization, which encourages norm minimization. Thus, our result suggests that GF in deep networks trained with the square loss and explicit $ℓ_{2}$ regularization encourages rank minimization.
- Shifting our attention to binary classification problems, we consider ReLU networks trained with exponentially-tailed classification losses. By lyu2019gradient, GF in such networks is biased towards networks that maximize the $ℓ_{2}$ margin. We show that for sufficiently deep networks, maximizing the margin implies rank minimization, where the rank is measured by the ratio between the norms as in the former case.

Additional Related Work

The implicit regularization in matrix factorization and linear neural networks with the square loss was extensively studied, as a first step toward understanding implicit regularization in more complex models (see, e.g., gunasekar2018implicit; razin2020implicit; arora2019implicit; belabbas2020implicit; eftekhari2020implicit; li2018algorithmic; ma2018implicit; woodworth2020kernel; gidel2019implicit; li2020towards; yun2020unifying; azulay2021implicit; razin2021implicit). As we already discussed, some of these works showed bias toward low ranks.

The implicit regularization in nonlinear neural networks with the square loss was studies in several works. oymak2019overparameterized showed that under some assumptions, gradient descent in certain nonlinear models is guaranteed to converge to a zero-loss solution with a bounded $ℓ_{2}$ norm. williams2019gradient and jin2020implicit studied the dynamics and implicit bias of gradient descent in wide depth- $2$ ReLU networks with input dimension $1$ . vardi2021implicit and azulay2021implicit

studied the implicit regularization in single-neuron networks. In particular,

vardi2021implicit showed that in single-neuron networks and single-hidden-neuron networks with the ReLU activation, the implicit regularization cannot be expressed by any non-trivial regularization function. Namely, there is no non-trivial regularization function

R (θ)

, where

θ

are the parameters of the model, such that if GF with the square loss converges to a global minimum, then it is a global minimum that minimizes

R

. However, this negative result does not imply that GF is not implicitly biased towards low-rank solutions, for two reasons. First, bias toward low ranks would not have implications in the case of networks of width

1

that these authors studied, and hence it would not contradict their negative result. Second, their result rules out the existence of a non-trivial regularization function which expresses the implicit bias for all possible datasets and initializations, but it does not rule out the possibility that GF acts as a heuristic for rank minimization, in the sense that it minimizes the ranks for “most” datasets and initializations.

The implicit bias of neural networks in classification tasks was also widely studied in recent years. soudry2018implicit showed that gradient descent on linearly-separable binary classification problems with exponentially-tailed losses, converges to the maximum $ℓ_{2}$

-margin direction. This analysis was extended to other loss functions, tighter convergence rates, non-separable data, and variants of gradient-based optimization algorithms

(nacson2019convergence; ji2018risk; ji2020gradient; gunasekar2018characterizing; shamir2021gradient; ji2021characterizing). lyu2019gradient and ji2020directional showed that GF on homogeneous neural networks, with exponentially-tailed losses, converges in direction to a KKT point of the maximum-margin problem in the parameter space. Similar results under stronger assumptions were previously obtained in nacson2019lexicographic; gunasekar2018bimplicit. vardi2021margin studied in which settings this KKT point is guaranteed to be a global/local optimum of the maximum-margin problem. The implicit bias in fully-connected linear networks was studied by ji2020directional; ji2018gradient; gunasekar2018bimplicit. As already mentioned, these results imply that GF minimizes the ranks of the weight matrices in linear fully-connected networks. The implicit bias in diagonal and convolutional linear networks was studied in gunasekar2018bimplicit; moroshko2020implicit; yun2020unifying. The implicit bias in infinitely-wide two-layer homogeneous neural networks was studied in chizat2020implicit.

Organization. In Sec. 2 we provide necessary notations and definitions. In Sec. 3 we state our negative results for depth- $2$ networks. In Sec. 4 and 5 we state our positive results for deep ReLU networks. In Sec. 6 we describe the ideas for the proofs of the main theorems, with all formal proofs deferred to the appendix.

2 Preliminaries

Notations.

We use boldface letters to denote vectors. For $x \in R^{d}$ we denote by $∥ x ∥$ the Euclidean norm. For a matrix $X$ we denote by ${∥ X ∥}_{F}$ the Frobenius norm and by ${∥ X ∥}_{σ}$ the spectral norm. We denote $R_{+} := {x \in R : x \geq 0}$ . For an integer $d \geq 1$ we denote $[d] := {1, \dots, d}$ . The angle between a pair of vectors $u_{1}, u_{2} \in R^{d}$ is $∡ (u_{1}, u_{2}) := arccos (\frac{u_{1}^{⊤} u_{2}}{∥ u_{1} ∥ \cdot ∥ u_{2} ∥}) \in [0, π]$ . The unit $d$ -sphere is $S^{d} := {u \in R^{d + 1} ∣ ∥ u ∥ = 1}$ . An open $d$ -ball that is centered at the origin is denoted by $B_{d} (ϵ) := {u \in R^{d} ∣ ∥ u ∥ < ϵ}$ for some $ϵ \in R$ . The closure of a set $A \in R^{d}$ , denoted as $cl A$ , is the intersection of all closed sets containing $A$ . The boundary of $A$ is $\partial A := (cl A) \cap (cl (R^{d} ∖ A))$ .

Neural networks.

A fully-connected neural network $N_{θ}$ of depth $k \geq 2$ is parameterized by a collection $θ := [W^{(l)}]_{l = 1}^{k}$ of weight matrices, such that for every layer $l \in [k]$ we have $W^{(l)} \in R^{d_{l} \times d_{l - 1}}$ . Thus, $d_{l}$ denotes the number of neurons in the $l$ -th layer, i.e., the width of the layer. We denote by $d_{in} := d_{0}$ , $d_{out} := d_{k}$ the input and output dimensions. The neurons in layers $[k - 1]$ are called hidden neurons. A fully-connected network computes a function $N_{θ} : R^{d_{in}} \to R^{d_{% out}}$ defined recursively as follows. For an input $x \in R^{d_{in}}$ we set $h_{0}^{'} := x$ , and define for every $j \in [k - 1]$ the input to the $j$ -th layer as $h_{j} := W^{(j)} h_{j - 1}^{'}$ , and the output of the $j$ -th layer as $h_{j}^{'} := σ (h_{j})$ , where $σ : R \to R$ is an activation function that acts coordinate-wise on vectors. In this work we focus on the ReLU activation $σ (z) = max {0, z}$ . Finally, we define $N_{θ} (x) := W^{(k)} h_{k - 1}^{'}$ . Thus, there is no activation in the output neurons. The width of the network $N_{θ}$ is the maximal width of its layers, i.e., ${max}_{l \in [k]} d_{l}$ . We sometimes apply the activation function $σ$ also on matrices, in which case it acts entry-wise. The parameters $θ$ of the neural network are given by a collection of matrices, but we often view $θ$ as the vector obtained by concatenating the matrices in the collection. Thus, $∥ θ ∥$ denotes the $ℓ_{2}$ norm of the vector $θ$ .

We often consider depth- $2$ networks. For matrices $W \in R^{d_{1} \times d_{0}}$ and $V \in R^{d_{2} \times d_{1}}$ we denote by $N_{W, V}$ the depth- $2$ ReLU network where $W^{(1)} = W$ and $W^{(2)} = V$ . We denote the $w_{1}^{⊤}, \dots, w_{d_{1}}^{⊤}$ the rows of $W$ , namely, the incoming weight vectors to the neurons in the hidden layer, and by $v_{1}, \dots, v_{d_{1}}$ the columns of $V$ , namely, the outgoing weight vectors from the neurons in the hidden layer.

Let $x_{1}, \dots, x_{n}$ be $n$ inputs, and let $X \in R^{d_{in} \times n}$ be a matrix whose columns are $x_{1}, \dots, x_{n}$ . We denote by $N_{θ} (X) \in R^{d_{out} \times n}$ the matrix whose $i$ -th column is $N_{θ} (x_{i})$ .

Optimization problem and gradient flow (GF).

Let $S = {(x_{i}, y_{i})}_{i = 1}^{n} \subseteq R^{d_{% i n}} \times R^{d_{out}}$ be a training dataset. We often represent the dataset by matrices $(X, Y) \in R^{d_{in} \times n} \times R^{d_{% out} \times n}$ . For a neural network $N_{θ}$ we consider empirical-loss minimization w.r.t. the square loss. Thus, the objective is given by:

(1)

We assume that the data is realizable, that is, ${min}_{θ} L (θ) = 0$ . Moreover, we focus on settings where the network is overparameterized, in the sense that $L$ has multiple (or even infinitely many) global minima.

We consider gradient flow (GF) on the objective given in Eq. (1). This setting captures the behavior of gradient descent with an infinitesimally small step size. Let $θ (t)$ be the trajectory of GF. Starting from an initial point $θ (0)$ , the dynamics of $θ (t)$ is given by the differential equation $\frac{d θ (t)}{d t} = - \nabla L_{X, Y} (θ (t))$ . Note that the ReLU function is not differentiable at $0$ . Practical implementations of gradient methods define the derivative $σ^{'} (0)$ to be some constant in $[0, 1]$ . In this work we assume for convenience that $σ^{'} (0) = 0$ . We say that GF converges if ${lim}_{t \to \infty} θ (t)$ exists. In this case, we denote $θ (\infty) := {lim}_{t \to \infty} θ (t)$ .

3 Gradient flow does not even approximately minimize ranks

In this section we consider rank minimization in depth- $2$ networks $N_{W, V}$ trained with the square loss. We show that even for the simple case of size- $2$ datasets, under mild assumptions, GF does not converge to a minimum-rank solution even approximately.

In what follows, we consider ReLU networks with vector-valued outputs, since for linear networks with the same architecture it was shown that GF can be viewed as a heuristic for rank minimization (cf. li2020towards; razin2020implicit). Specifically, let $(X, Y) \in R^{2 \times 2} \times R^{2 \times 2}$ be a training dataset, and let $W, V \in R^{2 \times 2}$ be weight matrices such that $N_{W, V} (X) = V σ (W X)$ is a zero-loss solution. Note that if $rank (Y) = 2$ then we must have $rank (V) = 2$ : Indeed, by definition of $N_{W, V}$ , we necessarily have $rank (Y) = rank (N_{W, V} (X)) \leq rank (V)$ . Therefore, to understand rank minimization in this simple setting, we consider the rank of $W$ in a zero-loss solution. Trivially, $rank (W) \leq 2$ , so $W$ can be considered low-rank only if $rank (W) \leq 1$ .

To make the setting non-trivial, we need to show that such low-rank zero-loss solutions exist at all. The following theorem shows that this is true for almost all size- $2$ datasets:

Theorem 1.

Given any labeled dataset $(X, Y) \in R^{2 \times 2} \times R^{2 \times 2}$ of two inputs $x_{1}, x_{2} \in R^{2}$ with a strictly positive angle between them, i.e., $∡ (x_{1}, x_{2}) > 0$ , there exists a zero-loss solution $N_{W, V}$ with $W, V \in R^{2 \times 2}$ , such that $rank (W) = 1$ .

The theorem follows by constructing a network where the weight vectors of the neurons in the first layer have opposite directions (and hence the weight matrix is of rank $1$ ), such that each neuron is active for exactly one input. Then, it is possible to show that for an appropriate choice of the weights in the second layer the network achieves zero loss. See Appendix A for the formal proof.

Thm. 1 implies that zero-loss solutions of rank $1$ exist. However, we now show that GF does not converge to such solutions. We prove this result under the following assumptions:

Assumption 1.

The two target vectors $y_{1}, y_{2}$ are on the unit sphere $S^{1}$ and are linearly independent.

Assumption 2.

The two inputs $x_{1}, x_{2}$ are on the unit sphere $S^{1}$ , and satisfy $\frac{π}{2} < ∡ (x_{1}, x_{2}) < π$ .

The assumptions that $x_{i}, y_{i}$ are of unit norm are mostly for technical convenience, and we believe that they are not essential.

Then, we have:

Theorem 2.

∥ w_{i} (0) ∥ < min {\frac{1}{2}, \frac{\sqrt{3}}{2} cos \frac{∡ (x_{1}, x_{2})}{2}}

and $∥ v_{i} (0) ∥ < \frac{1}{2}$ for all $i \in {1, 2}$ . If GF converges to a zero-loss solution $N_{W (\infty), V (\infty)}$ , then $rank (W (\infty)) = 2$ .

By the above theorem, GF does not minimize the rank even in a very simple setting where the dataset contains two inputs with angle larger than $π / 2$ (as long as the initialization point is sufficiently close to $0$

). In particular, if the dataset is drawn from the uniform distribution on the sphere then this condition holds with probability

1 / 2

While Thm. 2

shows that GF does not minimize the rank, it does not rule out the possibility that it converges to a solution which is close to a low-rank solution. There are many ways to define such closeness, such as the ratio of the Frobenius and spectral norms, the Frobenius distance from a low-rank solution, or the exponential of the entropy of the singular values (cf.

rudelson2007sampling; sanyal2019stable; razin2020implicit; roy2007effective). However, for

2 \times 2

matrices they all boil down to either having the two rows of the matrix being nearly aligned, or having at least one of them very small (at least compared to the other). In the following theorem, we show that under the assumptions stated above, for any fixed dataset, with at least constant probability, GF converges to a zero-loss solution, where the two row vectors are bounded away from

0

, the ratio of their norms are bounded, and the angle between them is bounded away from

0

and from

π

(all by explicit constants that depend just on the dataset and are large in general). Thus, with at least constant probability, GF does not minimize any reasonable approximate notion of rank.

Theorem 3.

Let $(X, Y) \in R^{2 \times 2} \times R^{2 \times 2}$ be a labeled dataset that satisfies Assumptions 1 and 2. Consider GF w.r.t. the loss function $L_{X, Y} (W, V)$ from Eq. (1). Suppose that $W, V \in R^{2 \times 2}$ are initialized such that for all $i \in {1, 2}$ we have $v_{i} (0) = 0$ , and $w_{i} (0)$ is drawn from a spherically symmetric distribution with

∥ w_{i} (0) ∥ \leq \frac{\sqrt{3}}{2} min {sin (\frac{π - ∡ (x_{1}, x_{2})}{4}), sin (∡ (x_{1}, x_{2}) - \frac{π}{2})} .

Let $E$ be the event that GF converges to a zero-loss solution $N_{W (\infty), V (\infty)}$ such that

$∡ (w_{1} (\infty), w_{2} (\infty)) \in [\frac{π}{2} - (∡ (x_{1}, x_{2}) - \frac{π}{2}), \frac{3 π}{4} + \frac{∡ (x_{1}, x_{2}) - π / 2}{2}]$ ,
$∥ w_{i} (\infty) ∥ \in (\frac{\sqrt{3}}{2}, \sqrt{\frac{1}{4} + \frac{4}{3 {(sin ∡ (x_{1}, x_{2}))}_{1}^{2}}})$ for all $i \in {1, 2}$ .

Then, $Pr [E] \geq 2 \cdot {(\frac{∡ (x_{1}, x_{2})}{2 π})}^{2}$ .

We note that in Thm. 3 the weights in the second layer are initialized to zero, while in Thm. 2 the assumption on the initialization is weaker. This difference is for technical convenience, and we believe that Thm. 3 should hold also under weaker assumptions on the initialization, as the next empirical result demonstrates.

3.1 An empirical result

Our theorems imply that for standard initialization schemes, GF will not converge close to low-rank solutions, with some positive probability. We now present a simple experiment that corroborates this and suggests that, furthermore, this holds with high probability.

Specifically, we trained ReLU networks in the same setup as in the previous section (w.r.t. two $2 \times 2$ weight matrices $W^{(1)}, W^{(2)}$ ) on the two data points ${(x_{i}, y_{i})}_{i = 1}^{2}$ where $y_{1}, y_{2}$ are the standard basis vectors in $R^{2}$ , and $x_{1}, x_{2}$ are $(1, 0.99)$ and $(- 1, 0.99)$ normalized to have unit norm. At initialization, every row of $W^{(1)}$ and every column of $W^{(2)}$ is sampled uniformly at random from the sphere of radius $10^{- 4}$ around the origin. To simulate GF, we performed $3 \cdot 10^{6}$ epochs of full-batch gradient descent of step size $10^{- 4}$ , w.r.t. the square loss. Of $288$ repeats of this experiment, $79$ converged to negligible loss (defined as $< 10^{- 4}$ ). In Fig. 1, we plot a histogram of the stable (numerical) ranks of the resulting weight matrices, i.e. the ratio ${∥ ∥ W^{(ℓ)} ∥ ∥}_{F}^{2} / {∥ ∥ W^{(ℓ)} ∥ ∥}_{σ}^{2}$ of layer $ℓ \in [2]$ . The figure clearly suggests that whenever convergence to zero loss occurs, the solutions are all of rank $2$ , and none are even close to being low-rank (in terms of the stable rank).

Figure 1: A histogram of the stable (numerical) ranks at convergence. In all runs, we converge to networks with stable ranks which seem bounded away from $1$ . Namely, gradient descent does not even approximately minimize the ranks.

4 Rank minimization in deep networks with small $ℓ_{2}$ norm

When training neural networks with gradient descent, it is common to use explicit $ℓ_{2}$ regularization on the parameters. In this case, gradient descent is biased towards solutions that minimize the $ℓ_{2}$ norm of the parameters. We now show that in deep overparameterized ReLU networks, if GF converges to a zero-loss solution that minimizes the $ℓ_{2}$ norm, then the ratios between the Frobenius and the spectral norms in the weight matrices tend to be small (we use here the ratio between these norms as a continuous surrogate for the exact rank, as discussed in the previous section). Formally, we have the following:

Theorem 4.

Let ${(x_{i}, y_{i})}_{i = 1}^{n} \subseteq R^{d_{in}} \times R_{+}$ be a dataset, and assume that there is $i \in [n]$ with $∥ x_{i} ∥ \leq 1$ and $y_{i} \geq 1$ . Assume that there is a fully-connected neural network $N$ of width $m \geq 2$ and depth $k \geq 2$ , such that for all $i \in [n]$ we have $N (x_{i}) = y_{i}$ , and the weight matrices $W_{1}, \dots, W_{k}$ of $N$ satisfy ${∥ W_{i} ∥}_{F} \leq B$ for some $B > 0$ . Let $N_{θ}$ be a fully-connected neural network of width $m^{'} \geq m$ and depth $k^{'} > k$ parameterized by $θ$ . Let $θ^{*} = [W_{1}^{*}, \dots, W_{k^{'}}^{*}]$ be a global optimum of the following problem:

min θ ∥ θ ∥ s.t. \forall i \in [n] N_{θ} (x_{i}) = y_{i} .

(2)

Then,

\frac{1}{k^{'}} k^{'} \sum i = 1 \frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}} \geq {(\frac{1}{B})}^{\frac{}{k} k^{'}} .

(3)

Equivalently, we have the following upper bound on the harmonic mean of the ratios

\frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}}

\frac{k^{'}}{\sum_{i = 1}^{k^{'}} {(\frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}})}^{- 1}} \leq B^{\frac{k}{k^{'}}} .

(4)

By the above theorem if $k^{'}$ is much larger than $k$ , then the average ratio between the spectral and the Frobenius norms (Eq. (3)) is at least roughly $1$ . Likewise, the harmonic mean of the ratio between the Frobenius and the spectral norms (Eq. (4)), namely, the square root of the stable rank, is at most roughly $1$ . Noting that both these ratios equal $1$ if and only if the matrix is of rank $1$ , we see that there is a bias towards low-rank solutions as the depth $k^{'}$ of the trained network increases. Note that the result does not depend on the width of the networks. Thus, even if the width $m^{'}$ is large, the average ratio is close to $1$ . Also, note that the network $N$ of depth $k$ in the theorem might have high ranks (e.g., rank $m$ for each weight matrix), but once we consider networks of a large depth $k^{'}$ then the dataset becomes realizable by a network of small average rank, and GF converges to such a network.

5 Rank minimization in deep networks with exponentially-tailed losses

In this section, we turn to consider GF in classification tasks with exponentially-tailed losses, namely, the exponential loss or the logistic loss.

Let us first formally define the setting. We consider neural networks of output dimension $1$ , i.e., $d_{out} = 1$ . Let $S = {(x_{i}, y_{i})}_{i = 1}^{n} \subseteq R^{d} \times {- 1, 1}$ be a binary classification training dataset. Let $X \in R^{d_{in} \times n}$ and $y \in R^{n}$ be the data matrix and labels that correspond to $S$ . Let $N_{θ}$ be a neural network parameterized by $θ$ . For a loss function $ℓ : R \to R$ , the empirical loss of $N_{θ}$ on the dataset $S$ is

L_{X, y} (θ) := n \sum i = 1 ℓ (y_{i} N_{θ} (x_{i})) .

(5)

We focus on the exponential loss $ℓ (q) = e^{- q}$ and the logistic loss $ℓ (q) = log (1 + e^{- q})$ . We say that the dataset is

correctly classified

by the network

N_{θ}

if for all

i \in [n]

we have

y_{i} N_{θ} (x_{i}) > 0

. We consider GF on the objective given in Eq. (5). We say that a network

N_{θ}

is homogeneous if there exists

M > 0

such that for every

α > 0

and

θ, x

we have

N_{α θ} (x) = α^{M} N_{θ} (x)

. Note that fully-connected ReLU networks are homogeneous. We say that a trajectory

θ (t)

of GF converges in direction to

~ θ

lim t \to \infty \frac{θ (t)}{∥ θ (t) ∥} = \frac{~ θ}{∥ ~ θ ∥} .

The following well-known result characterizes the implicit bias in homogeneous neural networks trained with the logistic or the exponential loss:

Lemma 1 (Paraphrased from lyu2019gradient and ji2020directional).

Let $N_{θ}$ be a homogeneous ReLU neural network. Consider minimizing the average of either the exponential or the logistic loss over a binary classification dataset using GF. Suppose that the average loss converges to zero as $t \to \infty$ . Then, GF converges in direction to a first order stationary point (KKT point) of the following maximum margin problem in parameter space:

min θ \frac{1}{2} {∥ θ ∥}^{2} s.t. \forall i \in [n] y_{i} N_{θ} (x_{i}) \geq 1 .

(6)

The above lemma suggests that GF tends to converge in direction to a network with margin $1$ and small $ℓ_{2}$ norm. In the following theorem we show that in deep overparameterized ReLU networks, if GF converges in direction to an optimal solution of Problem 6 (from the above lemma) then the ratios between the Frobenius and the spectral norms in the weight matrices tend to be small. Formally, we have the following:

Theorem 5.

Let ${(x_{i}, y_{i})}_{i = 1}^{n} \subseteq R^{d_{in}} \times {- 1, 1}$ be a binary classification dataset, and assume that there is $i \in [n]$ with $∥ x_{i} ∥ \leq 1$ . Assume that there is a fully-connected neural network $N$ of width $m \geq 2$ and depth $k \geq 2$ , such that for all $i \in [n]$ we have $y_{i} N (x_{i}) \geq 1$ , and the weight matrices $W_{1}, \dots, W_{k}$ of $N$ satisfy ${∥ W_{i} ∥}_{F} \leq B$ for some $B > 0$ . Let $N_{θ}$ be a fully-connected neural network of width $m^{'} \geq m$ and depth $k^{'} > k$ parameterized by $θ$ . Let $θ^{*} = [W_{1}^{*}, \dots, W_{k^{'}}^{*}]$ be a global optimum of Problem 6. Namely, $θ^{*}$ parameterizes a minimum-norm fully-connected network of width $m^{'}$ and depth $k^{'}$ that labels the dataset correctly with margin $1$ . Then, we have

\frac{1}{k^{'}} k^{'} \sum i = 1 \frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}} \geq \frac{1}{\sqrt{2}} \cdot {(\frac{\sqrt{2}}{B})}^{\frac{k}{k^{'}}} \cdot \sqrt{\frac{k^{'}}{k^{'} + 1}} .

(7)

Equivalently, we have the following upper bound on the harmonic mean of the ratios $\frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}}$ :

\frac{k^{'}}{\sum_{i = 1}^{k^{'}} {(\frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}})}^{- 1}} \leq \sqrt{2} \cdot {(\frac{B}{\sqrt{2}})}^{\frac{k}{k^{'}}} \cdot \sqrt{\frac{k^{'} + 1}{k^{'}}} .

(8)

By the above theorem, if $k^{'}$ is much larger than $k$ , then the average ratio between the spectral and the Frobenius norms (Eq. (7)) is at least roughly $1 / \sqrt{2}$ . Likewise, the harmonic mean of the ratio between the Frobenius and the spectral norms (Eq. (8)), i.e., the square root of the stable rank, is at most roughly $\sqrt{2}$ . Note that the result does not depend on the width of the networks. Thus, it holds even if the width $m^{'}$ is very large. Similarly to the case of Thm. 4, we note that the network $N$ of depth $k$ might have high ranks (e.g., rank $m$ for each weight matrix), but once we consider networks of a large depth $k^{'}$ , then the dataset becomes realizable by a network of small average rank, and GF converges to such a network.

The combination of the above result with Lemma 1 suggests that, in overparameterized deep fully-connected networks, GF tends to converge in direction to neural networks with low ranks. Note that we consider the exponential and the logistic losses, and hence if the loss tends to zero as $t \to \infty$ , then we have $∥ θ (t) ∥ \to \infty$ . To conclude, in our case, the parameters tend to have an infinite norm and to converge in direction to a low-rank solution. Moreover, note that the ratio between the spectral and the Frobenius norms is invariant to scaling, and hence it suggests that after a sufficiently long time, GF tends to reach a network with low ranks.

6 Proof ideas

In this section we describe the main ideas for the proofs of Theorems 2, 3, 4 and 5. The full proofs are given in the appendix.

6.1 Theorem 2

We define the following regions (see Fig. 2):

	$D := {w \in R^{2} ∣ \forall i \in {1, 2}, σ (w^{⊤} x_{i}) \leq 0},$
	$S := {w \in R^{2} ∣ \forall i \in {1, 2}, σ (w^{⊤} x_{i}) > 0},$
	$S_{1} := {w \in R^{2} ∣ σ (w^{⊤} x_{1}) > 0, σ (w^{⊤} x_{2}) \leq 0},$
	$S_{2} := {w \in R^{2} ∣ σ (w^{⊤} x_{2}) > 0, σ (w^{⊤} x_{1}) \leq 0} .$

Figure 2: Regions $D$ , $S_{1}$ , $S_{2}$ , $S$ .

Intuitively, $D$ defines the “dead” region where the relevant neuron will output $0$ on both $x_{1}, x_{2}$ ; $S$ is the “active” region where the relevant neuron will output a positive output on both $x_{1}, x_{2}$ ; and $S_{1}, S_{2}$ are the “partially active” regions, where the relevant neuron will output a positive output on one point, and $0$ on the other.

Assume towards contradiction that GF converges to some zero-loss network

N_{W (\infty), V (\infty)}

with

rank (W (\infty)) < 2

. Since

N_{W (\infty), V (\infty)}

attains zero loss, then

Y = V (\infty) σ (W (\infty) X)

, and hence

2 = rank (Y) = rank (V (\infty) σ (W (\infty) X)) \leq rank (σ (W (\infty) X)) .

(9)

Therefore, the weight vectors $w_{1} (\infty)$ and $w_{2} (\infty)$ are not in the region $D$ . Indeed, if $w_{1} (\infty)$ or $w_{2} (\infty)$ are in $D$ , then at least one of the rows of $σ (W (\infty) X)$ is zero, in contradiction to Eq. (9). In particular, it implies that $w_{1} (\infty)$ and $w_{2} (\infty)$ are non-zero. Since by our assumption we have $rank (W (\infty)) < 2$ , then we conclude that $rank (W (\infty)) = 1$ . We denote $w_{2} (\infty) = α w_{1} (\infty)$ where $α \neq 0$ . Note that if $α > 0$ , then $σ (w_{2} (\infty)^{⊤} x_{j}) = α σ (w_{1} (\infty)^{⊤} x_{j})$ for all $j \in {1, 2}$ , in contradiction to Eq. (9). Thus, $α < 0$ . Since we also have $w_{1} (\infty), w_{2} (\infty) \notin D$ , then one of these weight vectors is in $S_{1} ∖ \partial S_{1}$ and the other is in $S_{2} ∖ \partial S_{2}$ (as can be seen from Fig. 2). Assume w.l.o.g. that $w_{1} (\infty) \in S_{1} ∖ \partial S_{1}$ and $w_{2} (\infty) \in S_{2} ∖ \partial S_{2}$ .

By observing the gradients of $L_{X, Y}$ w.r.t. $w_{i}$ for $i \in {1, 2}$ , the following facts follow. First, if $w_{i} (t) \in D$ at some time $t$ , then $\frac{d}{d t} w_{i} (t) = 0$ , hence $w_{i}$ remains at $D$ indefinitely, in contradiction to $w_{i} (\infty) \in S_{i} ∖ \partial S_{i}$ . Thus, the trajectory $w_{i} (t)$ does not visit $D$ . Second, if $w_{i} (t) \in S_{i}$ at time $t$ , then $\frac{d}{d t} w_{i} (t) \in span {x_{i}}$ . Since $w_{i} (\infty) \in S_{i} ∖ \partial S_{i}$ , we can consider the last time $t^{'}$ that $w_{i}$ enters $S_{i}$ , which can be either at the initialization (i.e., $t^{'} = 0$ ) or when moving from $S$ (i.e., $t^{'} > 0$ ). For all time $t \geq t^{'}$ we have $\frac{d}{d t} w_{i} (t) \in span {x_{i}}$ . It allows us to conclude that $w_{i} (\infty)$ must be in a region $A_{i}$ which is illustrated in Fig. 3 (by the union of the orange and green regions).

Figure 3: Regions $F_{i}$ (in green) and $A_{i}$ (the union of the green and orange regions). A dashed line marks an open boundary.

Furthermore, we show that $∥ w_{i} (\infty) ∥$ cannot be too small, namely, obtaining a lower bound on $∥ w_{i} (\infty) ∥$ . First, a theorem from du2018algorithmic implies that ${∥ w_{i} (t) ∥}^{2} - {∥ v_{i} (t) ∥}^{2}$ remains constant throughout the training. Since at the initialization both $∥ w_{i} (0) ∥$ and $∥ v_{i} (0) ∥$ are small, the consequence is that $∥ v_{i} (\infty) ∥$ is small if $∥ w_{i} (\infty) ∥$ is small. Also, since $N_{W (\infty), V (\infty)}$ attains zero loss and $w_{i} (\infty) \in S_{i}$ for all $i \in {1, 2}$ , then we have $y_{i} = v_{i} (\infty) (w_{i} (\infty)^{⊤} x_{i})$ , namely, only the $i$ -th hidden neuron contributes to the output of $N_{W (\infty), V (\infty)}$ for the input $x_{i}$ . Since $∥ y_{i} ∥ = ∥ x_{i} ∥ = 1$ , it is impossible that both $∥ w_{i} (\infty) ∥$ and $∥ v_{i} (\infty) ∥$ are small. Hence, we are able to obtain a lower bound on $∥ w_{i} (\infty) ∥$ , which implies that $w_{i} (\infty)$ is in a region $F_{i}$ which is illustrated in Fig. 3.

Finally, we show that since $w_{1} (\infty) \in F_{1}$ and $w_{2} (\infty) \in F_{2}$ then the angle between $w_{1} (\infty)$ and $w_{2} (\infty)$ is smaller than $π$ , in contradiction to $w_{2} (\infty) = α w_{1} (\infty)$ .

6.2 Theorem 3

We show that if the initialization is such that $w_{1} (0) \in S_{1} ∖ \partial S_{1}$ and $w_{2} (0) \in S_{2} ∖ \partial S_{2}$ (or, equivalently, that $w_{1} (0) \in S_{2} ∖ \partial S_{2}$ and $w_{2} (0) \in S_{1} ∖ \partial S_{1}$ ), then GF converges to a zero-loss network, and $∥ w_{1} (\infty) ∥, ∥ w_{2} (\infty) ∥$ , $∡ (w_{1} (\infty), w_{2} (\infty))$ are in the required intervals. Since by simple geometric arguments we can show that the initialization satisfies this requirement with probability at least $2 \cdot {(\frac{∡ (x_{1}, x_{2})}{2 π})}^{2}$ , the theorem follows.

Indeed, suppose that $w_{1} (0) \in S_{1} ∖ \partial S_{1}$ and $w_{2} (0) \in S_{2} ∖ \partial S_{2}$ . We argue that GF converges to a zero-loss network and $∥ w_{1} (\infty) ∥, ∥ w_{2} (\infty) ∥, ∡ (w_{1} (\infty), w_{2} (\infty))$ are in the required intervals, as follows. By analyzing the dynamics of GF for such an initialization, we show that for all $t$ and $i$ we have $\frac{d}{d t} w_{i} (t) = C_{i} (t) x_{i}$ for some $C_{i} (t) \geq 0$ . Thus, $w_{i} (t)$ moves only in the direction of $x_{i}$ , and $w_{i} (t) \in S_{i} ∖ \partial S_{i}$ for all $t$ . Moreover, we are able to prove that these properties of the trajectories $w_{1} (t)$ and $w_{2} (t)$ imply that GF converges to a zero-loss network $N_{W (\infty), V (\infty)}$ . Then, by similar arguments to the proof of Thm. 2 we have $w_{i} (\infty) \in F_{i}$ for all $i \in {1, 2}$ , where $F_{i}$ are the regions from Fig. 3, and it allows us to obtain the required bounds on $∥ w_{1} (\infty) ∥, ∥ w_{2} (\infty) ∥$ , and $∡ (w_{1} (\infty), w_{2} (\infty))$ .

6.3 Theorems 4 and 5

The intuition for the proofs of both theorems can be roughly described as follows. If the dataset is realizable by a shallow network where the Frobenius norm of each layer is $B$ , then it is also realizable by a deep network where the Frobenius norm of each layer is $B^{*}$ , where $B^{*}$ is much smaller than $B$ . Moreover, if the network is sufficiently deep then $B^{*}$ is not much larger than $1$ . On the other hand, since for the input $x_{i}$ with $∥ x_{i} ∥ \leq 1$ the output of the network is of size at least $1$ , then the average spectral norm of the layers is at least $1$ . Hence, the average ratio between the spectral and the Frobenius norms cannot be too small.

We now describe the proof ideas in a bit more detail, starting with Thm. 4. We use the network $N$ of width $m$ and depth $k$ to construct a network $N^{'}$ of width $m^{'} \geq m$ and depth $k^{'} > k$ as follows. The first $k$ layers of $N^{'}$ are obtained by scaling the layers of $N$ by a factor $α := {(\frac{1}{B})}^{\frac{k^{'} - k}{k^{'}}}$ . Since the output dimension of $N$ is $1$ , then the $k$ -th hidden layer of $N^{'}$ has width $1$ . Then, the network $N^{'}$ has $k^{'} - k$ additional layers of width $1$ , such that the weight in each of these layers is $β := {(\frac{1}{B})}^{- \frac{k}{k^{'}}}$ . Overall, given input $x_{i}$ , we have

N^{'} (x_{i}) = N (x_{i}) \cdot α^{k} \cdot β^{k^{'} - k} = N (x_{i}) .

We denote by $θ^{'}$ the parameters of the network $N^{'}$ .

Let $θ^{*} = [W_{1}^{*}, \dots, W_{k^{'}}^{*}]$ be a global optimum of Problem 2. From the optimality of $θ^{*}$ it is possible to show that the layers in $θ^{*}$ must be balanced, namely, ${∥ ∥ W_{i}^{*} ∥ ∥}_{F} = {∥ ∥ W_{j}^{*} ∥ ∥}_{F}$ for all $i, j \in [k^{'}]$ . We denote by $B^{*}$ the Frobenius norm of the layers. From the global optimality of $θ^{*}$ we also have $∥ θ^{*} ∥ \leq ∥ ∥ θ^{'} ∥ ∥$ . Hence, by a calculation we can obtain

B^{*} \leq B^{\frac{k}{k^{'}}} .

Moreover, we show that since there is $i \in [n]$ with $∥ x_{i} ∥ \leq 1$ and $y_{i} \geq 1$ , then

\frac{1}{k^{'}} \sum i \in [k^{'}] {∥ ∥ W_{i}^{*} ∥ ∥}_{σ} \geq 1 .

Combining the last two displayed equations we get

\frac{1}{k^{'}} \sum i \in [k^{'}] \frac{{∥ ∥ W_{i}^{*} ∥ ∥}_{σ}}{{∥ ∥ W_{i}^{*} ∥ ∥}_{F}} = \frac{1}{B^{*}} \cdot \frac{1}{k^{'}} \sum i \in [k^{'}] {∥ ∥ W_{i}^{*} ∥ ∥}_{σ} \geq {(\frac{1}{B})}^{\frac{k}{k^{'}}},

as required.

Note that the arguments above do not depend on the ranks of the layers in $N$ . Thus, even if the weight matrices in $N$ have high ranks, once we consider deep networks which are optimal solutions to Problem 2, the ratios between the spectral and the Frobenius norms are close to $1$ .

We now turn to Thm. 5. The proof follows a similar approach to the proof of Thm. 4. However, here the outputs of the network $N$ can be either positive or negative. Hence, when constructing the network $N^{'}$ as above, we cannot have width $1$ in layers $k + 1, \dots, k^{'}$ , since the ReLU activation will not allow us to pass both positive and negative values. Still, we show that we can define a network $N^{'}$ such that the width in layers $k + 1, \dots, k^{'}$ is $2$ and we have $N^{'} (x_{i}) = N (x_{i})$ for all $i \in [n]$ . Then, the theorem follows by arguments similar to the proof of Thm. 4, with the required modifications.

Funding Acknowledgements

This research is supported in part by European Research Council (ERC) grant 754705.

References

Appendix A Proof of Thm. 1

Consider a matrix $W \in R^{2 \times 2}$ whose rows $w_{1}^{⊤}, w_{2}^{⊤}$ satisfy

	$w_{1}$	$= \frac{x_{1}}{∥ x_{1} ∥} - \frac{x_{2}}{∥ x_{2} ∥},$
	$w_{2}$	$= - w_{1} .$

The matrix $W$ has rank $1$ . To complete the proof, we need to show that we can choose a matrix $V \in R^{2 \times 2}$ such that $N_{W, V}$ attains zero loss. According to Lemma 2 below, it is enough to show that $w_{1}^{⊤} x_{1} > 0$ and $w_{1}^{⊤} x_{2} < 0$ . Since the angle between the inputs is strictly positive, namely, $∡ (x_{1}, x_{2}) > 0$ , it holds that $\frac{x_{1}^{⊤} x_{2}}{∥ x_{1} ∥ \cdot ∥ x_{2} ∥} < 1$ . Thus,

∥ x_{1} ∥ \cdot ∥ x_{2} ∥ - x_{1}^{⊤} x_{2} > 0 .

Then,

w_{1}^{⊤} x_{1}

= \frac{x_{1}^{⊤} x_{1}}{∥ x_{1} ∥} - \frac{x_{1}^{⊤} x_{2}}{∥ x_{2} ∥} = ∥ x_{1} ∥ - \frac{x_{1}^{⊤} x_{2}}{∥ x_{2} ∥} = \frac{∥ x_{1} ∥ \cdot ∥ x_{2} ∥ - x_{1}^{⊤} x_{2}}{∥ x_{2} ∥} > 0,

while

w_{1}^{⊤} x_{2}

= \frac{x_{1}^{⊤} x_{2}}{∥ x_{1} ∥} - \frac{x_{2}^{⊤} x_{2}}{∥ x_{2} ∥} = \frac{x_{1}^{⊤} x_{2}}{∥ x_{1} ∥} - ∥ x_{2} ∥ = \frac{x_{1}^{⊤} x_{2} - ∥ x_{1} ∥ \cdot ∥ x_{2} ∥}{} ∥ x_{1} ∥ < 0.

∎

Lemma 2.

Let $(X, Y) \in R^{d_{in} \times n} \times R^{d_{out} \times n}$ be a labeled dataset. Let $W \in R^{d_{hidden} \times d_{in}}$ . Suppose that for every data point $x_{j}$ there is at least one row $w_{i}^{⊤}$ in $W$ such that $w_{i}^{⊤} x_{j} > 0$ , and $w_{i}^{⊤} x_{ℓ} \leq 0$ for all $ℓ \neq j$ . Then, there exists $V$ such that $N_{W, V} (X) = Y$ .

Proof.

Consider the matrix $σ (W X)$ of size $d_{hidden} \times n$ , where $σ$ acts entrywise. Note that our assumption on $W$ implies that $rank (σ (W X)) = n$ . Thus, the $d_{hidden} \times d_{hidden}$ matrix $Z := [\begin{matrix} σ (W X)^{†} 0 \end{matrix}]$ satisfies $Z σ (W X) = [\begin{matrix} I_{n} 0 \end{matrix}]$ , where $A^{†}$ denotes the Moore-Penrose inverse of a matrix $A$ , and $I_{n}$ is the $n \times n$ identity matrix. Hence, the matrix $M := [\begin{matrix} Y & 0 \end{matrix}]$ of dimensions $d_{out} \times d_{hidden}$ yields $M Z σ (W X) = Y$ . By setting $V := M Z$ , the network $N_{W, V}$ achieves zero loss. Namely, $N_{W, V} (X) = Y$ . ∎

Appendix B Proof of Thm. 2

Definition 1.

We define the following regions of interest:

	$D$	$:= {w \in R^{2} ∣ \forall i \in {1, 2}, σ (w^{⊤} x_{i}) \leq 0},$
	$S$	$:= {w \in R^{2} ∣ \forall i \in {1, 2}, σ (w^{⊤} x_{i}) > 0} .$

Also, for $j \in {1, 2}$ we define

Implicit Regularization Towards Rank Minimization in ReLU Networks

Implicit Regularization in ReLU Networks with the Square Loss

Support Vectors and Gradient Dynamics for Implicit Bias in ReLU Networks

Training invariances and the low-rank phenomenon: beyond linear networks

Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widths

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks

On the Computational Efficiency of Training Neural Networks

Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions

1 Introduction

Additional Related Work

2 Preliminaries

Notations.

Neural networks.

Optimization problem and gradient flow (GF).

3 Gradient flow does not even approximately minimize ranks

Theorem 1.

Assumption 1.

Assumption 2.

Theorem 2.

Theorem 3.

3.1 An empirical result

4 Rank minimization in deep networks with small $ℓ_{2}$ norm

Theorem 4.

5 Rank minimization in deep networks with exponentially-tailed losses

Lemma 1 (Paraphrased from lyu2019gradient and ji2020directional).

Theorem 5.

6 Proof ideas

6.1 Theorem 2

6.2 Theorem 3

6.3 Theorems 4 and 5

Funding Acknowledgements

References

Appendix A Proof of Thm. 1

Lemma 2.

Proof.

Appendix B Proof of Thm. 2

Definition 1.

Implicit Regularization Towards Rank Minimization in ReLU Networks

Related Research

Implicit Regularization in ReLU Networks with the Square Loss

Support Vectors and Gradient Dynamics for Implicit Bias in ReLU Networks

Training invariances and the low-rank phenomenon: beyond linear networks

Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widths

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks

On the Computational Efficiency of Training Neural Networks

Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions

1 Introduction

Additional Related Work

2 Preliminaries

Notations.

Neural networks.

Optimization problem and gradient flow (GF).

3 Gradient flow does not even approximately minimize ranks

Theorem 1.

Assumption 1.

Assumption 2.

Theorem 2.

Theorem 3.

3.1 An empirical result

4 Rank minimization in deep networks with small ℓ2 norm

Theorem 4.

5 Rank minimization in deep networks with exponentially-tailed losses

Lemma 1 (Paraphrased from lyu2019gradient and ji2020directional).

Theorem 5.

6 Proof ideas

6.1 Theorem 2

6.2 Theorem 3

6.3 Theorems 4 and 5

Funding Acknowledgements

References

Appendix A Proof of Thm. 1

Lemma 2.

Proof.

Appendix B Proof of Thm. 2

Definition 1.

4 Rank minimization in deep networks with small $ℓ_{2}$ norm