# Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs

Deep learning models are often successfully trained using gradient descent, despite the worst case hardness of the underlying non-convex optimization problem. The key question is then under what conditions can one prove that optimization will succeed. Here we provide a strong result of this kind. We consider a neural net with one hidden layer and a convolutional structure with no overlap and a ReLU activation function. For this architecture we show that learning is NP-complete in the general case, but that when the input distribution is Gaussian, gradient descent converges to the global optimum in polynomial time. To the best of our knowledge, this is the first global optimality guarantee of gradient descent on a convolutional neural network with ReLU activations.

## Authors

• 8 publications
• 38 publications
• ### Learning One-hidden-layer ReLU Networks via Gradient Descent

We study the problem of learning one-hidden-layer neural networks with R...
06/20/2018 ∙ by Xiao Zhang, et al. ∙ 2

• ### Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima

We consider the problem of learning a one-hidden-layer neural network wi...
12/03/2017 ∙ by Simon S. Du, et al. ∙ 0

• ### When is a Convolutional Filter Easy To Learn?

We analyze the convergence of (stochastic) gradient descent algorithm fo...
09/18/2017 ∙ by Simon S. Du, et al. ∙ 0

• ### Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent

We prove the first superpolynomial lower bounds for learning one-layer n...
06/22/2020 ∙ by Surbhi Goel, et al. ∙ 0

• ### Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels

In this paper, we consider parameter recovery for non-overlapping convol...
11/08/2017 ∙ by Kai Zhong, et al. ∙ 0

• ### Learning Unitaries by Gradient Descent

We study the hardness of learning unitary transformations by performing ...
01/31/2020 ∙ by Bobak Toussi Kiani, et al. ∙ 0

• ### Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods

The optimization problem behind neural networks is highly non-convex. Tr...
10/28/2016 ∙ by Antoine Gautier, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Deep neural networks have achieved state-of-the-art performance on many machine learning tasks in areas such as natural language processing

(Wu et al., 2016)(Krizhevsky et al., 2012) and speech recognition (Hinton et al., 2012)

. Training of such networks is often successfully performed by minimizing a high-dimensional non-convex objective function, using simple first-order methods such as stochastic gradient descent.

Nonetheless, the success of deep learning from an optimization perspective is poorly understood theoretically. Current results are mostly pessimistic, suggesting that even training a 3-node neural network is NP-hard (Blum & Rivest, 1993)

, and that the objective function of a single neuron can admit exponentially many local minima

(Auer et al., 1996; Safran & Shamir, 2016)

. There have been recent attempts to bridge this gap between theory and practice. Several works focus on the geometric properties of loss functions that neural networks attempt to minimize. For some simplified architectures, such as linear activations, it can be shown that there are no bad local minima

(Kawaguchi, 2016). Extension of these results to the non-linear case currently requires very strong independence assumptions (Kawaguchi, 2016).

Since gradient descent is the main “work-horse” of deep learning it is of key interest to understand its convergence properties. However, there are no results showing that gradient descent is globally optimal for non-linear models, except for the case of many hidden neurons (Andoni et al., 2014) and non-linear activation functions that are not widely used in practice (Zhang et al., 2017).111See more related work in Section 2. Here we provide the first such result for a neural architecture that has two very common components: namely a ReLU activation function and a convolution layer.

The architecture considered in the current paper is shown in Figure 1. We refer to these models as no-overlap networks. A no-overlap network can be viewed as a simple convolution layer with non overlapping filters, followed by a ReLU activation function, and then average pooling. Formally, let denote the filter coefficient, and assume the input is in . Define and assume for simplicity that is integral. Partition into non-overlapping parts and denote the part. Finally, define to be the ReLU activation function, namely . Then the output of the network in Figure 1 is given by:

 f(x;w)=1k∑iσ(w⋅x[i]) (1)

We note that such architectures have been used in several works (Lin et al., 2013; Milletari et al., 2016), but we view them as important firstly because they capture key properties of general convolutional networks.

We address the realizable case, where training data is generated from a function as in Eq. 1

with weight vector

. Training data is then generated by sampling training points from a distribution , and assigning them labels using . The learning problem is then to find a that minimizes the squared loss. In other words, solve the optimization problem:

 minw1n∑i(f(xi;w)−yi)2 (2)

In the limit , this is equivalent to minimizing the population risk:

 ℓ(w)=Ex∼D[(f(x;w)−f(x;w∗))2] (3)

Like several recent works (Hardt et al., 2016; Hardt & Ma, 2016) we focus on minimizing the population risk, leaving the finite sample case to future work. We believe the population risk captures the key characteristics of the problem, since the large data regime is the one of interest.

Our key results are as follows:

• Worst Case Hardness: Despite the simplicity of No-Overlap Networks, we show that learning them is in fact hard if is unconstrained. Specifically, in Section 4, we show that learning No-Overlap Networks is NP complete via a reduction from a variant of the set splitting problem.

• Distribution Dependent Tractability: When corresponds to independent Gaussian variables with , we show in Section 5 that No-Overlap Networks can be learned in polynomial time using gradient descent.

The above two results nicely demonstrate the gap between worst-case intractability and tractability under assumptions on the data. We provide an empirical demonstration of this in Section 6 where gradient descent is shown to succeed on the Gaussian case and fail for a different distribution.

To further understand the role of overlap in the network, we consider networks that do have overlap between the filters. In Section 7.1

we show that in this case, even under Gaussian distributed inputs, there will be non-optimal local minima. Thus, gradient descent will no longer be optimal in the overlap case. In Section

7.2 we show empirically that these local optima may be overcome in practice by using gradient descent with multiple restarts.

Taken together, our results are the first to demonstrate distribution dependent optimality of gradient descent for learning a neural architecture with a convolutional like architecture and a ReLU activation function.

## 2 Related Work

Hardness of learning neural networks has been demonstrated for many different settings. For example, (Blum & Rivest, 1993) show that learning a neural network with one hidden layer with a sign activation function is NP-hard in the realizable case. (Livni et al., 2014) extend this to other activation functions and bounded norm optimization. Hardness can also be shown for improper learning under certain cryptographic assumptions (e.g., see Daniely et al., 2014; Klivans, 2008; Livni et al., 2014). Note that these hardness results do not hold for the regression and tied parameter setting that we consider.

Due to the above hardness results, it is clear that the success of deep-learning can only be explained by making additional assumptions about the data generating distribution. The classic algorithm by (Baum, 1990) shows that intersection of halfspaces (i.e., a specific instance of a one hidden layer network) is PAC learnable under any symmetric distribution. This was later extended in (Klivans et al., 2009) to log-concave distributions.

The above works do not consider gradient descent as the optimization method, leaving open the question of which assumptions can lead to global optimality of gradient descent. Such results have been hard to obtain, and we survey some recent ones below. One instance when gradient descent can succeed is when there are enough hidden units such that random initialization of the first layer can lead to zero error even if only the second layer is trained. Such over-specified networks have been considered in (Andoni et al., 2014; Livni et al., 2014) and it was shown that gradient descent can globally learn them in some cases (Andoni et al., 2014). However, the assumption of over-specification is very restrictive and limits generalization. In contrast, we show convergence of gradient descent to a global optimum for any network size and consider convolutional neural networks with shared parameters. Another interesting case is linear dynamical systems, where (Hardt et al., 2016) show that under independence assumptions maximum likelihood is quasi-concave and hence solvable with gradient ascent.

Recent work by (Mei et al., 2016) shows that regression with a single neuron and certain non-linear activation functions, can be learned with gradient descent for sub-Gaussian inputs. We note that their architecture is significantly simpler than ours, in that it uses a single neuron. In fact, their regression problem can also be solved via methods for generalized linear models such as (Kakade et al., 2011).

(Shamir, 2016) recently showed that there is a limit to what distribution dependent results can achieve. Namely, it was shown that for large enough one-hidden layer networks, no distributional assumptions can make gradient descent tractable. Importantly, the construction in (Shamir, 2016) does not use parameter tying and thus is not applicable to the architecture we study here.

Several works have focused on understanding the loss surface of neural network objectives, but without direct algorithmic implications. (Kawaguchi, 2016) show that linear neural networks do not suffer from bad local minima. (Hardt & Ma, 2016) consider objectives of linear residual networks and prove that there are no critical points other than the global optimum. (Soudry & Carmon, 2016) show that in the objective of over-parameterized neural networks with dropout-like noise, all differentiable local minima are global. Other works (Safran & Shamir, 2016; Haeffele & Vidal, 2015) give similar results for over-specified networks. All of these results are purely geometric and do not have direct implications on convergence of optimization algorithms. In a different approach, (Janzamin et al., 2015), suggest alternatives to gradient-based methods for learning neural networks. However, these algorithms are not widely used in practice. Finally, (Choromanska et al., 2015) use spin glass models to argue that, under certain generative modelling and architectural constraints, local minima are likely to have low loss values.

The theory of non-convex optimization is closely related to the theory of neural networks. Recently, there has been substantial progress in proving convergence guarantees of simple first-order methods in various machine learning problems, that don’t correspond to typical neural nets. These include for example matrix completion (Ge et al., 2016)

and tensor decompositions

(Ge et al., 2015).

Finally, recent work by (Zhang et al., 2016) shows that neural nets can perfectly fit random labelings of the data. Understanding this from an optimization perspective is largely an open problem.

## 3 Preliminaries

We use bold-faced letters for vectors and capital letters for matrices. The row of a matrix is denoted by .

In our analysis in Section 5 and Section 7.1 we assume that the input feature

is a vector of IID Gaussian random variables with zero mean and variance one.

222The variance per variable can be arbitrary. We choose one for simplicity. Denote this distribution by . We consider networks with one hidden layer, and hidden units. Our main focus will be on No-Overlap Networks, but we begin with a more general one-hidden-layer neural network with a fully-connected layer parameterized by followed by average pooling. The network output is then:

 f(x;W)=1k∑iσ(wi⋅x) (4)

where is the pointwise ReLU function.

We consider the realizable setting where there exists a true using which the training data is generated. The population risk (see Eq. 3) is then:

 ℓ(W)=EG[(f(x;W)−f(x;W∗))2] , (5)

As we show next, can be considerably simplified. First, define:

 g(u,v)=EG[σ(u⋅x)σ(v⋅x)] (6)

Simple algebra then shows that:

 ℓ(W)=1k2∑i,j[g(wi,wj)−2g(wi,w∗j)+g(w∗i,w∗j)] (7)

The next Lemma from (Cho & Saul, 2009) shows that has a simple form.

###### Lemma 3.1 ((Cho & Saul, 2009), Section 2).

Assume is a vector where the entries are IID Gaussian random variables with mean 0 and variance 1. Given denote by the angle between and . Then:

 g(u,v)=12π∥u∥∥v∥(sinθu,v+(π−θu,v)cosθu,v)

The gradient of with respect to also turns out to have a simple form, as stated in the lemma below. The proof is deferred to the Appendix A.

###### Lemma 3.2.

Let be as defined in Eq. 6. Then is differentiable at all points and

 ∂g(u,v)∂u=12π∥v∥u∥u∥sinθu,v+12π(π−θu,v)v

We conclude by special-casing the results above to No-Overlap Networks. In this case, the entire model is specified by a single filter vector . The rows are mostly zeros, except for the indices which take the values of . Namely, where is a zero vector. The same holds for the vectors with a weight vector . This simplifies the loss considerably, since for all : , and for all : and . Thus the loss for No-Overlap Networks yields (up to additive factors in ):

 l(w)=1k2[γ∥w∥2−2kg(w,w∗)−2β∥w∥∥w∗∥] (8)

where and .

## 4 Learning No-Overlap Networks is NP-Complete

The No-Overlap Networks architecture is a simplified convolutional layer with average pooling. However, as we show here, learning it is still a hard problem. This will motivate our exploration of distribution dependent results in Section 5.

Recall that our focus is on minimizing the squared error in Eq. 3. For this section, we do not make any assumptions on . Thus can be a distribution with uniform mass on training points , recovering the empirical risk in Eq. 2. We know that in Eq. 3 can be minimized by setting and the corresponding squared loss will be zero. However, we of course do not know , and the question is how difficult is it to minimize . In what follows we show that this is hard. Namely, it is an NP-complete problem to find a that comes close to the minimum of , for some constant .

We begin by defining the Set-Splitting-by-k-Sets problem, which is a variant of the classic Set-Splitting problem (Garey & Johnson, 1990). After establishing the hardness of Set-Splitting-by-k-Sets, we will provide a reduction from it to learning No-Overlap Networks.

###### Definition 1.

The Set-Splitting-by-k-Sets decision problem is defined as follows: Given a finite set of elements and a collection of at most subsets of , do there exist disjoint sets such that and for all and , ?

For and without the upper bound on this is known as the Set-Splitting decision problem which is NP-complete (Garey & Johnson, 1990). Next, we show that Set-Splitting-by-k-Sets is NP-complete. The proof is via a reduction from 3SAT and induction, and is provided in Appendix B.

###### Proposition 4.1.

Set-Splitting-by-k-Sets is NP-complete for all .

We next formulate the No-Overlap Networks optimization problem.

###### Definition 2.

The k-Non-Overlap-Opt problem is defined as follows. The input is a distribution over input-output pairs where . If the input is realizable by a no-overlap network with hidden neurons, then the output is a vector such that:

 EDX,Y[(f(x;w)−y))2]<14k5d (9)

Otherwise an arbitrary weight vector is returned.

The above problem returns a that minimizes the population-risk up to accuracy. It is thus easier than minimizing the risk to an arbitrary precision (see Section 5, Theorem 5.2).

We prove the following theorem, which uses some ideas from (Blum & Rivest, 1993), but introduces additional constructions needed for the no overlap case.

###### Theorem 4.2.

For all the , the k-Non-Overlap-Opt problem is NP-complete.

###### Proof.

We will show a reduction from Set-Splitting-by-k-sets to k-Non-Overlap-Opt. Assume a given instance of the Set-Splitting-by-k-sets problem with a set and collection of subsets . Denote and . Let be the all zeros vector. For a vector , define the vector to be the concatenation of vectors , followed by and vectors , and let .

We next define a training set for k-Non-Overlap-Opt. For each element define an input vector , where is the standard basis of . Assign the label to this input. In addition, for each subset define the vector and label . Thus we have inputs in . Let

be a uniform distribution over the training set points (i.e., each point with probability at least

since ).

We will now show that the given instance of Set-Splitting-by-k-sets has a solution (i.e., there exist splitting sets) if and only if k-Non-Overlap-Opt returns a weight vector with low risk. First, assume there exist splitting sets . For each define the vector such that for all , and otherwise. Define a No-Overlap Network with inputs and weight vector . Then for all we have:

 f(xi;w)=∑kl=1σ((aSl)Tei)k=1k=yi (10)

and for all :

 f(xd+j;w)=∑kl=1σ((aSl)T(∑i∈Cjei))k=0=yd+j (11)

where the last equality follows since for all and , . Therefore there exists a for which the error in Eq. 9 is zero and k-Non-Overlap-Opt will return a weight vector with low risk.

Conversely, assume that k-Non-Overlap-Opt returned a with risk less than on above. Denote by , where . We will show that this implies that there exist splitting sets. For all in the training set it holds that:333The LHS is true because for a non-negative random variable , for all , and in our case .

 (f(x′;w)−y′)2kd≤EDX,Y[(f(x;w)−y)2]<14k5d

This implies that for all and ,

 |f(d(ei);w)−1k|<12k2  ,  |f(d(∑i∈Cjei);w)|<12k2 (12)

Define sets for and WLOG assume they are disjoint by arbitrarily assigning points that belong to more than one set, to one of the sets they belong to. We will next show that these are splitting. Namely, it holds that and no subset is a subset of some .

Since for all , it follows that for each there exists such that . Therefore, by the definition of we deduce that . To show the second property, assume by contradiction that for some and , . Then , which implies that , a contradiction. This concludes our proof. ∎

To conclude, we have shown that No-Overlap Networks are hard to learn if one does not make any assumptions about the training data. In fact we have shown that finding a with loss at most is hard. In the next section, we show that certain distributional assumptions make the problem tractable.

## 5 No-Overlap Networks can be Learned for Gaussian Inputs

In this section we assume that the input features are generated via a Gaussian distribution , as in Section 3. We will show that in this case, gradient descent will converge with high probability to the global optimum of (Eq. 8) in polynomial time.

In order to analyze convergence of gradient descent on , we need a characterization of all the critical and non-differentiable points. We show that has a non-differentiable point and a degenerate saddle point.444

A saddle point is degenerate if the Hessian at the point has only non-negative eigenvalues and at least one zero eigenvalue.

Therefore, recent methods for showing global convergence of gradient-based optimizers on non-convex objectives (Lee et al., 2016; Ge et al., 2015) cannot be used in our case, because they assume all saddles are strict 555A saddle point is strict if the Hessian at the point has at least one negative eigenvalue. and the objective function is continuously differentiable everywhere.

The characterization is given in the following lemma. The proof relies on the fact that depends only on , and , and therefore w.l.o.g. it can be assumed that lies on one of the axes. Then by a symmetry argument, in order to prove properties of the gradient and the Hessian, it suffices to calculate partial derivatives with respect to at most three variables.

###### Lemma 5.1.

Let be defined as in Eq. 8. Then the following holds:

1. is differentiable if and only if .

2. For , has three critical points:

1. A local maximum at .

2. A unique global minimum at .

3. A degenerate saddle point at .

For , is not a local maximum and the unique global minimum is the only differentiable critical point.

We next consider a simple gradient descent update rule for minimizing and analyze its convergence. Let denote the step size. Then the update at iteration is simply:

 wt+1=wt−λ∇ℓ(wt) (13)

Our main result, stated formally below, is that the above update is guaranteed to converge to an accurate solution after iterations. We note that the dependence of the convergence rate on is similar to standard results on convergence of gradient descent to stationary points (e.g., see discussion in Allen-Zhu & Hazan, 2016).

###### Theorem 5.2.

Assume .666Assumed for simplicity, otherwise is a constant factor. For any and , there exists 777 can be found explicitly. such that with probability at least , gradient descent initialized randomly from the unit sphere with learning rate will get to a point such that 888 hides a linear factor in . in iterations.

The complete proof is provided in Appendix C. Here we provide a high level overview. In particular, we first explain why gradient descent will stay away from the two bad points mentioned in Lemma 5.1.

First we note that the gradient of at is given by:

 ∇ℓ(wt)=−c1(wt,w∗)wt−c2(wt,w∗)w∗ , (14)

where and are two functions such that and . Thus the gradient is a sum of a vector in the direction of and a vector in the direction of . At iteration we have:

 wt+1=(1+λc1(wt,w∗))wt+λc2(wt,w∗)w∗ (15)

It follows that for the angle between and will decrease in each iteration. Therefore, if has an angle with that is not , we will never converge to the saddle point in Lemma 5.1.

Next, assuming and that the angle between and is at most (which occurs with probability ), it can be shown that the norm of is always bounded away from zero by a constant .999 and hide factors of , , and . The proof is quite technical and follows from the fact that is a local maximum.101010The proof holds even for where is not a local maximum.

The fact that stays away from the problematic points allows us to show that has a Lipschitz continuous gradient on the line between and , with constant .9 By standard optimization analysis (Nesterov, 2004) it follows that after iterations we will have for some . This in turn can be used to show that is -close to . Finally, since , it follows that approximates the global minimum to within accuracy.

Theorem 5.2 implies that gradient descent converges to a point such that in time where is the input dimension.111111Note that the complexity of a gradient descent iteration is polynomial in . The following corollary thus follows.

###### Corollary 5.3.

Gradient descent solves the k-Non-Overlap-Opt problem under the Gaussian assumption on with high probability and in polynomial time.

## 6 Empirical Illustration of Tractability Gap

The results in the previous sections showed that No-Overlap Networks optimization is hard in the general case, but tractable for Gaussian inputs. Here we empirically demonstrate both the easy and hard cases. The training data for the two cases will be generated by using the same but different distributions over .

To generate the “hard” case, we begin with a set splitting problem. In particular, we consider a set with elements and a collection of subsets of , each of size . We choose such that there exists subsets , that split the subsets . We use the reduction in Section 4 to convert this into a No-Overlap Networks optimization problem. This results in a training set of size .

Since we know the that solves the set splitting problem, we can use it to label data from a different distribution. Motivated by Section 5 we use a Gaussian distribution as defined earlier and generate a training set of the same size (namely ) and labels given by the no-overlap network with weight .

For these two learning problems we used AdaGrad (Duchi et al., 2011) to optimize the empirical

risk (plain gradient descent also converges, but AdaGrad requires less tuning of step size). For both datasets we used a random normal initializer and for each we chose the best performing learning rate schedule. The training error for each setting as a function of the number of epochs is shown in Figure

2. It is clear that in the non-Gaussian case, AdaGrad gets trapped at a sub-optimal point, whereas the Gaussian case is solved optimally.121212We note that the value of attained by the non-Gaussian case is quite high, since the zero weight vector in this case has loss of order . In the Gaussian case AdaGrad converged to . Therefore, given the Gaussian dataset we were able to recover the true weight vector , whereas given the data constructed via the reduction we were not, even though both datasets were of the same size. We conclude that these empirical findings are in line with our theoretical results.

## 7 Networks with Overlapping Filters

Thus far we showed that the non-overlapping case becomes tractable under Gaussian inputs. A natural question is then what happens when overlaps are allowed (namely, the stride is smaller than the filter size). Will gradient descent still find a global optimum? Here we show that this is in fact

not the case, and that with probability greater than gradient descent will get stuck in a sub-optimal region. In Section 7.1 we analyze this setting for a two dimensional example and provide bounds on the level of suboptimality. In Section 7.2 we report on an empirical study of optimization for networks with overlapping filters. Our results suggest that by restarting gradient descent a constant number of times, it will converge to the global minimum with high probability. Complete proofs of the results are provided in Appendix D.

### 7.1 Suboptimality of Gradient Descent for R2

We consider an instance where there are neurons and matrices correspond to an overlapping filter of size with stride , i.e., for all , where , is a vector of parameters and , . Define the following vectors , , , and denote by the angle between two vectors and .

One might wonder why the analysis of the overlapping case should be any different than the non-overlapping case. However, even for a filter of size two, as above, the loss function and consequently the gradient, are more complex in the overlapping case. Indeed, the loss function in this case is given by:

 ℓ(w)=α(∥w∥2+∥w∗∥2)−βg(w,w∗)+(β−2)(g(wr,wl)−g(wl,w∗r)−g(wr,w∗l)+g(w∗r,w∗l))−γ∥w∥∥w∗∥ (16)

where , and .

Compared to the objective in Eq. 8 which depends only on , and , we see that the objective in Eq. 16 has new terms such as which has a more complicated dependence on the weight vectors and . This does not only have implications on the analysis, but also on the geometric properties of the loss function and the dynamics of gradient descent. In particular, in Figure 3 we see that the objective has a large sub-optimal region which is not the case when the filters are non-overlapping.

As in the previous section we consider gradient descent updates as in Eq. 13. The following Proposition shows that if is initialized in the interior of the fourth quadrant of , then it will stay there for all remaining iterations. The proof is a straightforward inspection of the components of the gradient, and is provided in the supplementary.

###### Proposition 7.1.

For any , if is in the interior of the fourth quadrant of then so is .

Note that in our example the global optimum is in the second quadrant (it’s easy to show that it is also unique). Hence, if initialized at the fourth quadrant, gradient descent will remain in a sub-optimal region. The sub-optimality can be clearly seen in Figure 3. In the proposition below we formalize this observation by giving a tight lower bound on the values of for in the fourth quadrant. Specifically, we show that the sub-optimality scales with . The proof idea is to express all angles between all the vectors that appear in Eq. 16 via a single angle parameter between in the fourth quadrant and the positive -axis. Then it is possible to prove the relatively simpler one dimensional inequality that depends on .

###### Proposition 7.2.

Let , then for all in the fourth quadrant and this lower bound is attained by .

The above two propositions result in the following characterization of the sub-optimality of gradient descent for and overlapping filters.

###### Theorem 7.3.

Define as in Proposition 7.2. Then with probability , a randomly initialized gradient descent with learning rate will get stuck in a sub-optimal region, where each point in this region has loss at least and this bound is tight.

### 7.2 Empirical study of Gradient Descent for m>2

In Section 7.1 we showed that already for , networks with and filter overlaps exhibit more complex behavior than those without overlap. This leaves open the question of what happens in the general case under the Gaussian assumption, for various values of and overlaps. We leave the theoretical analysis of this question to future work, but here report on empirical findings that hint at what the solution should look like.

We experimented with a range of and overlap values (see Appendix E for details of the experimental setup). For each value of , and overlap we sampled values of from various uniform input distributions with different supports and several pre-defined deterministic values. This resulted in more than 1200 different sampled . For each such we ran gradient descent multiple times, each initialized randomly from a different

. Using the results from these runs, we could estimate the probability of sampling a

that would converge to the unique global minimum. Viewed differently, this is the probability mass of the basin of attraction of the global optimum. We note that the uniqueness of the global minimum follows easily from equating the population risk (Eq. 3) to 0 and the full proof is deferred to Appendix F.

Our results are that across all values of , overlap and , the probability mass of the basin of attraction is at least . The practical implication is that multiple restarts of gradient descent (in this case a few dozen) will find the global optimum with high probability. We leave formal analysis of this intriguing fact for future work.

## 8 Discussion

The key theoretical question in deep learning is why it succeeds in finding good models despite the non-convexity of the training loss. It is clear that an answer must characterize specific settings where deep learning provably works. Despite considerable recent effort, such a case has not been shown. Here we provide the first analysis of a non-linear architecture where gradient descent is globally optimal, for a certain input distribution, namely Gaussian. Thus our specific characterization is both in terms of architecture (no-overlap networks, single hidden layer, and average pooling) and input distribution. We show that learning in no-overlap architectures is hard, so that some input distribution restriction is necessary for tractability. Note however, that it is certainly possible that other, non-Gaussian, distributions also result in tractability. Some candidates would be sub-Gaussian and log-concave distributions.

Our derivation addressed the population risk, which for the Gaussian case can be calculated in closed form. In practice, one minimizes an empirical risk. Our experiments in Section 6 suggest that optimizing the empirical risk in the Gaussian case is tractable. It would be interesting to prove this formally. It is likely that measure concentration results can be used to get similar results to those we had for the population risk (e.g., see Mei et al., 2016; Xu et al., 2016, for use of such tools).

Convolution layers are among the basic building block of neural networks. Our work is among the first to analyze optimization for these. The architecture we study is similar in structure to convolutional networks, in the sense of using parameter tying and pooling. However, most standard convolutional layers have overlap and use max pooling. In Section

7 we provide initial results for the case of overlap, showing there is hope for proving optimality for gradient descent with random restarts. Analyzing max pooling would be very interesting and is left for future work.

Finally, we note that distribution dependent tractability has been shown for intersection of halfspaces (Klivans et al., 2009), which is a non-convolutional architecture. However, these results do not use gradient descent. It would be very interesting to use our techniques to try and understand gradient descent for the population risk in these settings.

## Appendix A Proof of Lemma 3.2

First assume that . Then by straightforward calculation we have

 (17)

Hence,

 ∂g∂u=12π∥v∥u∥u∥sinθu,v+12π(π−θu,v)v (18)

Now we assume that is parallel to . We first show that is differentiable in this case. Without loss of generality we can assume that and lie on the axis. This follows since is a function of , and and therefore has a directional derivative in direction at if and only if has a directional derivative in direction at where is a rotation matrix. Hence is differentiable at if and only if is differentiable at . Furthermore, if and are on the axis, then by symmetry the partial derivatives with respect to other axes at are all equal, hence we only need to consider the partial derivative with respect to the and axes.

Let and where . In order to show differentiability, we will prove that has continuous partial derivatives at (by equality (18) the partial derivatives are clearly continuous at points that are not on the axis. Define . Then

 ∂g∂u2(u,v)=limϵ→012π∥uϵ∥∥v∥(sinθuϵ,v+(π−θuϵ,v)cosθuϵ,v)−g(u,v)ϵ

By L’hopital’s rule and the calculation of equality (18) we get

 ∂g∂u2(u,v)=limϵ→012π∥v∥ϵ∥uϵ∥sinθϵ=0

Furthermore, by equality (18) we see that since .

For a fixed equal to or , is the same as . Hence,

 ∂g∂u1(u,v)=12π∥v∥(sinθu,v+(π−θu,v)cosθu,v)={12 if u>00 if u<0

and the partial derivative is continuous since

 limu′→u∂g∂u1(u′,v)={12 if u>00 if u<0

Finally, we see that for the case where and are parallel, the values we got for the partial derivatives coincide with equation Eq. 18. This concludes the proof.

## Appendix B Proof of Proposition 4.1

We will prove the claim by induction on . For the base case we will show that Set-Splitting-by-2-Sets is NP-complete. We will prove this via a reduction from a variant of the 3-SAT problem with the restriction of equal number of variables and clauses, which we denote Equal-3SAT. We will first prove that Equal-3SAT is NP-complete.

###### Lemma B.1.

Equal-3SAT is NP-complete.

###### Proof.

This can be shown via a reduction from 3SAT. Given a formula with variables and clauses we can increase by by adding a new clause of the form () for new variables and . Furthermore, we can decrease by by adding two new identical clauses of the form () for a new variable . In each case the formula with the new clause(s) is satisfiable if and only if is. Therefore given a formula we can construct a new formula with equal number of variables and clauses such that is satisfiable if and only if is. ∎

We will now give a reduction from Equal-3SAT to Set-Splitting-by-2-Sets.

###### Lemma B.2.

Set-Splitting-by-2-Sets is NP-complete.

###### Proof.

The following reduction is exactly the reduction from 3SAT to Splitting-Sets and we include it here for completeness. Let be a formula with set of variables and equal number of variables and clauses. We construct the sets and as follows. Define

 S={¯x∣x∈V}∪V∪{n}

where is the negation of variable and is a new variable not in . For each clause with set of variables or negations of variables that appear in the clause (for example, if then ) construct a set . Furthermore, for each variable construct a set . Let be the family of subsets and for all clauses and . Note that which is required by the definition of Set-Splitting-by-2-Sets.

Assume that is satisfiable and let be the satisfying assignment. Define and . Note that . Assume by contradiction that there exists a set such that or . If then is not a set for some clause because . However, by the construction of a variable and its negation cannot be in . Hence is impossible. If then as in the previous claim cannot be a set for a variable . Hence for some clause . However, this implies that , a contradiction.

Conversely, assume there exists splitting sets and and w.l.o.g. . We note that it follows that no variable and its negation are both contained in one of the sets or . Define the following assignment for . For all if let , otherwise let . Note that is a well defined assignment. Assume by contradiction that there is a clause in which is not satisfiable. Since splits it follows that there exists a variable such that it or its negation are in (recall that ). If then and if then since . In both cases is satisfiable, a contradiction. ∎

This proves the base case. We will now prove the induction step by giving a reduction from Set-Splitting-by-k-Sets to Set-Splitting-by-(k+1)-Sets. Given and such that , define and where for all . Note that . Assume that there are that split the sets in . Then if we define , it follows that and are disjoint and split the sets in .

Conversely, assume that split the sets in . Let w.l.o.g. be the set that contains . Then for all we have . It follows that for all , , or equivalently, . Hence, and are disjoint and split the sets in , as desired.

## Appendix C Missing Proofs for Section 5

### c.1 Proof of Lemma 5.1

1. For , the claim follows from Lemma 3.2. As in the proof of Lemma 3.2 we can assume w.l.o.g. that and . Let . It suffices to show that does not exist. Indeed, let then by L’hopital’s rule

 limϵ→0+f(wϵ,w∗)−f(w,w∗)ϵ=limϵ→0+kπ∥w∗∥ϵ|ϵ|sinθwϵ,w∗+(k2−k)∥w∗∥π=kπ+k2−kπ

and

 limϵ→0−f(wϵ,w∗)−f(w,w∗)ϵ=limϵ→0−kπ∥w∗∥ϵ|ϵ|sinθwϵ,w∗−(k2−k)∥w∗∥π=−kπ−k2−kπ

Hence the left and right partial derivatives with respect to variable are not equal, and thus does not exist.

2. We first show that is a local maximum if and only if . Indeed, by considering the loss function as a function of the variable , for any fixed angle we get a quadratic function of the form , where and . Since is a non-negative function for and if and only if , it follows that if and only if and . Therefore if , then for all fixed angles , the minimum of is attained at , which implies that is a local maximum. If and the minimum of is attained at , and thus is not a local maximum in this case.

We will now find the other critical points of . By Lemma 3.2 we get

 (19)

and assume it vanishes.

Denote . If then let for some . It follows that

 k+k2−kπ−k2−kπ1α−kα=0

or equivalently , and thus .

If then and thus . By setting in the loss function, one can see that is a one-dimensional local minimum, whereas by fixing and decreasing , the loss function decreases. It follows that