Convexifying Sparse Interpolation with Infinitely Wide Neural Networks: An Atomic Norm Approach

07/15/2020
by   Akshay Kumar, et al.
0

This work examines the problem of exact data interpolation via sparse (neuron count), infinitely wide, single hidden layer neural networks with leaky rectified linear unit activations. Using the atomic norm framework of [Chandrasekaran et al., 2012], we derive simple characterizations of the convex hulls of the corresponding atomic sets for this problem under several different constraints on the weights and biases of the network, thus obtaining equivalent convex formulations for these problems. A modest extension of our proposed framework to a binary classification problem is also presented. We explore the efficacy of the resulting formulations experimentally, and compare with networks trained via gradient descent.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/22/2022

Edge convex smooth interpolation curve networks with minimum L_∞-norm of the second derivative

We consider the extremal problem of interpolation of convex scattered da...
research
02/19/2019

Interpolation of scattered data in R^3 using minimum L_p-norm networks, 1<p<∞

We consider the extremal problem of interpolation of scattered data in R...
research
11/08/2021

Optimal convex lifted sparse phase retrieval and PCA with an atomic matrix norm regularizer

We present novel analysis and algorithms for solving sparse phase retrie...
research
02/25/2020

Convex Geometry and Duality of Over-parameterized Neural Networks

We develop a convex analytic framework for ReLU neural networks which el...
research
02/24/2020

Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-Layer Networks

We develop exact representations of two layer neural networks with recti...
research
11/15/2022

REPAIR: REnormalizing Permuted Activations for Interpolation Repair

In this paper we look into the conjecture of Entezari et al.(2021) which...
research
04/23/2014

Forward - Backward Greedy Algorithms for Atomic Norm Regularization

In many signal processing applications, the aim is to reconstruct a sign...

Please sign up or login with your details

Forgot password? Click here to reset