Learn Like The Pro: Norms from Theory to Size Neural Computation

by   Margaret Trautner, et al.

The optimal design of neural networks is a critical problem in many applications. Here, we investigate how dynamical systems with polynomial nonlinearities can inform the design of neural systems that seek to emulate them. We propose a Learnability metric and its associated features to quantify the near-equilibrium behavior of learning dynamics. Equating the Learnability of neural systems with equivalent parameter estimation metric of the reference system establishes bounds on network structure. In this way, norms from theory provide a good first guess for neural structure, which may then further adapt with data. The proposed approach neither requires training nor training data. It reveals exact sizing for a class of neural networks with multiplicative nodes that mimic continuous- or discrete-time polynomial dynamics. It also provides relatively tight lower size bounds for classical feed-forward networks that is consistent with simulated assessments.



There are no comments yet.


page 1

page 2

page 3

page 4


Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical Systems

One of the most influential results in neural network theory is the univ...

Automatic Generation of Bounds for Polynomial Systems with Application to the Lorenz System

This study covers an analytical approach to calculate positively invaria...

On Neural Learnability of Chaotic Dynamics

Neural networks are of interest for prediction and uncertainty quantific...

On the Complexity of the Escape Problem for Linear Dynamical Systems over Compact Semialgebraic Sets

We study the computational complexity of the Escape Problem for discrete...

Nonlinear computations in spiking neural networks through multiplicative synapses

The brain performs many nonlinear computations through intricate spiking...

Monotone Learning with Rectifier Networks

We introduce a new neural network model, together with a tractable and m...

Physics-based polynomial neural networks for one-shot learning of dynamical systems from one or a few samples

This paper discusses an approach for incorporating prior physical knowle...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.