Optimizing Neural Networks via Koopman Operator Theory

06/03/2020
by   Akshunna S. Dogra, et al.
0

Koopman operator theory, a powerful framework for discovering the underlying dynamics of nonlinear dynamical systems, was recently shown to be intimately connected with neural network training. In this work, we take the first steps in making use of this connection. As Koopman operator theory is a linear theory, a successful implementation of it in evolving network weights and biases offers the promise of accelerated training, especially in the context of deep networks, where optimization is inherently a non-convex problem. We show that Koopman operator theory methods allow for accurate predictions of the weights and biases of a feedforward, fully connected deep network over a non-trivial range of training time. During this time window, we find that our approach is at least 10x faster than gradient descent based methods, in line with the results expected from our complexity analysis. We highlight additional methods by which our results can be expanded to broader classes of networks and larger time intervals, which shall be the focus of future work in this novel intersection between dynamical systems and neural network theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/15/2023

Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passing

Recently, Koopman operator theory has become a powerful tool for develop...
research
05/08/2020

Dynamical Systems Theory and Algorithms for NP-hard Problems

This article surveys the burgeoning area at the intersection of dynamica...
research
12/16/2020

Physical deep learning based on optimal control of dynamical systems

A central topic in recent artificial intelligence technologies is deep l...
research
08/22/2017

Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems

The Koopman operator has recently garnered much attention for its value ...
research
08/25/2020

Theory of Deep Q-Learning: A Dynamical Systems Perspective

Deep Q-Learning is an important algorithm, used to solve sequential deci...
research
04/29/2019

New optimization algorithms for neural network training using operator splitting techniques

In the following paper we present a new type of optimization algorithms ...
research
09/14/2022

Algorithmic (Semi-)Conjugacy via Koopman Operator Theory

Iterative algorithms are of utmost importance in decision and control. W...

Please sign up or login with your details

Forgot password? Click here to reset