Port-Hamiltonian Approach to Neural Network Training

09/06/2019
by   Stefano Massaroli, et al.
16

Neural networks are discrete entities: subdivided into discrete layers and parametrized by weights which are iteratively optimized via difference equations. Recent work proposes networks with layer outputs which are no longer quantized but are solutions of an ordinary differential equation (ODE); however, these networks are still optimized via discrete methods (e.g. gradient descent). In this paper, we explore a different direction: namely, we propose a novel framework for learning in which the parameters themselves are solutions of ODEs. By viewing the optimization process as the evolution of a port-Hamiltonian system, we can ensure convergence to a minimum of the objective function. Numerical experiments have been performed to show the validity and effectiveness of the proposed methods.

READ FULL TEXT

page 1

page 6

page 7

research
03/07/2023

Learning Hamiltonian Systems with Mono-Implicit Runge-Kutta Methods

Numerical integrators could be used to form interpolation conditions whe...
research
06/16/2023

Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers

Recent work has shown that renormalisation group theory is a useful fram...
research
04/24/2022

Numerical methods that preserve a Lyapunov function for Ordinary Differential Equations

The paper studies numerical methods that preserve a Lyapunov function of...
research
05/25/2021

Scaling Properties of Deep Residual Networks

Residual networks (ResNets) have displayed impressive results in pattern...
research
10/05/2022

Optimization-Informed Neural Networks

Solving constrained nonlinear optimization problems (CNLPs) is a longsta...
research
06/09/2012

A Connectionist Network Approach to Find Numerical Solutions of Diophantine Equations

The paper introduces a connectionist network approach to find numerical ...
research
01/31/2022

Neural Network Training with Asymmetric Crosspoint Elements

Analog crossbar arrays comprising programmable nonvolatile resistors are...

Please sign up or login with your details

Forgot password? Click here to reset