A unified framework for Hamiltonian deep neural networks

04/27/2021
by   Clara L. Galimberti, et al.
0

Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/exploding gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The time-invariant version of the corresponding Hamiltonian models enjoys marginal stability, a property that, as shown in previous works and for specific DNNs architectures, can mitigate convergence to zero or divergence of gradients. In the present paper, we formally study this feature by deriving and analysing the backward gradient dynamics in continuous time. The proposed Hamiltonian framework, besides encompassing existing networks inspired by marginally stable ODEs, allows one to derive new and more expressive architectures. The good performance of the novel DNNs is demonstrated on benchmark classification problems, including digit recognition using the MNIST dataset.

READ FULL TEXT
research
05/27/2021

Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design

Deep Neural Networks (DNNs) training can be difficult due to vanishing a...
research
03/21/2023

Universal Approximation Property of Hamiltonian Deep Neural Networks

This paper investigates the universal approximation capabilities of Hami...
research
05/12/2022

Feedback Gradient Descent: Efficient and Stable Optimization with Orthogonality for DNNs

The optimization with orthogonality has been shown useful in training de...
research
10/14/2019

Variation-aware Binarized Memristive Networks

The quantization of weights to binary states in Deep Neural Networks (DN...
research
10/31/2021

Can we learn gradients by Hamiltonian Neural Networks?

In this work, we propose a meta-learner based on ODE neural networks tha...
research
11/24/2020

ADCME: Learning Spatially-varying Physical Fields using Deep Neural Networks

ADCME is a novel computational framework to solve inverse problems invol...
research
03/22/2022

On Robust Classification using Contractive Hamiltonian Neural ODEs

Deep neural networks can be fragile and sensitive to small input perturb...

Please sign up or login with your details

Forgot password? Click here to reset