Fixed points of monotonic and (weakly) scalable neural networks

06/30/2021
by   Tomasz Piotrowski, et al.
0

We derive conditions for the existence of fixed points of neural networks, an important research objective to understand their behavior in modern applications involving autoencoders and loop unrolling techniques, among others. In particular, we focus on networks with nonnegative inputs and nonnegative network parameters, as often considered in the literature. We show that such networks can be recognized as monotonic and (weakly) scalable functions within the framework of nonlinear Perron-Frobenius theory. This fact enables us to derive conditions for the existence of a nonempty fixed point set of the neural networks, and these conditions are weaker than those obtained recently using arguments in convex analysis, which are typically based on the assumption of nonexpansivity of the activation functions. Furthermore, we prove that the shape of the fixed point set of monotonic and weakly scalable neural networks is often an interval, which degenerates to a point for the case of scalable networks. The chief results of this paper are verified in numerical simulations, where we consider an autoencoder-type network that first compresses angular power spectra in massive MIMO systems, and, second, reconstruct the input spectra from the compressed signal.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/20/2020

Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks

Analysing and computing with Gaussian processes arising from infinitely ...
02/16/2022

Learning a Single Neuron for Non-monotonic Activation Functions

We study the problem of learning a single neuron 𝐱↦σ(𝐰^T𝐱) with gradient...
08/16/2019

Iterative Neural Networks with Bounded Weights

A recent analysis of a model of iterative neural network in Hilbert spac...
10/26/2020

On reaction network implementations of neural networks

This paper is concerned with the utilization of deterministically modele...
10/16/2012

New Advances and Theoretical Insights into EDML

EDML is a recently proposed algorithm for learning MAP parameters in Bay...
08/02/2014

Weakly monotone averaging functions

Monotonicity with respect to all arguments is fundamental to the definit...
11/26/2020

Spectral Analysis and Stability of Deep Neural Dynamics

Our modern history of deep learning follows the arc of famous emergent d...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.