Neural network identifiability for a family of sigmoidal nonlinearities

06/11/2019
by   Verner Vlačić, et al.
0

This paper addresses the following question of neural network identifiability: Does the input-output map realized by a feed-forward neural network with respect to a given nonlinearity uniquely specify the network architecture, weights, and biases? Existing literature on the subject Sussman 1992, Albertini, Sontag et.al 1993, Fefferman 1994 suggests that the answer should be yes, up to certain symmetries induced by the nonlinearity, and provided the networks under consideration satisfy certain "genericity conditions". The results in Sussman 1992 and Albertini, Sontag et.al 1993 apply to networks with a single hidden layer and in Fefferman 1994 the networks need to be fully connected. In an effort to answer the identifiability question in greater generality, we derive necessary genericity conditions for the identifiability of neural networks of arbitrary depth and connectivity with an arbitrary nonlinearity. Moreover, we construct a family of nonlinearities for which these genericity conditions are minimal, i.e., both necessary and sufficient. This family is large enough to approximate many commonly encountered nonlinearities to arbitrary precision in the uniform norm.

READ FULL TEXT
research
06/21/2020

Affine Symmetries and Neural Network Identifiability

We address the following question of neural network identifiability: Sup...
research
02/27/2015

Norm-Based Capacity Control in Neural Networks

We investigate the capacity, convexity and characterization of a general...
research
11/26/2022

Why Neural Networks Work

We argue that many properties of fully-connected feedforward neural netw...
research
10/28/2019

Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes

Wide neural networks with random weights and biases are Gaussian process...
research
06/19/2022

Artificial intelligence system based on multi-value classification of fully connected neural network for construction management

This study is devoted to solving the problem to determine the profession...
research
09/21/2018

Local Computation Algorithms for the Lovász Local Lemma

We consider the task of designing Local Computation Algorithms (LCA) for...
research
05/17/2023

On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations

Recent research in neural networks and machine learning suggests that us...

Please sign up or login with your details

Forgot password? Click here to reset