Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples

01/18/2021
by   Christian Fiedler, et al.
0

In this paper we approach the problem of unique and stable identifiability of generic deep artificial neural networks with pyramidal shape and smooth activation functions from a finite number of input-output samples. More specifically we introduce the so-called entangled weights, which compose weights of successive layers intertwined with suitable diagonal and invertible matrices depending on the activation functions and their shifts. We prove that entangled weights are completely and stably approximated by an efficient and robust algorithm as soon as 𝒪(D^2 × m) nonadaptive input-output samples of the network are collected, where D is the input dimension and m is the number of neurons of the network. Moreover, we empirically observe that the approach applies to networks with up to 𝒪(D × m_L) neurons, where m_L is the number of output neurons at layer L. Provided knowledge of layer assignments of entangled weights and of remaining scaling and shift parameters, which may be further heuristically obtained by least squares, the entangled weights identify the network completely and uniquely. To highlight the relevance of the theoretical result of stable recovery of entangled weights, we present numerical experiments, which demonstrate that multilayered networks with generic weights can be robustly identified and therefore uniformly approximated by the presented algorithmic pipeline. In contrast backpropagation cannot generalize stably very well in this setting, being always limited by relatively large uniform error. In terms of practical impact, our study shows that we can relate input-output information uniquely and stably to network parameters, providing a form of explainability. Moreover, our method paves the way for compression of overparametrized networks and for the training of minimal complexity networks.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/30/2019

Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

We address the structure identification and the uniform approximation of...
09/21/2021

Neural networks with trainable matrix activation functions

The training process of neural networks usually optimize weights and bia...
10/28/2020

Estimating Multiplicative Relations in Neural Networks

Universal approximation theorem suggests that a shallow neural network c...
12/10/2019

Almost Uniform Sampling From Neural Networks

Given a length n sample from R^d and a neural network with a fixed archi...
02/22/2021

Elementary superexpressive activations

We call a finite family of activation functions superexpressive if any m...
10/13/2021

Two-argument activation functions learn soft XOR operations like cortical neurons

Neurons in the brain are complex machines with distinct functional compa...
12/14/2020

System identification of biophysical neuronal models

After sixty years of quantitative biophysical modeling of neurons, the i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.