Stochastic Feedforward Neural Networks: Universal Approximation

by   Thomas Merkh, et al.

In this chapter we take a look at the universal approximation question for stochastic feedforward neural networks. In contrast to deterministic networks, which represent mappings from a set of inputs to a set of outputs, stochastic networks represent mappings from a set of inputs to a set of probability distributions over the set of outputs. In particular, even if the sets of inputs and outputs are finite, the class of stochastic mappings in question is not finite. Moreover, while for a deterministic function the values of all output variables can be computed independently of each other given the values of the inputs, in the stochastic setting the values of the output variables may need to be correlated, which requires that their values are computed jointly. A prominent class of stochastic feedforward networks which has played a key role in the resurgence of deep learning are deep belief networks. The representational power of these networks has been studied mainly in the generative setting, as models of probability distributions without an input, or in the discriminative setting for the special case of deterministic mappings. We study the representational power of deep sigmoid belief networks in terms of compositions of linear transformations of probability distributions, Markov kernels, that can be expressed by the layers of the network. We investigate different types of shallow and deep architectures, and the minimal number of layers and units per layer that are sufficient and necessary in order for the network to be able to approximate any given stochastic mapping from the set of inputs to the set of outputs arbitrarily well.


page 1

page 2

page 3

page 4


Universal Approximation of Markov Kernels by Shallow Stochastic Feedforward Networks

We establish upper bounds for the minimal number of hidden units for whi...

Kernels and Submodels of Deep Belief Networks

We study the mixtures of factorizing probability distributions represent...

Geometry and Expressive Power of Conditional Restricted Boltzmann Machines

Conditional restricted Boltzmann machines are undirected stochastic neur...

Universal Approximation Depth and Errors of Narrow Belief Networks with Discrete Units

We generalize recent theoretical work on the minimal number of layers of...

Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions

In recent years, deep learning-based sequence modelings, such as languag...

Synthesis of Stochastic Flow Networks

A stochastic flow network is a directed graph with incoming edges (input...

Selectivity in Probabilistic Causality: Drawing Arrows from Inputs to Stochastic Outputs

Given a set of several inputs into a system (e.g., independent variables...

Please sign up or login with your details

Forgot password? Click here to reset