Artificial neural networks are becoming indispensible tools in a variety of spheres of human activity and society in general. It is therefore of utmost importance to understand the way they process the supplied data. We build on the recent works [1, 3]
which established fundamental properties of artificial neural networks in a general Hilbert space settings, such as existence of the fixed points sets, convergence analysis, and Lipschitz continuity. In this paper, we focus on exploiting a natural assumption that weights of the network are bounded. This assumption allows us to provide a simple proof of the fact that recurrent neural network possesses exactly one fixed cycle, obtained in the iteration limit of the network and originating from its unique fixed point. We also provide a bound on the norm of this fixed point in terms of norm of weights and biases. Finally, we discuss why a feed-forward neural network model used in this paper is not able to accomodate Hopfield networks under our assumption, which provides a motivation to derive and analyze a more generic neural network models.
Ii-a Functions in Hilbert spaces
Let and be arbitrary Hilbert spaces. For a function , by we denote fixed point set of . For a set-valued function (with non-empty values) , stands for the set of zeros of function . The closure of a subset is denoted by . For a linear operator , by and we denote its range and kernel, respectively. Finally, by we denote the adjoint operator of , and by we denote identity operator on . To simplify the notation, we omit the subscript of the identity operator if the space under consideration is clear from the context.
Ii-B Convex analysis
Let be a Hilbert space. We denote by the class of lower semi-continuous (l.s.c.) convex functions , which are proper, i.e. such that
For , the proximal operator is defined as
The subdifferential of is the set-valued operator given by
The sets are closed and convex [2, Proposition 16.4].
Operator is called monotone, provided for each we have . Monotone operator is maximally monotone if there is no extension of to larger monotone operator in the sense that . An example of maximally monotone operator is the subdifferential
Denote by the set of functions from to , which are increasing, -Lipschitz, and take value for argument . This set of functions can be characterized using proximal operators as follows:
[1, Proposition 2.3] Let . Then if and only if there exists , which has as its minimizer and .
This fact allows the following definition:
[1, Definition 2.20] Let be real Hilbert space and let . Then belongs to if there exists a function such that it has minimium at and .
Let , and let be real Hilbert spaces. Let , i.e. if , then , , , and if , then . For simplicity we will write instead of .
Let , i.e. , for a certain with .
Let , be bounded linear operators and , . Moreover, let us define by the formula
An -layer feed-forward neural network defined on is the composition
In the theory of neural networks, the functions are called activation operators, operators are called weight operators and elements are called bias parameters.
We note that Assumption 2 implies that the neural network is already trained, by which we understand that there exists an optimal setting of weight operators and bias parameters which fits the input and output of the network to a certain set of training data. In the case of recurrent neural networks, we have a situation where the input and output of the network are in the same space , so it is natural to ask about the existence of periodic points and about the shape of set of these points.
Thus, from now on we assume that and denote
The set consists of fixed (periodic) points of the recurrent neural network
and the set describes trajectories across layers of the neural network (5) of these fixed points.
Let and let us introduce operators
Observe that .
Let for and let us define by the formula
Moreover, let be defined by the formula
where we denoted , . We also note that
[2, Proposition 16.9] Under the above assumptions one has
[1, Part of Proposition 4.3] In terms of the model introduced above, consider the following problem: find such that
The following holds.
The set of solutions of system of inclusions (12) is .
Let us assume that the operator is monotone. Then the set is closed and convex. Moreover, and are nonempty if any of the following conditions is satisfied:
, and .
Note that problem of finding fixed points of the recurrent neural network (5) reduces to a problem of solving a system of equations
The following proposition shows that, under a single mild assumption, the neural network converges to its unique fixed point.
Then, is a singleton. Denote the unique element of as . Furthermore, let and Then , where denotes the th coordinate of , and
By Assumption 1, from [2, Proposition 12.28], the activation operators are firmly nonexpansive. Thus, from [3, Proposition 3.3], we obtain in particular that the -layered neural network in (8) is Lipschitz continuous with constant Therefore, from the Banach Fixed Point Theorem it admits a unique fixed point, thus is a singleton. Denote this unique fixed point of as Then, the fact that is a singleton follows immediately from the definition of in (7). We also note en passant that ∎
The above proposition extends the results of Theorem 1 proved in  to the case when condition (15) holds. It is remarkable that, in such a case, no other conditions are required to ensure that is not only nonempty, closed and convex, but actually a singleton. Moreover, note that the sequence of interations convergences in the strong topology to the unique fixed point.
In particular, if
then is a singleton.
The next proposition provides a bound on the norm of the unique element of
Thus, from [2, Proposition 16.44] one has
For , this implies that
The fact that neural network converges to a single fixed point may not be desirable in certain applications. The following remark demonstrates that in such a case, a more general network model must be considered.
Consider Hopfield neural network model given as follows
Any equlibrium point of the above network satisfies
Denote . Since, by [2, Proposition 16.44], , then
which is of the form (12). In particular, under appropriate assumptions on operators and , one can achieve that , which according to Proposition 2 leads to a situation where we have only one fixed point of network (23). Hence, our (and the one described in ) model may not be adequate in this case, as Hopfield network learning relies on memorizing many distinct fixed points of the network.
-  P. L. Combettes and J.-C. Pesquet, Deep Neural Network Structures Solving Variational Inequalities, arXiv preprint, arXiv:1808.07526, 2019.
-  H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Second Edition. New York: Springer, 2017.
-  P. L. Combettes and J.-C. Pesquet, Lipschitz Certificates for Neural Network Structures Driven by Averaged Activation Operators. arXiv preprint arXiv:1903.01014, 2019.