Iterative Neural Networks with Bounded Weights

I Introduction

Artificial neural networks are becoming indispensible tools in a variety of spheres of human activity and society in general. It is therefore of utmost importance to understand the way they process the supplied data. We build on the recent works [1, 3]

which established fundamental properties of artificial neural networks in a general Hilbert space settings, such as existence of the fixed points sets, convergence analysis, and Lipschitz continuity. In this paper, we focus on exploiting a natural assumption that weights of the network are bounded. This assumption allows us to provide a simple proof of the fact that recurrent neural network possesses exactly one fixed cycle, obtained in the iteration limit of the network and originating from its unique fixed point. We also provide a bound on the norm of this fixed point in terms of norm of weights and biases. Finally, we discuss why a feed-forward neural network model used in this paper is not able to accomodate Hopfield networks under our assumption, which provides a motivation to derive and analyze a more generic neural network models.

Ii Preliminaries

Ii-a Functions in Hilbert spaces

Let $H$ and $G$ be arbitrary Hilbert spaces. For a function $f : H \to H$ , by $Fix (f) := {x ∣ f (x) = x} \subset H$ we denote fixed point set of $f$ . For a set-valued function (with non-empty values) $g : H ⊸ G$ , $Zer (f) := {x ∣ 0 \in f (x)} \subset H$ stands for the set of zeros of function $f$ . The closure of a subset $X \subset H$ is denoted by $¯ ¯¯¯ ¯ X$ . For a linear operator $A : H \to G$ , by $Ran (A) := {A x ∣ x \in H} \subset G$ and $Ker (A) := {x ∣ A x = 0} \subset H$ we denote its range and kernel, respectively. Finally, by $A^{*} : G \to H$ we denote the adjoint operator of $A$ , and by $I_{H} : H \to H$ we denote identity operator on $H$ . To simplify the notation, we omit the subscript of the identity operator if the space under consideration is clear from the context.

Ii-B Convex analysis

Let $H$ be a Hilbert space. We denote by $Γ_{0} (H)$ the class of lower semi-continuous (l.s.c.) convex functions $ϕ : H \to (- \infty, + \infty]$ , which are proper, i.e. such that

Dom ϕ := {x \in H ∣ ϕ (x) < + \infty} \neq \emptyset .

(1)

For $ϕ \in Γ_{0} (H)$ , the proximal operator ${prox}_{ϕ} : H \to H$ is defined as

{prox}_{ϕ} (x) := {argmin}_{y \in H} (ϕ (y) + 1 / 2 ∥ x - y ∥^{2}) .

(2)

The subdifferential of $ϕ \in Γ_{0} (H)$ is the set-valued operator $\partial ϕ : H ⊸ H$ given by

\partial ϕ (x) := {u \in H ∣ ⟨ y - x, u ⟩ + ϕ (x) \leq ϕ (y) for all y \in H} .

(3)

The sets $\partial ϕ (x)$ are closed and convex [2, Proposition 16.4].

Operator $A : H ⊸ H$ is called monotone, provided for each $(x, u), (x^{'}, u^{'}) \in gra (A) := {(y, v) ∣ v \in A y}$ we have $⟨ x - x^{'}, u - u^{'} ⟩ \geq 0$ . Monotone operator $A$ is maximally monotone if there is no extension of $A$ to larger monotone operator $B : H ⊸ H$ in the sense that $gra (A) ⊊ gra (B)$ . An example of maximally monotone operator is the subdifferential $\partial ϕ .$

Denote by $A (R)$ the set of functions from $R$ to $R$ , which are increasing, $1$ -Lipschitz, and take value $0$ for argument $0$ . This set of functions can be characterized using proximal operators as follows:

Proposition 1.

[1, Proposition 2.3] Let $ρ : R \to R$ . Then $ρ \in A (R)$ if and only if there exists $ϕ \in Γ_{0} (R)$ , which has $0$ as its minimizer and $ρ = prox ϕ$ .

This fact allows the following definition:

Definition 1.

[1, Definition 2.20] Let $H$ be real Hilbert space and let $σ : H \to H$ . Then $σ$ belongs to $A (H)$ if there exists a function $ϕ \in Γ_{0} (H)$ such that it has minimium at $0$ and $σ = prox ϕ$ .

Iii Setting

Let $n \in N$ , and let $(H_{i})_{0 \leq i \leq n}$ be real Hilbert spaces. Let $H := H_{1} \otimes \dots \otimes H_{n - 1} \otimes H_{n}$ , i.e. if $x \in H$ , then $x = (x_{1}, \dots, x_{n})$ , $x_{i} \in H_{i}$ , $i = 1, \dots, n$ , and if $y \in H$ , then $⟨ x, y ⟩_{H} := \sum_{i = 1}^{n} ⟨ x_{i}, y_{i} ⟩_{H_{i}}$ . For simplicity we will write $⟨ \cdot, \cdot ⟩$ instead of $⟨ x, y ⟩_{H}$ .

Assumption 1.

Let $σ_{i} \in A (H_{i})$ , i.e. $σ_{i} = prox ϕ_{i}$ , for a certain $ϕ_{i} \in Γ_{0} (H_{i})$ with $ϕ_{i} (0) = {inf}_{x \in H_{i}} ϕ_{i} (x)$ .

Assumption 2.

Let $W_{i} : H_{i - 1} \to H_{i}$ , $i = 1, \dots, n$ be bounded linear operators and $b_{i} \in H_{i}$ , $i = 1, \dots, n$ . Moreover, let us define $g_{i} : H_{i - 1} \to H_{i}$ by the formula

g_{i} (x) := σ_{i} (W_{i} x + b_{i}), for x \in H_{i - 1}, i = 1, \dots, n .

(4)

Definition 2.

An $n$ -layer feed-forward neural network defined on $(H_{i})_{0 \leq i \leq n}$ is the composition

g_{n} \circ \dots \circ g_{1} .

(5)

In the theory of neural networks, the functions $σ_{i}$ are called activation operators, operators $W_{i}$ are called weight operators and elements $b_{i}$ are called bias parameters.

Remark 1.

We note that Assumption 2 implies that the neural network is already trained, by which we understand that there exists an optimal setting of weight operators $W_{i}$ and bias parameters $b_{i}$ which fits the input and output of the network to a certain set of training data. In the case of recurrent neural networks, we have a situation where the input and output of the network are in the same space $H_{0}$ , so it is natural to ask about the existence of periodic points and about the shape of set of these points.

Thus, from now on we assume that $H_{n} = H_{0}$ and denote

G := Fix (g_{n} \circ \dots \circ g_{1}) \subset H_{0},

(6)

and

F := {(x_{1}, \dots, x_{n}) \in H ∣ x_{1} = g_{1} (x_{n}), x_{2} = g_{2} (x_{1}), \dots, x_{n} = g_{n} (x_{n - 1})} .

(7)

The set $G = Fix (g)$ consists of fixed (periodic) points of the recurrent neural network

g := g_{n} \circ \dots \circ g_{1} : H_{0} \to H_{1} \to \dots \to H_{n} = H_{0},

(8)

and the set $F$ describes trajectories across layers of the neural network (5) of these fixed points.

Let $\to H := H_{n} \otimes H_{1} \otimes \dots \otimes H_{n - 1}$ and let us introduce operators

	$S$	$: H \to \to H : (x_{1}, \dots, x_{n - 1}, x_{n}) \mapsto (x_{n}, x_{1}, \dots, x_{n - 1}),$
	$W$	$: \to H \to H : (x_{n}, x_{1}, \dots, x_{n - 1}) \mapsto (W_{1} x_{n}, W_{2} x_{1}, \dots, W_{n} x_{n - 1}) .$

Observe that $∥ W \circ S ∥ = ∥ W ∥ := {max}_{i = 1, \dots, n} ∥ W_{i} ∥$ .

Let $ϕ_{i} \in Γ_{0} (H_{i})$ for $i = 1, \dots, n$ and let us define $ϕ := ϕ_{1} \oplus \dots \oplus ϕ_{n} : H \to (- \infty, + \infty]$ by the formula

(9)

Moreover, let $ψ : H \to (- \infty, + \infty]$ be defined by the formula

ψ (x_{1}, \dots, x_{n}) := n \sum i = 1 (ϕ_{i} (x_{i}) - ⟨ x_{i}, b_{i} ⟩) = ϕ (x) - ⟨ x, b ⟩,

(10)

where we denoted $x := (x_{1}, \dots, x_{n})$ , $b := (b_{1}, \dots, b_{n}) \in H$ . We also note that $ϕ, ψ \in Γ_{0} (H) .$

Fact 1.

[2, Proposition 16.9] Under the above assumptions one has

\partial ϕ (x_{1}, \dots, x_{n}) = \partial ϕ_{1} (x_{1}) \times \dots \times \partial ϕ_{n} (x_{n}) .

(11)

Consequently, $\partial ψ (x) = \partial ϕ (x) - b$ .

Theorem 1.

[1, Part of Proposition 4.3] In terms of the model introduced above, consider the following problem: find ${¯ ¯ ¯ x}_{1} \in H_{1}, \dots, {¯ ¯ ¯ x}_{n} \in H_{n}$ such that

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} b_{1} \in {¯ ¯ ¯ x}_{1} - W_{1} {¯ ¯ ¯ x}_{n} + \partial ϕ_{1} ({¯ ¯ ¯ x}_{1}), b_{2} \in {¯ ¯ ¯ x}_{2} - W_{2} {¯ ¯ ¯ x}_{1} + \partial ϕ_{2} ({¯ ¯ ¯ x}_{2}), ⋮ b_{n} \in {¯ ¯ ¯ x}_{n} - W_{n} {¯ ¯ ¯ x}_{n - 1} + \partial ϕ_{n} ({¯ ¯ ¯ x}_{n}) . \end{matrix}

(12)

The following holds.

The set of solutions of system of inclusions (12) is $F$ .
$F = Zer (I - W \circ S + \partial ψ) = Fix ({prox}_{ψ} \circ W \circ S)$ .
Let us assume that the operator $I - W \circ S$ is monotone. Then the set $F$ is closed and convex. Moreover, $G$ and $F$ are nonempty if any of the following conditions is satisfied:
1. $I - W \circ S + \partial ϕ$ is surjective.
2. $∥ W ∥ \leq 1$ , $Ran (S^{*} - W) = ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ Ran (S^{*} - W)$ and $Ker (S - W^{*}) = {0}$ .

Remark 2.

Note that problem of finding fixed points of the recurrent neural network (5) reduces to a problem of solving a system of equations

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} {¯ ¯ ¯ x}_{1} = g_{1} ({¯ ¯ ¯ x}_{n}) = σ_{1} (W_{1} {¯ ¯ ¯ x}_{n} + b_{1}), {¯ ¯ ¯ x}_{2} = g_{2} ({¯ ¯ ¯ x}_{1}) = σ_{2} (W_{2} {¯ ¯ ¯ x}_{1} + b_{2}), ⋮ {¯ ¯ ¯ x}_{n} = g_{n} ({¯ ¯ ¯ x}_{n - 1}) = σ_{n} (W_{n} {¯ ¯ ¯ x}_{n - 1} + b_{n}) . \end{matrix}

(13)

Under Assumption 1 and using [2, Proposition 16.44], which states that

prox ϕ_{i} = (I + \partial ϕ_{i})^{- 1},

(14)

the above system (13) can be rewritten to the system of inclusions (12). That is why that inclusion is crucial for our further considerations.

Iv Results

The following proposition shows that, under a single mild assumption, the neural network $g$ converges to its unique fixed point.

Proposition 2.

Let

n \prod i = 1 ∥ W_{i} ∥ < 1.

(15)

Then, $F$ is a singleton. Denote the unique element of $F$ as $x^{F}$ . Furthermore, let $x_{0} \in H_{0}$ and $x_{k} = g^{k} (x_{0}) .$ Then ${lim}_{k \to \infty} x_{k} = x_{n}^{F}$ , where $x_{n}^{F} \in H_{n}$ denotes the $n$ th coordinate of $x^{F}$ , and $G = {x_{n}^{F}} .$

Proof.

By Assumption 1, from [2, Proposition 12.28], the activation operators are firmly nonexpansive. Thus, from [3, Proposition 3.3], we obtain in particular that the $n$ -layered neural network $g$ in (8) is Lipschitz continuous with constant $θ_{n} := \prod_{i = 1}^{n} ∥ W_{i} ∥ < 1.$ Therefore, from the Banach Fixed Point Theorem it admits a unique fixed point, thus $G$ is a singleton. Denote this unique fixed point of $g$ as $x_{n}^{F} := {lim}_{k \to \infty} x_{k} .$ Then, the fact that $F$ is a singleton follows immediately from the definition of $F$ in (7). We also note en passant that $F = {(g_{1} (x_{n}^{F}), g_{2} (g_{1} (x_{n}^{F})), \dots, x_{n}^{F})} .$ ∎

Remark 3.

The above proposition extends the results of Theorem 1 proved in [1] to the case when condition (15) holds. It is remarkable that, in such a case, no other conditions are required to ensure that $F$ is not only nonempty, closed and convex, but actually a singleton. Moreover, note that the sequence of interations convergences in the strong topology to the unique fixed point.

Corollary 1.

In particular, if

∥ W ∥ < 1,

(16)

then $F$ is a singleton.

The next proposition provides a bound on the norm of the unique element of $F .$

Proposition 3.

Let $W$ satisfies condition (16). Then, the unique element $x^{F} \in F$ (cf. Proposition 2) is such that

∥ x^{F} ∥ \leq \frac{∥ b ∥}{1 - ∥ W ∥} .

(17)

In particular, if no bias terms are used in neural network (5), i.e., $b = 0$ , then $F = {0} .$

Proof.

We note first that $x^{F} = (x_{1}^{F}, \dots, x_{n - 1}^{F}, x_{n}^{F})$ is the unique solution of system (12), if condition (15) is assumed, and in such a case

b + (W \circ S) x^{F} \in x^{F} + \partial ϕ (x^{F}) = (I + \partial ϕ) (x^{F}) .

(18)

Thus, from [2, Proposition 16.44] one has

x^{F} = {prox}_{ϕ} (W \circ S x^{F} + b) .

(19)

Hence, using the fact that ${prox}_{ϕ} \in A (H)$ (cf. Definition 1), from [1, Proposition 2.21] one has in particular that for all $x \in H$

∥ {prox}_{ϕ} (x) ∥ \leq ∥ x ∥ .

(20)

For $x^{F}$ , this implies that

	$∥ x^{F} ∥ \leq$	$∥ W \circ S x^{F} + b ∥ \leq ∥ W \circ S x^{F} ∥ + ∥ b ∥ \leq$		(21)
		$∥ W \circ S ∥ \cdot ∥ x^{F} ∥ + ∥ b ∥ = ∥ W ∥ \cdot ∥ x^{F} ∥ + ∥ b ∥,$		(22)

where we have used the fact that $∥ W \circ S ∥ = ∥ W ∥ .$ The inequality (17) follows. If $b = 0$ , then $x^{F} = 0$ , and from Proposition 2 one concludes that in such a case $F = {0} .$ ∎

The fact that neural network $g$ converges to a single fixed point may not be desirable in certain applications. The following remark demonstrates that in such a case, a more general network model must be considered.

Remark 4.

Consider Hopfield neural network model given as follows

x^{'} (t) = - D x (t) + W σ (x (t)) + b,

(23)

where $x (t) \in H$

is the state vector,

D := diag (d_{1} I_{H_{1}}, d_{2} I_{H_{2}}, \dots, d_{n} I_{H_{n}})

with

d_{i} > 0

is block diagonal matrix of self-inhibition of neurons and

σ

, given by

σ (x_{1}, \dots, x_{n}) := (σ_{1} (x_{1}), \dots, σ_{n} (x_{n}))

x_{i} \in H_{i}

i = 1, \dots, n

, is continuous activation function of the neural network.

Any equlibrium point $x$ of the above network satisfies

0 = - D x + W σ (x) + b .

(24)

Therefore,

x = D^{- 1} (W σ (x) + b) .

(25)

Denote $z = σ (x)$ . Since, by [2, Proposition 16.44], $x \in σ^{- 1} (z) = z + \partial ϕ (z)$ , then

D^{- 1} (W z + b) \in z + \partial ϕ (z) .

(26)

Thus,

D^{- 1} b \in z - D^{- 1} W z + \partial ϕ (z),

(27)

which is of the form (12). In particular, under appropriate assumptions on operators $D$ and $W$ , one can achieve that $∥ D^{- 1} W ∥ < 1$ , which according to Proposition 2 leads to a situation where we have only one fixed point of network (23). Hence, our (and the one described in [1]) model may not be adequate in this case, as Hopfield network learning relies on memorizing many distinct fixed points of the network.

References

[1] P. L. Combettes and J.-C. Pesquet, Deep Neural Network Structures Solving Variational Inequalities, arXiv preprint, arXiv:1808.07526, 2019.
[2] H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, Second Edition. New York: Springer, 2017.
[3] P. L. Combettes and J.-C. Pesquet, Lipschitz Certificates for Neural Network Structures Driven by Averaged Activation Operators. arXiv preprint arXiv:1903.01014, 2019.

Iterative Neural Networks with Bounded Weights

Authors

TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNT

On the Relation of Impulse Propagation to Synaptic Strength

Convergent Graph Solvers

Fixed points of monotonic and (weakly) scalable neural networks

Fixed Point Networks: Implicit Depth Models with Jacobian-Free Backprop

Connections between spectral properties of asymptotic mappings and solutions to wireless network problems

Feature extraction of machine learning and phase transition point of Ising model

I Introduction

Ii Preliminaries

Ii-a Functions in Hilbert spaces

Ii-B Convex analysis

Proposition 1.

Definition 1.

Iii Setting

Assumption 1.

Assumption 2.

Definition 2.

Remark 1.

Fact 1.

Theorem 1.

Remark 2.

Iv Results

Proposition 2.

Proof.

Remark 3.

Corollary 1.

Proposition 3.

Proof.

Remark 4.

References

Iterative Neural Networks with Bounded Weights

Authors

Related Research

TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNT

On the Relation of Impulse Propagation to Synaptic Strength

Convergent Graph Solvers

Fixed points of monotonic and (weakly) scalable neural networks

Fixed Point Networks: Implicit Depth Models with Jacobian-Free Backprop

Connections between spectral properties of asymptotic mappings and solutions to wireless network problems

Feature extraction of machine learning and phase transition point of Ising model

This week in AI

I Introduction

Ii Preliminaries

Ii-a Functions in Hilbert spaces

Ii-B Convex analysis

Proposition 1.

Definition 1.

Iii Setting

Assumption 1.

Assumption 2.

Definition 2.

Remark 1.

Fact 1.

Theorem 1.

Remark 2.

Iv Results

Proposition 2.

Proof.

Remark 3.

Corollary 1.

Proposition 3.

Proof.

Remark 4.

References