A Simple Proof of the Universality of Invariant/Equivariant Graph Neural Networks

1 Introduction

1.1 Background and Motivation

In this study, we consider a graph regression problem. Let $G$ be the set of simple directed graphs with vertex and edge weights.

Problem 1.

We are given pairs $(G_{1}, y_{1}), (G_{2}, y_{2}), \dots$ of input graphs $G_{i} \in G$ and outcomes $y_{i} \in Y$ . The task is to learn a hypothesis $h : G \to Y$ such that $h (G_{i}) \approx y_{i}$ .

This problem naturally arises in practice. For example, in a toxicity detection problem [23], we want to learn a function $h : G \to Y$ ( $= R$ ) such that $h (G) = 1$ if $G$ has a toxicity and $h (G) = 0$ otherwise. For another example, in a community detection problem [1], we want to learn a function $h : G \to Y$ ( $= R^{(V (G))_{2}}$ ) such that $h (G)_{u v} = 1$ if the vertices $u$ and $v$ are in the same community and $h (G)_{u v} = 0$ otherwise. Here, we denote by $(V)_{k} = {(x_{1}, \dots, x_{k}) \in V^{k} : x_{i} \neq x_{j} (i \neq j)}$ for a finite set $V$ and a positive integer $k$ .³³3We only consider the values of equivariant functions at pairwise different indices. The general case is easily handled by considering each pattern separately, but this complicates the notation. Thus we concentrate on this case.

We are often interested in a hypothesis that merely depends on the topology of the graph (see Section 1.3). Mathematically, this condition is represented by invariance and equivariance. A function $h : G \to Y$ ( $= R$ ) is invariant if for any graph $G = (V (G), E (G))$ and a permutation $σ$ of $V (G)$ , the following equation holds:

h (G^{σ}) = h (G),

(1)

where $G^{σ}$ is the graph whose indices of vertices are permuted by $σ$ , i.e., $G^{σ} = (V (G), E (G)^{σ})$ with $E (G)^{σ} = {(π (u), π (v)) : (u, v) \in E (G)}$ . A function $h : G \to R^{(V (G))_{k}}$ is equivariant if for any graph $G = (V (G), E (G))$ and any permutation $σ$ on $V (G)$ , the following equation holds:

h (G^{σ}) = h (G)^{σ},

(2)

where $h (G)^{σ}$ is defined by the relation $h (G)_{x_{1}, \dots, x_{k}} = h (G)_{π (x_{1}) \dots π (x_{k})}^{π}$ for all $\to x = (x_{1}, \dots, x_{k}) \in (V (G))_{k}$ . The invariance and equivariance mean that the output of the function $h$ is determined up to isomorphism.

One desirable property of a hypothesis space is the universal approximation property (universality for short) [3, 5], i.e., for any continuous function is arbitrary accurately by a function in the hypothesis space. Maron et al. [14] introduced a feed-forward invariant neural network and proved that this model has the universality in the continuous invariant functions. They also characterized all the invariant linear layers [13]. Keriven and Peyré [6] extended the tensorized graph neural network (Figure 0(a)) to represent equivariant functions and proved the universality for the $k = 1$ dimensional output case. Then, they left the universality for the higher-order output case as an open problem.

1.2 Contribution

In this study, we give a simple proof of the universality of tensorized neural networks for both invariant and (higher-order) equivariant cases; the latter solves an open problem posed in [6]. Our proof relies on a result in graph theory (see Section 3 for a comparison of proof techniques in the existing studies).

Let $F$ be the set of simple unweighted graphs, and let $F = (V (F), E (F)) \in F$ . Let $G = (V (G), E (G); W)$ be a weighted graph, where $W : V (G) \times V (G) \to R$ is the weighted adjacency matrix. Let $S (V)$ be the set of all permutations on $V$ . Then, the homomorphism number is defined by

	$h o m (F, W) = \sum π : V (F) \to V (G)$	$\prod i \in V (F) W (π (i), π (i))$
	$\times$	$\prod (i, j) \in E (F) W (π (i), π (j)) .$		(3)

Similarly, for a given $\to x \in (V (G))_{k}$ , the $k$ -labeled homomorphism number is defined by

	${h o m}_{\to x} (F, W) = \sum \begin{matrix} π : V (F) \to V (G) π (i) = x_{i} (i \in [k]) \end{matrix}$	$\prod i \in V (F) W (π (i), π (i))$
	$\times$	$\prod (i, j) \in E (F) W (π (i), π (j)) .$		(4)

By the definition, homomorphism numbers and $k$ -labeled homomorphism numbers are continuous functions in $W$ that is invariant and equivariant, respectively.

Let $G_{0}$ be the set of weighted directed graphs whose edge weights are bounded by one and the number of vertices is $n$ . Let $A$ and $A^{'}$ be the set of functions of the following forms

	$A$	$= ⎧ ⎨ ⎩ W \mapsto finite \sum F \in F a_{F} h o m (F, W + 2 I) : a_{F} \in R ⎫ ⎬ ⎭,$		(5)
	$A^{'}$	$= ⎧ ⎨ ⎩ W \mapsto finite \sum F \in F a_{F} {h o m}_{\to x} (F, W + 2 I) : a_{F} \in R ⎫ ⎬ ⎭,$		(6)

where $I$

denotes the identity matrix of

n \times n

. We prove the following theorems.

Theorem 2.

$A$ is dense in the continuous invariant functions.

Theorem 3.

$A^{'}$ is dense in the continuous equivariant functions.

The translated homomorphism number $W \mapsto h o m (F, W + 2 I)$ and the translated $k$ -labeled homomorphism number $W \mapsto {h o m}_{\to x} (F, W + 2 I)$ are invariant and equivariant linear function on the $| E (F) |$ -fold tensor product, respectively. Therefore, we can implement them in a tensorized neural network. This means that our model is less powerful than the tensorized graph neural network. On the other hand, because our models have the universality, we obtain the following.

Corollary 4.

The invariant (resp., equivariant) tensorized graph neural network has the universality in continuous invariant (resp., equivariant) functions. ∎

1.3 Related Work

In practice, the design of a machine learning model (e.g. neural networks) usually follows some prior knowledge about the target functions since restriction bias helps to simplify the learning process. For instance, in image processing, convolutional neural networks

[10] are designed to be translation invariant [9] or shift-invariant [22]. Therefore, much research has been conducted to address the universality of general invariant neural networks. More recently, graph neural networks predicting labels of vertices [4, 7, 19] have hinted the importance of equivariant models. A natural question in learning theory is whether these aforementioned models are universal? [5, 13]. Here, we discuss related work that answered this question.

Invariant models

The invariant property of a model is usually discussed in the context of learning from points clouds and sets [16, 21, 18], then generalized to symmetries [13] and group actions [2]. While universality analyses for models on sets are well developed, the analysis for graphs is limited [13, 14, 8]. Recently, Maron et al. [14] proved a neural network which is $G$ -invariant is universal. Similarly, Keriven and Peyré [6] obtained the universal result on tensorized graph neural networks by using a more direct application of the Stone-Weierstrass theorem.

Equivariant models

The existence of equivariant models only makes practical sense in some specific cases, for example, learning on graphs’ vertices. Therefore, comparing to the invariant case, there are only a limited number of work addressing equivariance [17, 6, 18]. Consequently, the universality of equivariant graph models have only has recently proven by Keriven and Peyré [6].

2 Proofs

To prove the universality of a class of functions, we use the Stone–Weierstrass theorem:

Theorem 5 (Stone–Weierstrass Theorem [15, Theorem 1.1]).

Let $X$ be a compact Hausdorff space and $C (X, R)$ be the set of continuous functions from $X$ to $R$ , equipped with the $\infty$ -norm. If a subalgebra $A \subseteq C (X, R)$ satisfies the following two conditions:

$A$ separates points, i.e., for any $x \neq y$ , there exists $h \in A$ such that $h (x) \neq h (y)$ , and
There exists $u \in A$ that is bounded away from zero, i.e., ${inf}_{x \in X} | u (x) | > 0$ ,

then $A$ is dense in $C (X, R)$ . ∎

The proofs are devoted to verify the conditions of the Stone–Weierstrass theorem.

2.1 Proof of Theorem 2 (Invariant Case)

We first define the graph space. Let $n$ be the number of vertices in input graphs. Let

G_{0} = {

W \in R^{n \times n} : | W (i, j) | \leq 1, \forall i, j}

(7)

be the set of the weighted adjacency matrices. We denote by $[n] = {1, \dots, n}$ . The $l_{1}$ norm⁴⁴4Because $G_{0}$

is a finite-dimensional vector space, any norms are equivalent; thus, the result in this section is invariant with respect to the choice of the norm.

of the graphs is given by

∥ W ∥_{1} = \sum u, v \in [n] | W (u, v) | .

(8)

Then, we introduce the edit distance $δ_{1}$ by

δ_{1} (W_{1}, W_{2}) = min σ \in S ([n]) ∥ W_{1} - W_{2}^{σ} ∥_{1},

(9)

where $S ([n])$ is the set of all permutations on $[n]$ . The edit distance $δ_{1}$ is nonnegative and satisfies the triangle inequality, i.e., it is a pseudo-metric. We define the graph space by the metric identification as ${~ G}_{0} = G_{0} / \sim$ where $W_{1} \sim W_{2}$ if and only if $δ_{1} (W_{1}, W_{2}) = 0$ . This forms a metric space.

Any invariant function $f : G_{0} \to R$ is identified as a function $f : {~ G}_{0} \to R$ . Our goal is to prove the universality for the set $C ({~ G}_{0}, R)$ of continuous functions from ${~ G}_{0}$ to $R$ . Now, we check the conditions of the Stone–Weierstrass theorem.

First, we check the condition of the space.

Lemma 6.

The graph space $({~ G}_{0}, δ_{1})$ is a compact Hausdorff space.

Proof.

It is Hausdorff because it is a metric space. We show the sequential compactness. Let $W_{1}, W_{2}, \dots$ be an arbitrary sequence in ${~ G}_{0}$ , which is also identified as a sequence in ${~ G}_{0}$ . Because $G_{0}$ is compact in the $l_{1}$ norm, we can choose a convergent subsequence. Such sequence is also a convergent subsequence in ${~ G}_{0}$ . Thus, ${~ G}_{0}$ is compact. ∎

Next, we check the conditions of $A$ .

Lemma 7.

$A$ forms an algebra.

Proof.

Clearly, it is closed under the addition and the scalar multiplication. It is closed under the product because of the following identity:

(10)

where is the disjoint union of $F_{1}$ and $F_{2}$ . ∎

Lemma 8.

$A$ contains an element that is bounded away from zero.

Proof.

Let $\circ$ be the singleton graph. Then, $W \mapsto h o m (\circ, W + 2 I) \geq n$ is bounded away from zero. ∎

To prove the separate point property, we use the following theorem.

Theorem 9 ([11, Lemma 2.4], $k = 0$ case, in our terminology).

Let $W_{1}, W_{2} \in R^{n \times n}$ be matrices with positive diagonal elements. Then, $W_{1}$ and $W_{2}$ are isomorphic if and only if $h o m (F, W_{1}) = h o m (F, W_{2})$ for all simple unweighted graph $F$ . ∎

Lemma 10.

$A$ separates points in ${~ G}_{0}$ .

Proof.

If $W_{1}, W_{2} \in G_{0}$ are non-isomorphic, then $W_{1} + 2 I$ and $W_{2} + 2 I$ are also non-isomorphic. Because $W_{1} + 2 I$ and $W_{2} + 2 I$ satisfy the condition in Theorem 9, there exists $F$ such that $h o m (F, W_{1} + 2 I) \neq h o m (W_{2} + 2 I)$ . This means that $A$ separates points in ${~ G}_{0}$ . ∎

Therefore, we proved Theorem 2.

2.2 Proof of Theorem 3 (Equivariant Case)

We identify an array-valued function $h : G_{0} \to R^{([n])_{k}}$ as a two-argument function $h^{'} : G_{0} \times ([n])_{k} \to R$ . Let $G_{0}^{'} = G_{0} \times ([n])_{k}$ . Then, each element $(W, \to x) \in G_{0}^{'}$ is identified as a $k$ -labeled graph, which is a graph with $k$ distinguished vertices $\to x = (x_{1}, \dots, x_{k}) \in (V (G))_{k}$ .

For a permutation $σ \in S ([n])$ , we define

(W, \to x)^{σ} := (W^{σ}, {\to x}^{σ}),

(11)

where ${\to x}^{σ} = (x_{1}, \dots, x_{k})^{σ} := (x_{1}^{σ}, \dots, x_{k}^{σ})$ . Then, $h$ is equivariant if and only if $h^{'}$ is invariant in the sense that $h^{'} ((W, \to x)^{σ}) = h^{'} ((W, \to x))$ . We say that $(W_{1}, {\to x}_{1})$ and $(W_{2}, {\to x}_{2})$ are isomorphic if $δ_{1}^{'} ((W_{1}, {\to x}_{1}), (W_{2}, {\to x}_{2})) = 0$ .

Now we define the $k$ -labeled edit distance $δ_{1}^{'}$ by

δ_{1}^{'} ((W_{1}, {\to x}_{1}), (W_{2}, {\to x}_{2})) := min \begin{matrix} σ \in S ([n]) {\to x}_{1} = {\to x}_{2}^{σ} \end{matrix} ∥ W_{1} - W_{2}^{σ} ∥_{1} .

(12)

Then, we define the $k$ -labeled graph space by the metric identification as ${~ G}_{0}^{'} = G_{0}^{'} / \sim$ where $(W_{1}, {\to x}_{1}) \sim (W_{2}, {\to x}_{2}$ if and only if $δ_{1} ((W_{1}, {\to x}_{1}), (W_{2}, {\to x}_{2})) = 0$ , i.e., these are isomorphic. This forms a metric space.

Any equivariant function $f : G_{0} \to R^{([n])_{k}}$ is identified as a function $f : {~ G}_{0}^{'} \to R$ . Our goal is to prove the universality for the set $C ({~ G}_{0}^{'}, R)$ of continuous functions from ${~ G}_{0}^{'}$ to $R$ . Now, we check the condition of the Stone–Weierstrass theorem. This part is very similar to that of the invariant case.

First, we check the condition of the space.

Lemma 11.

The $k$ -labeled graph space $({~ G}_{0}^{'}, δ_{1})$ is compact.

Proof.

It is Hausdorff because it is a metric space. We show the sequentially compactness. Let $(W_{1}, {\to x}_{1}), (W_{2}, {\to x}_{2}), \dots$ be an arbitrary sequence in ${~ G}_{0}^{'}$ , which is also identified as a sequence in $G_{0}^{'}$ . Because the number of possibilities of ${\to x}_{i}$ is finite, we can select an (infinite) subsequence that has the same value of ${\to x}_{i}$ . The remaining part is the same as the proof of Lemma 6. ∎

Next, we check the conditions of $A^{'}$ .

Lemma 12.

$A^{'}$ forms an algebra.

Proof.

Clearly, it is closed under the addition and the scalar multiplication. It is closed under the product because of the following identity.

(13)

where $F_{1} \cup^{'} F_{2}$ is the graph obtained from the disjoint union of $F_{1}$ and $F_{2}$ by glueing the labeled vertices. ∎

Lemma 13.

$A^{'}$ contains an element that is bounded away from zero.

Proof.

Let $\circ \dots \circ$ be the graph of $k$ isolated vertices. Then, $W \mapsto {h o m}_{\to x} (\circ \dots \circ, W + 2 I) \geq n - k$ is bounded away from zero. ∎

To prove the separate point property, we use the following theorem.

Theorem 14 ([11, Lemma 2.4] in our terminology).

Let $W_{1}, W_{2} \in R^{n \times n}$ be matrices with positive diagonal elements. Let ${\to x}_{1}, {\to x}_{2} \in ([n])_{k}$ . Then, $(W_{1}, {\to x}_{1})$ and $(W_{2}, {\to x}_{2})$ are isomorphic if and only if ${h o m}_{{\to x}_{1}} (F, W_{1}) = {h o m}_{{\to x}_{2}} (F, W_{2})$ for all simple unweighted graph $F$ . ∎

Lemma 15.

$A^{'}$ separates points in ${~ G}_{0}^{'}$ .

Proof.

It is proved similarly as Lemma 10 by using Theorem 14. ∎

Therefore, we proved Theorem 3.

3 Comparison with Other Proofs

Compare with Keriven and Peyré [6].

Our proofs are similar to that of their proofs. For the invariant case, they used the standard Stone–Weierstrass theorem and verified the separable points property by constructing functions on higher-order tensor space. For the equivariant case, they developed a new Stone–Weierstrass type theorem, and verified the corresponding separate point properties by a similar technique to the invariant case. On the other hand, for both cases, we used the standard Stone–Weierstrass theorem, and verified the separate point property using the property of the homomorphism number. This unified treatment allows us to establish the result on arbitrary higher-order outputs.

One advantage of their method is that it is applicable to hypergraphs. Our method could be applicable for hypergraphs; however, there is a gap because the theory of weighted homomorphism number of hypergraphs is not well established (compared with graphs).

Note that they considered graphs with different but bounded numbers of vertices (i.e., $| V (G_{i}) | \leq n_{max}$ ). However, this is not effective because such space is disconnected, and each connected components corresponds to the graphs having the same number of vertices. If we have to consider a set of graphs with different numbers of vertices, it is promising to consider graphons; see Section 4.

Compare with Maron et al. [13]

They only considered the invariant case. They used the universality of symmetric polynomials by Yarotsky [20]. Then, they approximated the polynomials by a tensorized neural network.

One advantage of their method is that one can bound the order of tensors. Our method can also bound the order of tensors by bounding the size of the subgraphs $F$ [12, Theorem 5.33]; however, this may give a loose bound. On the other hand, our method shows a very restricted form of the linear invariant (or equivariant) layers are sufficient to obtain the universality.

4 Concluding Remarks

In this study (and the existing studies [13, 6]), the number $n$ of vertices in the input graphs are fixed. This is reasonable because the graph space is disconnected and each connected component corresponds to the graphs of the same number of vertices; hence, a continuous function in the graph space of different numbers of vertices is just a collection of continuous functions in each connected component.

If we want to consider graphs of different numbers of vertices, it is promising to consider graphons [12]. Below we explain that all the results obtained in this paper can be extended to graphons.

An (asymmetric) graphon is a measurable function $[0, 1]^{2} \to [0, 1]$ . This is a continuous generalization of the weighted adjacency matrix. The set of graphons is denoted by $W_{0}$ . The cut-norm $∥ \cdot ∥_{□}$ is defined by

∥ W ∥_{□} = sup S, T \subseteq [0, 1] ∣ ∣ ∣ \int_{S \times T} W (x, y) d x d y ∣ ∣ ∣ .

(14)

and the cut-distance $δ_{□}$ is defined by

δ_{□} (W_{1}, W_{2}) = inf σ ∥ W_{1} - W_{2}^{σ} ∥_{□},

(15)

where $σ$ runs over all measure-preserving bijections. The graphon space ${~ W}_{0}$ is defined by the metric identification.

The graphon space contains infinitely many large graphs. However, it is still compact with respect to the cut distance. Note that this does not hold for the edit distance.

Theorem 16 ([12, Theorem 9.23]).

The graphon space $({~ W}_{0}, δ_{□})$ is compact.

In graphons, we use the homomorphism density instead of the homomorphism number. Let $F = (V (F), E (F))$ be a simple unweighted graph. Then, the homomorphism density is given by

t (F, W) = \int \prod (i, j) \in E (F) W (x_{i}, x_{j}) \prod i \in V (F) d x_{i} .

(16)

The set of linear combination of the homomorphism densites also forms an unital algebra. To show the separate points property, we can use the following theorem.

Theorem 17 (Directed version of [12, Corollary 10.34]).

Let $W_{1}, W_{2} \in {~ W}_{0}$ . $W_{1}$ and $W_{2}$ are isomorphic if and only if $t (F, W_{1}) = t (F, W_{2})$ for all simple unweighted graph $F$ . ∎

Therefore, we obtain the following result.

Theorem 18.

The set of finite linear combinations of the homomorphism densities are dense in the continuous invariant graphon functions. ∎

Note that this fact is already proved in [12, Theorem 17.6] for a different context (for symmetric graphons).

The equivariant case is also handled by considering $k$ -labeled graphons. The $k$ -labeled homomorphism density is given by

t_{\to x} (F, W) = \int \prod (i, j) \in E (F) W (x_{i}, x_{j}) \prod i \in V (F) ∖ [k] d x_{i} .

(17)

Then, we obtain the following result.

Theorem 19.

The set of finite linear combinations of the $k$ -labeled homomorphism densities are dense in the continuous equivariant graphon functions. ∎

References

[1] Joan Bruna and X Li. Community detection with graph neural networks. stat, 1050:27, 2017.
[2] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990–2999, 2016.
[3] George Cybenko.
Approximation by superpositions of a sigmoidal function.
Mathematics of Control, Signals and Systems, 2(4):303–314, 1989.
[4] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844–3852, 2016.
[5] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991.
[6] Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. arXiv preprint arXiv:1905.04943, 2019.
[7] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
[8] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. arXiv preprint arXiv:1802.03690, 2018.
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
[10] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989.
[11] László Lovász. The rank of connection matrices and the dimension of graph algebras. European Journal of Combinatorics, 27(6):962–970, 2006.
[12] László Lovász. Large networks and graph limits, volume 60. American Mathematical Soc., 2012.
[13] Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. arXiv preprint arXiv:1812.09902, 2018.
[14] Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. arXiv preprint arXiv:1901.09342, 2019.
[15] L. D. Nel. Theorems of stone-weierstrass type for non-compact spaces. Mathematische Zeitschrift, 104(3):226–230, 1968.
[16] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.
Pointnet: Deep learning on point sets for 3d classification and segmentation.
In
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, July 2017.
[17] Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2892–2901. JMLR. org, 2017.
[18] Akiyoshi Sannai, Yuuki Takai, and Matthieu Cordonnier. Universal approximations of permutation invariant/equivariant functions by deep neural networks. arXiv preprint arXiv:1903.01939, 2019.
[19] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. International Conference on Learning Representations, 2019.
[20] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. arXiv preprint arXiv:1804.10306, 2018.
[21] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in neural information processing systems, pages 3391–3401, 2017.
[22] Richard Zhang. Making convolutional networks shift-invariant again. arXiv preprint arXiv:1904.11486, 2019.
[23] Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):i190–i198, 2017.

A Simple Proof of the Universality of Invariant/Equivariant Graph Neural Networks

Authors

Bridging the Gap Between Spectral and Spatial Domainsin Graph Neural Networks

Universal Invariant and Equivariant Graph Neural Networks

Graph Neural Networks and Reinforcement Learning for Behavior Generation in Semantic Environments

Simplicial Neural Networks

Contrastive Graph Neural Network Explanation

A Note on Over-Smoothing for Graph Neural Networks

Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

This week in AI

1 Introduction

1.1 Background and Motivation

Problem 1.

1.2 Contribution

Theorem 2.

Theorem 3.

Corollary 4.

1.3 Related Work

Invariant models

Equivariant models

2 Proofs

Theorem 5 (Stone–Weierstrass Theorem [15, Theorem 1.1]).

2.1 Proof of Theorem 2 (Invariant Case)

Lemma 6.

Proof.

Lemma 7.

Proof.

Lemma 8.

Proof.

Theorem 9 ([11, Lemma 2.4], k=0 case, in our terminology).

Lemma 10.

Proof.

2.2 Proof of Theorem 3 (Equivariant Case)

Lemma 11.

Proof.

Lemma 12.

Proof.

Lemma 13.

Proof.

Theorem 14 ([11, Lemma 2.4] in our terminology).

Lemma 15.

Proof.

3 Comparison with Other Proofs

Compare with Keriven and Peyré [6].

Compare with Maron et al. [13]

4 Concluding Remarks

Theorem 16 ([12, Theorem 9.23]).

Theorem 17 (Directed version of [12, Corollary 10.34]).

Theorem 18.

Theorem 19.

References

Theorem 9 ([11, Lemma 2.4], $k = 0$ case, in our terminology).