Approximation with Tensor Networks. Part I: Approximation Spaces

06/30/2020
by   Mazen Ali, et al.
0

We study the approximation of functions by tensor networks (TNs). We show that Lebesgue L^p-spaces in one dimension can be identified with tensor product spaces of arbitrary order through tensorization. We use this tensor product structure to define subsets of L^p of rank-structured functions of finite representation complexity. These subsets are then used to define different approximation classes of tensor networks, associated with different measures of complexity. These approximation classes are shown to be quasi-normed linear spaces. We study some elementary properties and relationships of said spaces. In part II of this work, we will show that classical smoothness (Besov) spaces are continuously embedded into these approximation classes. We will also show that functions in these approximation classes do not possess any Besov smoothness, unless one restricts the depth of the tensor networks. The results of this work are both an analysis of the approximation spaces of TNs and a study of the expressivity of a particular type of neural networks (NN) – namely feed-forward sum-product networks with sparse architecture. The input variables of this network result from the tensorization step, interpreted as a particular featuring step which can also be implemented with a neural network with a specific architecture. We point out interesting parallels to recent results on the expressivity of rectified linear unit (ReLU) networks – currently one of the most popular type of NNs.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

06/30/2020

Approximation with Tensor Networks. Part II: Approximation Rates for Smoothness Classes

We study the approximation by tensor networks (TNs) of functions from cl...
01/28/2021

Approximation with Tensor Networks. Part III: Multivariate Approximation

We study the approximation of multivariate functions with tensor network...
04/04/2018

Identification of Shallow Neural Networks by Fewest Samples

We address the uniform approximation of sums of ridge functions ∑_i=1^m ...
12/14/2021

Simplicial approximation to CW complexes in practice

We describe an algorithm that takes as an input a CW complex and returns...
12/17/2019

Tensor Rank bounds for Point Singularities in R^3

We analyze rates of approximation by quantized, tensor-structured repres...
12/11/2017

Results in descriptive set theory on some represented spaces

Descriptive set theory was originally developed on Polish spaces. It was...
02/08/2021

High-dimensional nonlinear approximation by parametric manifolds in Hölder-Nikol'skii spaces of mixed smoothness

We study high-dimensional nonlinear approximation of functions in Hölder...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.