Singular Value Decomposition in Sobolev Spaces: Part II

12/24/2019
by   Mazen Ali, et al.
Ecole Centrale Nantes
Universität Ulm
0

Under certain conditions, an element of a tensor product space can be identified with a compact operator and the singular value decomposition (SVD) applies to the latter. These conditions are not fulfilled in Sobolev spaces. In the previous part of this work (part I), we introduced some preliminary notions in the theory of tensor product spaces. We analyzed low-rank approximations in H1 and the error of the SVD performed in the ambient L2 space. In this work (part II), we continue by considering variants of the SVD in norms stronger than the L2-norm. Overall and, perhaps surprisingly, this leads to a more difficult control of the H1-error. We briefly consider an isometric embedding of H1 that allows direct application of the SVD to H1-functions. Finally, we provide a few numerical examples that support our theoretical findings.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/16/2021

Quaternion tensor singular value decomposition using a flexible transform-based approach

A flexible transform-based tensor product named ⋆_QT-product for Lth-ord...
02/08/2019

A Fast Algorithm for Cosine Transform Based Tensor Singular Value Decomposition

Recently, there has been a lot of research into tensor singular value de...
06/05/2019

RIP-based performance guarantee for low-tubal-rank tensor recovery

The essential task of multi-dimensional data analysis focuses on the ten...
12/07/2020

A Singular Value Perspective on Model Robustness

Convolutional Neural Networks (CNNs) have made significant progress on s...
05/17/2020

Identification/Segmentation of Indian Regional Languages with Singular Value Decomposition based Feature Embedding

language identification (LID) is identifing a language in a given spoken...
10/13/2021

Comparison of SVD and factorized TDNN approaches for speech to text

This work concentrates on reducing the RTF and word error rate of a hybr...
07/19/2019

A Note on Exploratory Item Factor Analysis by Singular Value Decomposition

In this note, we revisit a singular value decomposition (SVD) based algo...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

A function in the tensor product of two Hilbert spaces and may be identified with a compact operator . This identification is possible when the norm on is not weaker than the injective norm, i.e., in a certain sense the norm on is compatible with the norms on and . In such a case we can decompose as

(1.1)

for a non-negative non-increasing sequence and orthonormal systems and . This is the well known singular value decomposition (SVD) and it provides low-rank approximations via

if, e.g., is the canonical norm on a Hilbert tensor space.

If is the Sobolev space of once weakly differentiable functions, the above assumption is not satisfied and there is no SVD for a function in general. The focus of this work is to explore variants of the SVD in different ambient spaces in the -norm. In part I, we showed that low-rank approximations in the Tucker format in exist. More precisely,

Theorem 1.1.

is weakly closed and therefore proximinal in .

We also showed under which conditions

i.e., belongs to the tensor product of its minimal subspaces. Finally, we analyzed the -error of the -SVD for a general order .

In this part, we consider the intersection space structure of

We analyze the -error of the - and -SVDs. We also consider an isometric embedding of into a space which allows the direct application of the SVD.

The paper is organized as follows. In Section 2, we consider the - and -SVDs and generalizations to higher dimensions. In Section 3, we consider the SVD in higher-dimensional spaces of mixed smoothness, exponential sum approximations and an isometric embedding of that allows a direct application of the SVD. We conclude in Section 4 with some simple numerical experiments with different types of low-rank approximations. Section 5 summarizes the results of part I and part II.

2. SVD in and

Before we proceed with analyzing SVDs in and , we consider the corresponding singular values and compare them to -singular values.

2.1. and Singular Values

We consider a function as an element of the intersection space We first consider as a Hilbert Schmidt operator defined by

The difference to simply viewing as an integral kernel arises when we consider the adjoint

The corresponding left and right singular functions and are respectively given by

and

(2.1)

with the corresponding singular values sorted in decreasing order. Note that, unlike in the previous subsection, in general . To guarantee this we would have to require . This means that the sum does not make sense in in general, only in .

Similarly, we can interpret as a Hilbert Schmidt operator
defined by

with an adjoint given by The corresponding singular functions and satisfy

and

(2.2)

where are the corresponding singular values, sorted in decreasing order. We make the following immediate observation.

Proposition 2.1.

Let and let , and be the SVD of interpreted as an element of , and respectively. Then, we have for all

Proof.

The first statement is given by

and

Analogously for the second statement. ∎

Note that the upper bounds in Proposition 2.1 do not necessarily hold component-wise, i.e., the inequalities

do not hold in general. This is due to the fact that when estimating the injective norm

the functions are not orthonormal in and the sequence is not necessarily decreasing.

Naturally, we can ask whether we can derive a bound of the sort

for some sequence . Though we do not believe this is possible without further assumptions, we can nonetheless improve the bounds. This indicates that indeed the quantities and are closely related. This will later be confirmed by numerical observations.

Theorem 2.2.

Let and assume the -SVD converges in . Then, we have

Proof.

We consider the -SVD . is identified with an operator . For any ,
converges in and for any , convergences in . Thus,

On the other hand, utilizing the -SVD of , we have
and thus

Substituting , we obtain since are -orthonormal. Finally, taking the -norm of both sides and since are -orthonormal, we obtain
The statement for follows analogously by identifying with an operator from to . This completes the proof. ∎

The factors in the bounds in Theorem 2.2 reflect how , normalized in , scales w.r.t. , normalized in . For instance, if behaves like Fourier or wavelet basis, then . In this case, the right hand side in Theorem 2.2 evaluates to This leads precisely to the upper bound of Proposition 2.1. Analogous conclusions hold when considering , and .

Extending the results of this subsection to using HOSVD singular values and, e.g., the Tucker format is straightforward. Since we can consider matricizations w.r.t. to each separately, the analysis effectively reduces to the case . Difficulties arise only when considering simultaneous projections in different components of the tensor product space. There we have to assume the rescaled singular values converge, as was done in part I of this work.

2.2. and projections

Given the singular functions
and associated with and SVDs of respectively, we consider the finite dimensional subspaces

(2.3)

and the corresponding -orthogonal projections

(2.4)

The tensor product is well defined on , and on this space it holds

(2.5)

However, the interpretation is problematic when considering on the closure of . Take, e.g., the projection . This is an orthogonal projection on and we have But in general unless . Thus, the subsequent application does not necessarily make sense and is not continuous.

Notice the difference with the projections and for -SVD from part I (for and ). First, we had since both the left and right projections already give the best rank approximation in . Second, we required only -orthogonality, thus preserving -regularity in the image. Thus, made sense on , although the sequence of projections does not necessarily converge in . To that end, we had to additionally assume in part I the convergence of the rank- approximations , or convergence of the rescaled -singular values.

In the present case, although we obtain optimality in the stronger -norm, we lose convergence or possibly even boundedness in the -norm. Thus, we can ask ourselves if is bounded from to , i.e., if ?

Specifically, what are the minimal assumptions - if any - that we require in order to achieve this? The next example shows that indeed even for simple projections this property is not guaranteed.

Example 2.3.

Let and consider the space . We know . Consider defined by Clearly, is a linear functional. Moreover, since any such is absolutely continuous, is bounded in the -norm. Thus, . By the Riesz representation theorem, there exists a unique , such that , for all .

Define the one dimensional subspace . The corresponding -orthogonal projection is given by Consider the sequence

. Clearly, for any , , and , for all . Thus, can not be continuous in .

A closer look at the preceding example shows that such a function differentiated twice yields the delta distribution. Therefore, it can not be in . On the other hand, if the function has -regularity, as the next statement shows, we can indeed obtain boundedness in .

Lemma 2.4.

Let . In addition, assume the second unidirectional derivatives of exist in the distributional sense and are bounded, i.e., Finally, assume satisfies either zero Dirichlet or zero Neumann boundary conditions. Then, the projections defined in (2.4) can be bounded as

Proof.

One can easily verify that and are twice weakly differentiable for any . For any , we can write The coefficients can be written as

where the boundary term vanishes due to the boundary conditions. Thus, we get

Analogously for . This completes the proof. ∎

Note that in principle the assumption on the boundary conditions can be replaced or avoided, as long as we can estimate the appearing boundary term. The assumption can be avoided entirely by using an estimate for the -norm via the Gagliardo-Nirenberg inequality, although this would yield a crude estimate and dimension dependent regularity requirements.

Under the conditions of Lemma 2.4, we can assert that is indeed continuous. Since is a uniform crossnorm

and similarly for . Thus, By density, we can uniquely extend onto and the identity (2.5) holds.

One might argue that requiring and to be continuous in is unnecessary, since we only need that the mappings and are continuous. The following example shows that indeed need not be continuous even on elementary tensor products, if is not continuous.

Example 2.5.

Take to be the projection from Example 2.3. Consider the same sequence as in Example 2.3. Take another sequence as

Then,

Thus, since is a crossnorm

On the other hand

Hence, is not continuous on even on .

To summarize our findings, let us define the finite dimensional subspaces Under the assumptions of Lemma 2.4, and . This can also be observed by, e.g., considering (2.1) and integrating the second term by parts. We can estimate the error as follows.

Theorem 2.6.

Let the assumptions of Lemma 2.4 be satisfied. Moreover, define the constants

Then, the projection error is bounded as

Proof.

For the lower bound observe first that

Since is the optimal rank approximation in the -norm, we can further estimate

and similarly for . This gives the lower bound.

For the upper bound, since is orthogonal in the -norm, we get