 # Tensor Norm, Cubic Power and Gelfand Limit

We establish two inequalities for the nuclear norm and the spectral norm of tensor products. The first inequality indicts that the nuclear norm of the square matrix is a matrix norm. We extend the concept of matrix norm to tensor norm. We show that the 1-norm, the Frobenius norm and the nuclear norm of tensors are tensor norms but the infinity norm and the spectral norm of tensors are not tensor norms. We introduce the cubic power for a general third order tensor, and show that a Gelfand formula holds for a general third order tensor. In that formula, for any norm, a common spectral radius-like limit exists for that third order tensor. We call such a limit the Gelfand limit. The Gelfand limit is zero if the third order tensor is nilpotent, and is one or zero if the third order tensor is idempotent. The Gelfand limit is not greater than any tensor norm of that third order tensor, and the cubic power of that third order tensor tends to zero as the power increases to infinity if and only if the Gelfand limit is less than one. The cubic power and the Gelfand limit can be extended to any higher odd order tensors.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

The tensor completion and recovery problem aims to fill the missing or unobserved entries of partially observed tensors. It has received wide attention and achievements in areas like data mining, computer vision, signal processing, and neuroscience

[9, 10]. Former approaches often proceed by unfolding tensors to matrices and then apply for matrix completion. Yuan and Zhang 

showed that such matricization fails to exploit the structure of tensors and may lead to sub-optimality. They proposed to minimize a tensor nuclear norm directly and proved that such an approach improves the sample size requirement. This leads research enthusiasm on the tensor nuclear norm and its dual norm, i.e., the tensor spectral norm

[4, 5, 6, 7, 8], though this is in fact a NP-hard problem .

In matrix analysis 

, the matrix norm is a concept different from the vector norm. We may regard a matrix space as a vector space. A norm on that matrix space is called a vector norm. If in additional, that norm satisfies the axiom that the norm of the product of two arbitrary matrices is always not greater than the product of the norms of these two matrices, then that norm is called a matrix norm. It was shown that the

-norm, the -norm and all the induced norms of matrices are matrix norms, but the infinity norm of matrices is not a matrix norm .

The matrix norm of a square matrix is closely linked with the spectral radius of , i.e., , by the well-known Gelfand formula (1941):

 ρ(A)=limk→∞|||Ak|||1k,

for any matrix norm .

However, there is no “tensor norm” concept or a Gelfand formula for higher order tensors until now. In this paper, we explore this unknown territory.

In the next section, we show that the nuclear norm of the tensor product of two tensors is not greater than the product of the nuclear norms of these two tensors.

However, in general, the spectral norm of the tensor product of two tensors may be greater than the product of the spectral norms of these two tensors. In Section 3, we give a counterexample to illustrate this. As a substitute, in that section, we show that the spectral norm of the tensor product of two tensors is not greater than the product of the spectral norm of one tensor, and the nuclear norm of another tensor.

By the result in Section 2, we conclude in Section 4 that the nuclear norm of the square matrix is also a matrix norm. Viewing the significance e of the nuclear norm of matrices in the matrix completion problem , and the importance of the matrix norm in matrix analysis , this result may be useful in the related research.

In Section 5, we extend the concept of matrix norm to tensor norm. A real function defined for all real tensors is called a tensor norm if it is a norm for any tensor space with fixed dimensions, and the norm of the tensor product of two tensors is always not greater than the product of the norms of these two tensors. We show that the -norm, the Frobenius norm and the nuclear norm of tensors are tensor norms but the infinity norm and the spectral norm of tensors are not tensor norms.

In Section 6, we introduce the cubic power for a general third order tensor. The cubic power preserves nonnegativity, symmetry and the diagonal property. We show that the cubic power of a general third order tensor tends to zero as the power increases to infinity, if there is a tensor norm such that the tensor norm of that third order tensor is less than one.

We show in Section 7 that a Gelfand formula holds for a general third order tensor. In that formula, for any norm, a common spectral radius-like limit exists for that third order tensor. We call such a limit the Gelfand limit of that third order tensor. The Gelfand limit is zero if the third order tensor is nilpotent. We show that the Gelfand limit is less than or equal to any tensor norm of that third order tensor, and the cubic power of that third order tensor tends to zero as the power increases to infinity if and only if the Gelfand limit is less than one.

The cubic power and the Gelfand limit can be extended to any higher odd order tensors.

Some final remarks are made in Section 8.

The main theorem of this paper is Theorem 7.1, which shows that the Gelfand limit of a general third order tensor exists, and takes the same value for any norm. This implies that the Gelfand limit is an essential value of that third order tensor. In Sections 2, 3, 6 and 7, there is a subsection for each section. These subsections describe some sub-developments of such sections. The reader may skip them in the first reading.

In this paper, unless otherwise stated, all the discussions will be carried out in the filed of real numbers. We use small letters , etc., to denote scalars, small bold letters , etc., to denote vectors, capital letters , etc., to denote matrices, and calligraphic letters , etc., to denote tensors, with as the zero tensor with adequate order and dimensions.

## 2 Nuclear Norm of Tensor Product

Let be the set of positive integers, and be the set of nonnegative integers. For , we use to denote the set . For a vector , we use to denote its 2-norm. Thus,

 ∥u∥2:=√u21+⋯+u2n.

Suppose that a th order tensor , where is called the order of , and for are called the dimensions of . We use to denote tensor outer product. Then for nonzero ,

 u(1)∘⋯∘u(k)

is a rank-one th order tensor. The nuclear norm of is defined [2, 4, 7] as

 (2.1)

We have the following theorem.

###### Theorem 2.1

Suppose that , , and a tensor product of and is defined as by

 ci1⋯ikik+p+1⋯ik+p+q=nk+1∑ik+1=1⋯nk+p∑ik+p=1ai1⋯ik+pbik+1⋯ik+p+q,

for , with , . Then

 ∥C∥∗≤∥A∥∗∥B∥∗. (2.2)

Proof Let . Then we have

 A=r1∑j=1λju(1,j)∘⋯∘u(k+p,j),

where for , and , and

 B=r2∑l=1μlv(k+1,l)∘⋯∘v(k+p+q,l),

where for , and , such that

 r1∑j=1|λj|≤∥A∥∗+ϵ

and

 r2∑l=1|μl|≤∥B∥∗+ϵ.

This implies that

 C=r1∑j=1r2∑l=1λjμlp∏i=1⟨u(k+i,j),v(k+i,l)⟩u(1,j)∘⋯∘u(k,j)∘v(k+p+1,l)∘⋯∘v(k+p+r,l).

Hence,

 ∥C∥∗ ≤r1∑j=1r2∑l=1|λjμlp∏i=1⟨u(k+i,j),v(k+i,l)⟩| ≤r1∑j=1r2∑l=1|λjμl|p∏i=1∥u(k+i,j)∥2∥v(k+i,l)∥2 =r1∑j=1r2∑l=1|λjμl| =(r1∑j=1|λj|)(r2∑l=1|μl|) ≤(∥A∥∗+ϵ)(∥B∥∗+ϵ),

where the first inequality is by the definition (2.1), and the second inequality is by the Cauchy inequality. Letting , we have (2.2). .

The remaining part of this section is an application of this theorem, and is not related with the main part of this paper. A reader who is interested in tensor norm, cubic power and Gelfand limit may skip the remaining part of this section.

### 2.1 An Application: Lower Bounds for the Nuclear Norm of a Tensor

In this subsection, we present a lower bound for the nuclear norm of an arbitrary even order tensor for and .

We first assume that . Then is a third order tensor. It is not easy to compute its nuclear norm.

###### Proposition 2.2

Suppose that , where . Let , and be defined by

 ci=n2∑i2=1n3∑i3=1aii2i3bi2i3, (2.3)

for . Then,

 ∥A∥∗≥max{∥c∥∗:c is calculated by (???),B∈Rn2×n3,∥B∥∗=1}. (2.4)

Proof Applying Theorem 2.1 with and , , and , we have the conclusion. .

Here, is a matrix, and is a vector. Their nuclear norms are not difficult to be calculated. The tensor in the following example is originally from .

###### Example 2.3

Let , , and be a third order symmetric tensor defined by

 A=12(e(1)∘e(1)∘e(2)+e(1)∘e(2)∘e(1)+e(2)∘e(1)∘e(1)−e(2)∘e(2)∘e(2)),

where and . Let be the all one matrix in . We may let . We may calculate by (2.4). Then by (2.4), we have

 ∥A∥∗≥∥c∥∗≡0.6455.

Actually, . This verifies (2.4) somehow.

We then consider the case that . Then is a fourth order tensor. It is also not easy to compute its nuclear norm.

###### Proposition 2.4

Suppose that , where . Let , and be defined by

 ci1i2=n3∑i3=1n4∑i4=1ai1i2i3i4bi3i4, (2.5)

for . Then,

 ∥A∥∗≥max{∥C∥∗:C is calculated by (???),B∈Rn3×n4,∥B∥∗=1}. (2.6)

Proof Applying Theorem 2.1 with and , and , we have the conclusion. .

Here, and are matrices. Their nuclear norms are not difficult to be calculated. The following example is from .

###### Example 2.5

Let , , and be a fourth order symmetric tensor defined by

where , , and . Let be the all one matrix in . We may let . We may calculate by (2.5). Then by (2.6), we have

 ∥A∥∗≥∥C∥∗≡10.3757.

Actually, . This verifies (2.6) somehow.

We now extend Propositions 2.2 and 2.4 to the case that . We have the following two propositions.

###### Proposition 2.6

Suppose that , where , . Let for , and be defined by

 ci=n2∑i2=1⋯nk∑ik=1aii2⋯ikb(1)i2i3⋯b(l)i2li2l+1, (2.7)

for . Then,

 ∥A∥∗≥max{∥c∥∗:c is calculated by (???),B(j)∈Rn2j×n2j+1,∥B(j)∥∗=1,j∈[l]}. (2.8)

Proof Applying Theorem 2.1 repetitively, we have the conclusion. .

Here, for are matrices, and is a vector. Their nuclear norms are not difficult to be calculated.

###### Proposition 2.7

Suppose that , where , . Let for , and be defined by

 ci1i2=n3∑i3=1⋯nk∑ik=1ai1⋯ikb(1)i3i4⋯b(l)i2l+1i2l+2, (2.9)

for and . Then,

 ∥A∥∗≥max{∥C∥∗:C is calculated by (???),B(j)∈Rn2j+1×n2j+2,∥B(j)∥∗=1,j∈[l]}. (2.10)

Proof Applying Theorem 2.1 repetitively, we have the conclusion. .

Here, for , and are matrices. Their nuclear norms are not difficult to be calculated.

## 3 Spectral Norm of Tensor Product

For , their inner product is defined as

 ⟨A,B⟩:=n1∑i1=1⋯nk∑ik=1ai1⋯ikbi1⋯ik.

Then the spectral norm of is defined [2, 4, 7, 10] as

 ∥A∥S:=max{⟨A,u(1)∘⋯∘u(k)⟩,u(i)∈Rni,∥∥u(i)∥∥2=1, for i∈[k]}. (3.11)

It is known  that we always have

 ∥A∥S≤∥A∥∗.

Note that in general, we do not have

 ∥C∥S≤∥A∥S∥B∥S,

if is a tensor product of and , as in Theorem 2.1. See the following example.

###### Example 3.1

Let , and for . Let be defined by

 a1111=2, a1211=3, a2111=−6, a2211=3, a1121=−6, a1221=3, a2121=4, a2221=3, a1112=3, a1212=9, a2112=3, a2212=−3, a1122=3, a1222=−3, a2122=3, a2222=15.

Then we have

 c1111=58, c1211=6, c2111=−18, c2211=24, c1121=−18, c1221=12, c2121=70, c2221=30, c1112=6, c1212=108, c2112=12, c2212=−54, c1122=24, c1222=−54, c2122=30, c2222=1252.

By calculation, we have , and . Then , which is slightly less than .

However, we may establish the following theorem.

###### Theorem 3.2

Suppose that , , and a tensor product of and is defined as by

 ci1⋯ikik+p+1⋯ik+p+q=nk+1∑ik+1=1⋯nk+p∑ik+p=1ai1⋯ik+pbik+1⋯ik+p+q,

for , with , . Then

 ∥C∥S≤∥A∥S∥B∥∗. (3.12)

Proof Let . Then we have

 B=r∑j=1μjv(k+1,j)∘⋯∘v(k+p+q,j),

where for , and , such that

 r∑j=1|μj|≤∥B∥∗+ϵ.

Then

 bik+1⋯ik+p+q=r∑j=1μjv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+q,

for , and .

We have

 ci1⋯ikik+p+1⋯ik+p+r=r∑j=1nk+1∑ik+1=1⋯nk+p∑ik+p=1μjai1⋯ik+pv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+q,

for , and . This implies that

 ∥C∥S = = max⎧⎨⎩r∑j=1μjn1∑i1=1⋯nk+p+q∑ik+p+q=1ai1⋯ik+pu(1)i1⋯u(k)ikv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+qu(k+p+1)il+p+1⋯u(k+p+q)ik+p+q⎫⎬⎭ ≤ max{r∑j=1∣∣μj∣∣∣∣⟨A,u(1)∘⋯∘u(k)∘v(k+1,j)∘⋯∘v(k+p,j)⟩∣∣k+p+q∏l=k+p+1∣∣⟨v(l,j),u(l)⟩∣∣} ≤ max{r∑j=1∣∣μj∣∣∥A∥S} ≤ ∥A∥S(∥B∥∗+ϵ),

where the second inequality is by the definition (3.11) and the Cauchy inequality. Letting , we have (3.12). .

The remaining part of this section contains some applications of this theorem, and is not related with the main part of this paper. A reader who is interested in tensor norm, cubic power and Gelfand limit may skip the remaining part of this section.

### 3.1 An Alternative Formula for the Spectral Norm of a Tensor

We present an alternative formula for the spectral norm of a tensor in this subsection. This formula does not reduce the complexity of the problem, as this is impossible, but gives an alternative approach to handle the spectral norm.

###### Proposition 3.3

Suppose that , where , . For , define by

 ci1⋯ik−2=nk−1∑ik−1=1nk∑ik=1ai1⋯ikbik−1ik, (3.13)

for . Then

 ∥A∥S=max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}. (3.14)

Proof By Theorem 3.2, we have

 ∥A∥S≥max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}.

On the other hand,

 ∥A∥S =max{⟨A,u(1)∘⋯∘u(k)⟩,u(i)∈Rni,∥∥u(i)∥∥2=1, for i∈[k]} ≤max{⟨A,u(1)∘⋯∘u(k−2)∘B⟩,u(i)∈Rni,∥∥u(i)∥∥2=1, for i∈[k−2],B∈Rnk−1×nk,∥B∥∗=1} =max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}.

This proves (3.14). .

### 3.2 Lower Bounds for the Product of the Nuclear Norm and Spectral Norm of a Tensor

We now present some lower bounds for the product of the nuclear norm and the spectral norm of . Denote the spectral radius of a matrix by . We first introduce contraction matrices of .

###### Definition 3.4

Let , where . Assume that . Let . Define a symmetric matrix by

 a(j)rs=n1∑i1=1⋯nj−1∑ij−1=1nj+1∑ij+1=1⋯nk∑ik=1ai1⋯ik−1rik+1⋯ikai1⋯ik−1sik+1⋯ik,

for . We call the th contraction matrix of .

###### Proposition 3.5

Suppose that , where . Assume that . Let . Then,

 ρ(A(j))≡∥A(j)∥S≤∥A∥∗∥A∥S. (3.15)

Proof Apply Theorem 3.2 with and . Note that is symmetric. Hence, its spectral norm is its spectral radius. We then have the conclusion. .

As

is a symmetric matrix, its spectral norm, i.e., its spectral radius is the largest absolute value of its eigenvalues, which is not difficult to be calculated. For

, this theorem gives lower bounds for the product of the nuclear norm and the spectral norm of . The tensor in the following example is the same as the tensor in Example 3.2. It is originally from .

###### Example 3.6

Let , , and be a third order symmetric tensor defined by

 A=12(e(1)∘e(1)∘e(2)+e(1)∘e(2)∘e(1)+e(2)∘e(1)∘e(1)−e(2)∘e(2)∘e(2)),

where and . Then

 A(1)=A(2)=A(3)=0.5I2,

where

is the identity matrix in

. Then we have , and . This verifies (3.15).

## 4 Matrix Norm

In matrix analysis , the matrix norm is a concept different from the vector norm. Consider matrices in . Let . If it is not only a vector norm in , but it also satisfies the following additional axiom: for any ,

 |||AB|||≤|||A|||⋅|||B|||, (4.16)

then is a matrix norm in . Otherwise, it is only a vector norm. In particular, -norm, -norm and any induced norm are matrix norms, but -norm is only a vector norm, not a matrix norm. Matrix norms play an important role in matrix analysis. See  for more details on matrix norms.

By Theorem 2.1, we have the following proposition.

###### Proposition 4.1

For , the nuclear norm is a matrix norm.

As the nuclear norm plays a significant role in the matrix completion problem , this conclusion for matrix nuclear norm should be useful in the related research.

By Theorems 2.1 and 3.2, we have the following further results.

###### Proposition 4.2

Suppose that is invertible. Then we have

 ∥A∥∗∥A−1∥∗≥n, (4.17)

and

 ∥A∥∗∥A−1∥S≥1. (4.18)

Proof Apply (2.2) and (3.12) to and . Note that and , where is the identity matrix in . The conclusions hold. .

###### Proposition 4.3

Suppose that . If , then

 limk→∞Ak=0.

Proof If , then

 ∥Ak∥∗≤∥A∥k∗→0.

The result follows. .

## 5 Tensor Norm

We are now ready to extend the concept of matrix norms to tensor norms.

###### Definition 5.1

Suppose that is a function defined for all real tensors, and in any real tensor space of fixed dimensions with , it is a vector norm. If furthermore for any two real tensors and such that and have an outer tensor product , we have

 |||C|||≤|||A|||⋅|||B|||, (5.19)

then is called a tensor norm.

Clearly, a tensor norm must be a matrix norm if it is restricted to . We have the following theorem.

###### Theorem 5.2

The nuclear norm, the 1-norm, the Frobenius norm are tensor norms, but the infinity norm and the spectral norm are not tensor norms.

Proof By Theorem 2.1, the nuclear norm is a tensor norm. Since the infinity norm is not a matrix norm, it is also not a tensor norm. By the counter example in Example 4.1, the spectral norm is not a tensor norm. What we need to check are the 1-norm and the Frobenius norm.

Suppose that , , and a tensor product of and is defined as