# Nuclear Norm and Spectral Norm of Tensor Product

We show that the nuclear norm of the tensor product of two tensors is not greater than the product of the nuclear norms of these two tensors. As an application, we give lower bounds for the nuclear norm of an arbitrary tensor. We show that the spectral norm of the tensor product of two tensors is not greater than the product of the spectral norm of one tensor, and the nuclear norm of another tensor. By this result, we give some lower bounds for the product of the nuclear norm and the spectral norm of an arbitrary tensor. The first result also shows that the nuclear norm of the square matrix is a matrix norm. We then extend the concept of matrix norm to tensor norm. A real function defined for all real tensors is called a tensor norm if it is a norm for any tensor space with fixed dimensions, and the norm of the tensor product of two tensors is always not greater than the product of the norms of these two tensors. We show that the 1-norm, the Frobenius norm and the nuclear norm of tensors are tensor norms but the infinity norm and the spectral norm of tensors are not tensor norms.

## Authors

• 20 publications
• 7 publications
• 12 publications
• ### Norm of Tensor Product, Tensor Norm, and Cubic Power of A Third Order Tensor

We establish two inequalities for the nuclear norm and the spectral norm...
09/24/2019 ∙ by Liqun Qi, et al. ∙ 0

• ### Spectral Norm and Nuclear Norm of a Third Order Tensor

The spectral norm and the nuclear norm of a third order tensor play an i...
09/04/2019 ∙ by Liqun Qi, et al. ∙ 0

Biquadratic tensors play a central role in many areas of science. Exampl...
09/30/2019 ∙ by Liqun Qi, et al. ∙ 0

• ### Nuclear Norm Under Tensor Kronecker Products

Derksen proved that the spectral norm is multiplicative with respect to ...
01/20/2020 ∙ by Robert Cochrane, et al. ∙ 0

• ### Spectral norm of random tensors

We show that the spectral norm of a random n_1× n_2×...× n_K tensor (or ...
07/07/2014 ∙ by Ryota Tomioka, et al. ∙ 0

• ### On orthogonal tensors and best rank-one approximation ratio

As is well known, the smallest possible ratio between the spectral norm ...
07/09/2017 ∙ by Zhening Li, et al. ∙ 0

• ### Convex Coupled Matrix and Tensor Completion

We propose a set of convex low rank inducing norms for a coupled matrice...
05/15/2017 ∙ by Kishan Wimalawarne, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

The tensor completion and recovery problem aims to fill the missing or unobserved entries of partially observed tensors. It has received wide attention and achievements in areas like data mining, computer vision, signal processing, and neuroscience

[10, 11]. Former approaches often proceed by unfolding tensors to matrices and then apply for matrix completion. Yuan and Zhang [11]

showed that such matricization fails to exploit the structure of tensors and may lead to sub-optimality. They proposed to minimize a tensor nuclear norm directly and proved that such an approach improves the sample size requirement. This leads research enthusiasm on the tensor nuclear norm and its dual norm, i.e., the tensor spectral norm

[4, 5, 6, 7, 8, 9], though this is in fact a NP-hard problem [2].

In this paper, we establish two inequalities for the nuclear norm and the spectral norm of tensor products.

In the next section, we show that the nuclear norm of the tensor product of two tensors is not greater than the product of the nuclear norms of these two tensors.

As an application, in Section 3, we give lower bounds for the nuclear norm of an arbitrary tensor.

However, in general, the spectral norm of the tensor product of two tensors may be greater than the product of the spectral norms of these two tensors. In Section 4, we give a counterexample to illustrate this. Then, as a substitute, in that section, we show that the spectral norm of the tensor product of two tensors is not greater than the product of the spectral norm of one tensor, and the nuclear norm of another tensor.

By the inequality established in Section 4, in Section 5, we give an alternative formula for the spectral norm of a tensor. Then we introduce contraction matrices for a tensors and show that the spectral norms of the contraction matrices of a tensor are lower bounds for the product of the nuclear norm and the spectral norm of that tensor.

In matrix analysis [3]

, the matrix norm is a concept different from the vector norm. We may regard a matrix space as a vector space. A norm on that matrix space is called a matrix norm. If in additional, that norm satisfies the axiom that the norm of the product of two arbitrary matrices is always not greater than the product of the norms of these two matrices, then that norm is called a matrix norm. It was shown that the

-norm, the -norm and all the induced norms of matrices are matrix norms, but the infinity norm of matrices is not a matrix norm [3]. Then, by the result in Section 2, we conclude that the nuclear norm of the square matrix is also a matrix norm. We study this in Section 6. Viewing the significance e of the nuclear norm of matrices in the matrix completion problem [1], and the importance of the matrix norm in matrix analysis [3], this result may be useful in the related research.

In Section 7, we extend the concept of matrix norm to tensor norm. A real function defined for all real tensors is called a tensor norm if it is a norm for any tensor space with fixed dimensions, and the norm of the tensor product of two tensors is always not greater than the product of the norms of these two tensors. We show that the -norm, the Frobenius norm and the nuclear norm of tensors are tensor norms but the infinity norm and the spectral norm of tensors are not tensor norms.

Some final remarks are made in Section 8.

In this paper, unless otherwise stated, all the discussions will be carried out in the filed of real numbers. We use small letters , etc., to denote scalars, small bold letters , etc., to denote vectors, capital letters , etc., to denote matrices, and calligraphic letters , etc., to denote tensors.

## 2 Nuclear Norm of Tensor Product

Let be the set of positive integers, and be the set of nonnegative integers. For , we use to denote the set . For a vector , we use to denote its 2-norm. Thus,

 ∥u∥2:=√u21+⋯+u2n.

Suppose that a th order tensor , where is called the order of , and for are called the dimensions of . We use to denote tensor outer product. Then for ,

 u(1)∘⋯∘u(k)

is a rank-one th order tensor. The nuclear norm of is defined [2, 4, 7] as

 (2.1)

We have the following theorem.

###### Theorem 2.1

Suppose that , , and a tensor product of and is defined as by

 ci1⋯ikik+p+1⋯ik+p+q=nk+1∑ik+1=1⋯nk+p∑ik+p=1ai1⋯ik+pbik+1⋯ik+p+q,

for , with , . Then

 ∥C∥∗≤∥A∥∗∥B∥∗. (2.2)

Proof Let . Then we have

 A=r1∑j=1λju(1,j)∘⋯∘u(k+p,j),

where for , and , and

 B=r2∑l=1μlv(k+1,l)∘⋯∘v(k+p+q,l),

where for , and , such that

 r1∑j=1|λj|≤∥A∥∗+ϵ

and

 r2∑l=1|μl|≤∥B∥∗+ϵ.

This implies that

 C=r1∑j=1r2∑l=1λjμlp∏i=1⟨u(k+i,j),v(k+i,l)⟩u(1,j)∘⋯∘u(k,j)∘v(k+p+1,l)∘⋯∘v(k+p+r,l).

Hence,

 ∥C∥∗ ≤r1∑j=1r2∑l=1|λjμlp∏i=1⟨u(k+i,j),v(k+i,l)⟩| ≤r1∑j=1r2∑l=1|λjμl|p∏i=1∥u(k+i,j)∥2∥v(k+i,l)∥2 =r1∑j=1r2∑l=1|λjμl| =(r1∑j=1|λj|)(r2∑l=1|μl|) ≤(∥A∥∗+ϵ)(∥B∥∗+ϵ),

where the first inequality is by the definition (2.1), and the second inequality is by the Cauchy inequality. Letting , we have (2.2). .

This establishes the first main result of this paper.

## 3 An Application: Lower Bounds for the Nuclear Norm of a Tensor

In this section, we present a lower bound for the nuclear norm of an arbitrary even order tensor for and .

We first assume that . Then is a third order tensor. It is not easy to compute its nuclear norm.

###### Proposition 3.1

Suppose that , where . Let , and be defined by

 ci=n2∑i2=1n3∑i3=1aii2i3bi2i3, (3.3)

for . Then,

 ∥A∥∗≥max{∥c∥∗:c is calculated by (???),B∈Rn2×n3,∥B∥∗=1}. (3.4)

Proof Applying Theorem 2.1 with and , , and , we have the conclusion. .

Here, is a matrix, and is a vector. Their nuclear norms are not difficult to be calculated. The tensor in the following example is originally from [2].

###### Example 3.2

Let , , and be a third order symmetric tensor defined by

 A=12(e(1)∘e(1)∘e(2)+e(1)∘e(2)∘e(1)+e(2)∘e(1)∘e(1)−e(2)∘e(2)∘e(2)),

where and . Let be the all one matrix in . We may let . We may calculate by (3.4). Then by (3.4), we have

 ∥A∥∗≥∥c∥∗≡0.6455.

Actually, . This verifies (3.4) somehow.

We then consider the case that . Then is a fourth order tensor. It is also not easy to compute its nuclear norm.

###### Proposition 3.3

Suppose that , where . Let , and be defined by

 ci1i2=n3∑i3=1n4∑i4=1ai1i2i3i4bi3i4, (3.5)

for . Then,

 ∥A∥∗≥max{∥C∥∗:C is calculated by (???),B∈Rn3×n4,∥B∥∗=1}. (3.6)

Proof Applying Theorem 2.1 with and , and , we have the conclusion. .

Here, and are matrices. Their nuclear norms are not difficult to be calculated. The following example is from [8].

###### Example 3.4

Let , , and be a fourth order symmetric tensor defined by

where , , and . Let be the all one matrix in . We may let . We may calculate by (3.5). Then by (3.6), we have

 ∥A∥∗≥∥C∥∗≡10.3757.

Actually, . This verifies (3.6) somehow.

We now extend Propositions 3.1 and 3.3 to the case that . We have the following two propositions.

###### Proposition 3.5

Suppose that , where , . Let for , and be defined by

 ci=n2∑i2=1⋯nk∑ik=1aii2⋯ikb(1)i2i3⋯b(l)i2li2l+1, (3.7)

for . Then,

 ∥A∥∗≥max{∥c∥∗:c is calculated by (???),B(j)∈Rn2j×n2j+1,∥B(j)∥∗=1,j∈[l]}. (3.8)

Proof Applying Theorem 2.1 repetitively, we have the conclusion. .

Here, for are matrices, and is a vector. Their nuclear norms are not difficult to be calculated.

###### Proposition 3.6

Suppose that , where , . Let for , and be defined by

 ci1i2=n3∑i3=1⋯nk∑ik=1ai1⋯ikb(1)i3i4⋯b(l)i2l+1i2l+2, (3.9)

for and . Then,

 ∥A∥∗≥max{∥C∥∗:C is calculated by (???),B(j)∈Rn2j+1×n2j+2,∥B(j)∥∗=1,j∈[l]}. (3.10)

Proof Applying Theorem 2.1 repetitively, we have the conclusion. .

Here, for , and are matrices. Their nuclear norms are not difficult to be calculated.

## 4 Spectral Norm of Tensor Product

For , their inner product is defined as

 ⟨A,B⟩:=n1∑i1=1⋯nk∑ik=1ai1⋯ikbi1⋯ik.

Then the spectral norm of is defined [2, 4, 7, 11] as

 ∥A∥S:=max{⟨A,u(1)∘⋯∘u(k)⟩,u(i)∈Rni,∥∥u(i)∥∥=1, for i∈[k]}. (4.11)

It is known [11] that we always have

 ∥A∥S≤∥A∥∗.

Note that in general, we do not have

 ∥C∥S≤∥A∥S∥B∥S,

if is a tensor product of and , as in Theorem 2.1. See the following example.

###### Example 4.1

Let , and for . Let be defined by

 a1111=2, a1211=3, a2111=−6, a2211=3, a1121=−6, a1221=3, a2121=4, a2221=3, a1112=3, a1212=9, a2112=3, a2212=−3, a1122=3, a1222=−3, a2122=3, a2222=15.

Then we have

 c1111=58, c1211=6, c2111=−18, c2211=24, c1121=−18, c1221=12, c2121=70, c2221=30, c1112=6, c1212=108, c2112=12, c2212=−54, c1122=24, c1222=−54, c2122=30, c2222=1252.

By calculation, we have , and . Then , which is slightly less than .

However, we may establish the following theorem.

###### Theorem 4.2

Suppose that , , and a tensor product of and is defined as by

 ci1⋯ikik+p+1⋯ik+p+q=nk+1∑ik+1=1⋯nk+p∑ik+p=1ai1⋯ik+pbik+1⋯ik+p+q,

for , with , . Then

 ∥C∥S≤∥A∥S∥B∥∗. (4.12)

Proof Let . Then we have

 B=r∑j=1μjv(k+1,j)∘⋯∘v(k+p+q,j),

where for , and , such that

 r∑j=1|μj|≤∥B∥∗+ϵ.

Then

 bik+1⋯ik+p+q=r∑j=1μjv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+q,

for , and .

We have

 ci1⋯ikik+p+1⋯ik+p+r=r∑j=1nk+1∑ik+1=1⋯nk+p∑ik+p=1μjai1⋯ik+pv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+q,

for , and . This implies that

 ∥C∥S = = max⎧⎨⎩r∑j=1μjn1∑i1=1⋯nk+p+q∑ik+p+q=1ai1⋯ik+pu(1)i1⋯u(k)ikv(k+1,j)ik+1⋯v(k+p+q,j)ik+p+qu(k+p+1)il+p+1⋯u(k+p+q)ik+p+q⎫⎬⎭ ≤ max{r∑j=1∣∣μj∣∣∣∣⟨A,u(1)∘⋯∘u(k)∘v(k+1,j)∘⋯∘v(k+p,j)⟩∣∣k+p+q∏l=k+p+1∣∣⟨v(l,j),u(l)⟩∣∣} ≤ max{r∑j=1∣∣μj∣∣∥A∥} ≤ ∥A∥S(∥B∥∗+ϵ),

where the second inequality is by the definition (4.11) and the Cauchy inequality. Letting , we have (4.12). .

This establishes the second main result of this paper.

## 5 Applications

In this section, as applications of Theorem 4.2, we present an alternative formula for the spectral norm of a tensor, and give some lower bounds for the product of the nuclear norm and the spectral norm of an arbitrary tensor.

### 5.1 An Alternative Formula for the Spectral Norm of a Tensor

We present an alternative formula for the spectral norm of a tensor in this subsection. This formula does not reduce the complexity of the problem, as this is impossible, but gives an alternative approach to handle the spectral norm.

###### Proposition 5.1

Suppose that , where , . For , define by

 ci1⋯ik−2=nk−1∑ik−1=1nk∑ik=1ai1⋯ikbik−1ik, (5.13)

for . Then

 ∥A∥S=max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}. (5.14)

Proof By Theorem 4.2, we have

 ∥A∥S≥max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}.

On the other hand,

 ∥A∥S =max{⟨A,u(1)∘⋯∘u(k)⟩,u(i)∈Rni,∥∥u(i)∥∥=1, for i∈[k]} ≤max{⟨A,u(1)∘⋯∘u(k−2)∘B⟩,u(i)∈Rni,∥∥u(i)∥∥=1, for i∈[k−2],B∈Rnk−1×nk,∥B∥∗=1} =max{∥C∥S:C is calculated by (???),B∈Rnk−1×nk,∥B∥∗=1}.

This proves (5.14). .

### 5.2 Lower Bounds for the Product of the Nuclear Norm and Spectral Norm of a Tensor

We now present some lower bounds for the product of the nuclear norm and the spectral norm of . Denote the spectral radius of a matrix by . We first introduce contraction matrices of .

###### Definition 5.2

Let , where . Assume that . Let . Define a symmetric matrix by

 a(j)rs=n1∑i1=1⋯nj−1∑ij−1=1nj+1∑ij+1=1⋯nk∑ik=1ai1⋯ik−1rik+1⋯ikai1⋯ik−1sik+1⋯ik,

for . We call the th contraction matrix of .

###### Proposition 5.3

Suppose that , where . Assume that . Let . Then,

 ρ(A(j))≡∥A(j)∥S≤∥A∥∗∥A∥S. (5.15)

Proof Apply Theorem 4.2 with and . Note that is symmetric. Hence, its spectral norm is its spectral radius. We then have the conclusion. .

As

is a symmetric matrix, its spectral norm, i.e., its spectral radius is the largest absolute value of its eigenvalues. which is not difficult to be calculated. For

, this theorem gives lower bounds for the product of the nuclear norm and the spectral norm of . The tensor in the following example is the same as the tensor in Example 3.2. It is originally from [2].

###### Example 5.4

Let , , and be a third order symmetric tensor defined by

 A=12(e(1)∘e(1)∘e(2)+e(1)∘e(2)∘e(1)+e(2)∘e(1)∘e(1)−e(2)∘e(2)∘e(2)),

where and . Then

 A(1)=A(2)=A(3)=0.5I2,

where

is the identity matrix in

. Then we have , and . This verifies (5.15).

## 6 Matrix Norm

In matrix analysis [3], the matrix norm is a concept different from the vector norm. Consider matrices in . Let . If it is not only a vector norm in , but it also satisfies the following additional axiom: for any ,

 |||AB|||≤|||A|||⋅|||B|||, (6.16)

then is a matrix norm in . Otherwise, it is only a vector norm. In particular, -norm, -norm and any induced norm are matrix norms, but -norm is only a vector norm, not a matrix norm. Matrix norms play an important role in matrix analysis. See [3] for more details on matrix norms.

By Theorem 2.1, we have the following proposition.

###### Proposition 6.1

For , the nuclear norm is a matrix norm.

As the nuclear norm plays a significant role in the matrix completion problem [1], this conclusion for matrix nuclear norm should be useful in the related research.

By Theorems 2.1 and 4.2, we have the following further results.

###### Proposition 6.2

Suppose that is invertible. Then we have

 ∥A∥∗∥A−1∥∗≥n, (6.17)

and

 ∥A∥∗∥A−1∥≥1. (6.18)

Proof Apply (2.2) and (4.12) to and . Note that and , where is the identity matrix in . The conclusions hold. .

## 7 Tensor Norm

We are now ready to extend the concept of matrix norms to tensor norms.

###### Definition 7.1

Suppose that is a function defined for all real tensors, and in any real tensor space of fixed dimensions with , it is a vector norm. If furthermore for any two real tensors and such that and have an outer tensor product , we have

 |||C|||≤|||A|||⋅|||B|||, (7.19)

then is called a tensor norm.

Clearly, a tensor norm must be a matrix norm if it is restricted to . We have the following theorem.

###### Theorem 7.2

The nuclear norm, the 1-norm, the Frobenius norm are tensor norms, but the infinity norm and the spectral norm are not tensor norms.

Proof By Theorem 2.1, the nuclear norm is a tensor norm. Since the infinity norm is not a matrix norm, it is also not a tensor norm. By the counter example in Example 4.1, the spectral norm is not a tensor norm. What we need to check are the 1-norm and the Frobenius norm.

Suppose that , , and a tensor product of and is defined as by

 ci1⋯ikik+p+1⋯ik+p+q=nk+1∑ik+1=1⋯nk+p∑ik+p=1ai1⋯ik+pbik+1⋯ik+p+q,

for