Connections Between Nuclear Norm and Frobenius Norm Based Representations

02/26/2015 ∙ by Xi Peng, et al. ∙ Sichuan University 0

A lot of works have shown that frobenius-norm based representation (FNR) is competitive to sparse representation and nuclear-norm based representation (NNR) in numerous tasks such as subspace clustering. Despite the success of FNR in experimental studies, less theoretical analysis is provided to understand its working mechanism. In this paper, we fill this gap by building the theoretical connections between FNR and NNR. More specially, we prove that: 1) when the dictionary can provide enough representative capacity, FNR is exactly NNR even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption, 2) otherwise, FNR and NNR are two solutions on the column space of the dictionary.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Many problems in machine learning and computer vision begin with the processing of linearly inseparable data. The goal of processing is to distinct linearly inseparable data with linear methods. To achieve this, the inputs are always projected from the original space into another space. This is so-called representation learning and three methods have been extensively investigated in the community of computer vision,

i.e., sparse representation (SR), low rank representation (LRR), and Frobenius-norm based representation (FNR).

During the past decade, sparse representation [1, 2] has been one of the most popular representation learning methods. It linearly reconstructs each sample using a few of basis and has shown the effectiveness in a lot of applications, e.g., image repairing [3]

, face recognition 

[4], online learning control [5], dimension reduction [6], and subspace clustering [7, 8].

As another popular method, low rank representation [9, 10, 11, 12, 13, 14] has been proposed for subspace learning and subspace clustering. Different from SR, LRR computes the representation of a data set rather than a data point by solving a nuclear norm minimization problem. Thus, LRR is also known as nuclear norm based representation (NNR). Both LRR and SR benefit from the compressive sensing theory [15] which establishes the equivalence between - (rank-minimization w.r.t. matrix space) and -norm (nuclear-norm w.r.t. matrix space) based optimization problems. More specifically, compressive sensing provides the theoretical foundation to transform the non-convex problem caused by -norm into a convex problem using -norm.

Recently, several works have shown that the Frobenius norm based representation (FNR) is competitive to SR and NNR in face recognition [16, 17, 18], subspace learning [19, 20]

, feature selection 

[21], and subspace clustering [22, 23]. The advantage of FNR is that the objective only involves a strictly convex problem and thus the trap of local minimal is avoided.

Although more and more experimental evidences have been provided to show the effectiveness of FNR, the success of FNR is counter-intuitive as FNR is generally considered to be inferior to SR and NNR. Furthermore, fewer theoretical studies have been done to explore what makes FNR effective. Motivated by two NNR works [10, 11], this paper provides a novel theoretical explanation by bridging FNR and NNR. In other words, we show that under some mild conditions, the convex problem caused by nuclear norm can be converted to a strictly convex problem based on the Frobenius norm. More specifically, we prove that: 1) when the dictionary has enough representative capacity, FNR is equivalent to the NNR [10, 11] even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption; 2) when the dictionary has limited representative capacity, FNR and NNR are two solutions of the column space spanned by inputs. Our theoretical results unify FNR and NNR into a framework, i.e., FNR and NNR are in the form of , where

is the singular value decomposition (SVD) of a given data matrix and

denotes the shrinkage-thresholding operator. The difference between FNR and NNR lies in the different choices of the shrinkage-thresholding operator. To the best of our knowledge, this is one of the first several works to establish the connections between FNR and NNR.

Ii Background

For a given data set (each column denotes a data point), it can be decomposed as the linear combination of by

(1)

where denotes the constraint enforced over the representation . The main difference among most existing works is their objective functions, basically, the choice of . Different assumptions motivate different and this work focuses on the discussion of two popular objective functions, i.e., nuclear-norm and Frobenius-norm.

By assuming is low rank and the input contains noise, Liu et al. [9] propose solving the following nuclear norm based minimization problem:

(2)

where , is the th singular value of , and could be chosen as -, -, or Frobenius-norm.

-norm is usually adopted to depict the sample-specific corruptions such as outliers,

-norm is used to characterize the Laplacian noise, and Frobenius norm is used to describe the Gaussian noise.

Although Eq.(2) can be easily solved by the Augmented Lagrangian method (ALM) [24], its computational complexity is still very high. Recently, Favaro and Vidal [10, 11] proposed a new formulation of LRR which can be calculated very fast. The proposed objective function is as follows:

(3)

where denotes the clean dictionary and denotes the Frobenius-norm of a given data matrix. Different from Eq.(2), Eq.(3) calculates the low rank representation using a clean dictionary instead of the original data . Moreover, Eq.(3) has a closed-form solution. In this paper, we mainly investigate this formulation of NNR.

Another popular representation is based on -norm or its induced matrix norm (i.e., the Frobenius norm). The basic formulation of FNR is as follows:

(4)

In our previous work [25], we have shown that the optimal solution to Eq.(4) is also the lowest rank solution, i.e.,

Theorem 1 ([25]).

Assume and has feasible solution(s), i.e., . Then

(5)

is the unique minimizer to Eq.(4), where is the pseudo-inverse of .

Considering nuclear norm based minimization problem, Liu et al. [9] have shown that

Theorem 2 ([9]).

Assume and has feasible solution(s), i.e., . Then

(6)

is the unique minimizer to

(7)

where is the pseudo-inverse of .

Theorems 1 and 2 actually imply the equivalence between NNR and FNR when the dictionary can exactly reconstruct inputs and the data set is immune from corruptions. In this paper, we will further investigate the connections between NNR and FNR by considering more complex situations, e.g., the data set is corrupted by Gaussian noise.

Objective Function or or
Nil Nil
Nil Nil
Nil
Nil
TABLE I: Connections between nuclear norm () and Frobenius norm () based representation in the case of the noise-free and the Gaussian noise situations, where denotes the th singular value of . is the full SVD of the dictionary and .
Objective Function or or
TABLE II: Connections between NNR and FNR in the case of the Laplacian noise and the sample-specified corruption. is the full SVD of , , is calculated using the augmented Lagrange multiplier method, and and are ALM parameters. Note that, the Laplacian noise and the sample-specified corruption will lead to different .

Iii Connections Between Nuclear Norm and Frobenius Norm Based Representation

For a data matrix , let and be the full SVD and skinny SVD of , where and are in descending order and denotes the rank of . , and consist of the top (i.e., largest)

singular vectors and singular values of

. Similar to [4, 7, 9, 10, 11], we assume , where denotes the clean data set and denotes the errors.

Our theoretical results will show that the optimal solutions of Frobenius-norm and nuclear-norm based objective functions are in the form of , where denotes the shrinkage-thresholding operator. In other words, FNR and NNR are two solutions on the column space of and they are identical in some situations. This provides a unified framework to understand FNR and NNR. The analysis will be performed considering several popular cases including exact/relax constraint and non-corrupted/corrupted data. When the dictionary has enough representative capacity, the objective function can be formulated with the exact constraint. Otherwise, the objective function is with the relax constraint. Noticed that, the exact constraint is considerably mild since most of data sets can be naturally reconstructed by itself in practice. With the exact constraint, many methods [10, 11] have been proposed and shown competitive performance comparing with the relax case. Besides the situation of noise-free, we will also investigate the connections between FNR and NNR when the data set contains the Gaussian noise, the Laplacian noise, or sample-specified corruption. Fig. 1, Tables I and II summary our results. Noticed that, in another independent work [26], Pan et al. proposed a subspace clustering method based on Frobenius norm and and reported some similar conclusions with this work. Different from this work, we mainly devote to build the theoretical connections between NNR and FNR involving different settings rather than developing new algorithm.

Fig. 1: An overview of the connections between FNR () and NNR (), where can be chosen as -, -, and -norm corresponding to the Laplacian noise, Gaussian noise, and outliers, respectively. With the relax constraint, the major difference between NNR and FNR is the value of . More details are summarized in Tables I and II.

Iii-a Exact Constraint and Uncorrupted Data

In the following analysis, we mainly focus on the case of self-expression because almost all works on NNR are carried out under such settings.

When the data set is uncorrupted and the dictionary has enough representative capacity, Liu et al. [9] have shown that:

Corollary 1 ([9]).

Assume and have feasible solution(s), i.e., . Then

(8)

is the unique minimizer to

(9)

where is the skinny SVD of .

Considering the Frobenius norm, we can obtain the following result:

Corollary 2.

Let be the skinny SVD of the data matrix . The unique solution to

(10)

is given by , where is the rank of and denotes a given data set without corruptions.

Proof.

Let be the skinny SVD of . The pseudo-inverse of is . By Theorem 1, we obtain , as desired. ∎

From Corollaries 1 and 2, ones can find that NNR and FNR have the same optimal solution . This solution is also known as the shape interaction matrix [27].

Iii-B Exact Constraint and Data Corrupted by Gaussian Noise

When the data set contains Gaussian noises (i.e., and is characterized by the Frobenius norm), we prove that

(11)

and

(12)

have the same minimizer in the form of , where is a parameter. By a simple transformation, we have the following results.

Theorem 3 ([10]).

Let be the SVD of the data matrix . The optimal solution to

(13)

is given by , where , , and correspond to the top singular values and singular vectors of , respectively.

Theorem 4.

Let be the full SVD of , where the diagonal entries of are in descending order, and are the left and right singular vectors of , respectively. Suppose there exists a clean data set and errors, denoted by and , respectively. The optimal to

(14)

is given by

(15)

where the operator performs hard thresholding on the diagonal entries of by

(16)

is a balanced parameter, , and denotes the -th diagonal entry of . i.e., consists of the first column vectors of .

Proof.

Let be the optimal solution to Eq.(14) and its skinny SVD be , where is the rank of . Let and be the basis that orthogonal to and , respectively. Clearly, . By Corollary 1, we have . Next, we will bridge and .

Use the method of Lagrange multipliers, we obtain

(17)

where is the Lagrange multiplier.
Letting , it gives that

(18)

Letting , it gives that

(19)

Thus, must be in the form of for some . Substituting this into (18), it given that

(20)

Then, we have . Since , is minimized when is a diagonal matrix and can be chosen as . Then, . Consequently, the SVD of can be rewritten as

(21)

Thus, the minimal cost of (14) is given by

where is the -th largest singular value of . Let be the optimal , then, . ∎

From Theorems 3 and 4, ones can find that the values of are slightly different. However, such difference cannot affect the equivalence conclusion because depends on the user-specified parameter . Moreover, this difference actually results from the constant term in our objective function.

Iii-C Relaxed Constraint and Uncorrupted Data

In this section, we discuss the connections between FNR and NNR when the dictionary is uncorrupted and has limited representative capacity. The objective functions are

(22)

and

(23)

In a lot works such as [22, 18], (22) is minimized at . In this paper, we will give another form of the solution to (22

) and the new solution is performing shrinkage operation on the right eigenvectors of

, like NNR.

Theorem 5 ([10]).

Let be the SVD of a given matrix . The optimal solution to

(24)

is

(25)

where , , and are partitioned according to the sets and .

Theorem 6.

Let be the full SVD of , where the diagonal entries of are in descending order, and are corresponding left and right singular vectors, respectively. The optimal to

(26)

is given by

(27)

where is a balanced factor and the operator performs shrinkage-thresholding on the diagonal entries of by

(28)

and is the rank of and denotes the th diagonal entry of .

Proof.

Letting denote the loss, and then we have

(29)

Next, we will show that (27) is the minimizer of since . Letting and substituting (27) into (29), we have

(30)

Let and be mutually orthogonal, then . Moreover, let the skinny SVD of be , we obtain

(31)

as desired. ∎

Iii-D Relax Constraint and Data Corrupted by Gaussian Noise

Suppose the data set is corrupted by and has limited representative capacity, the problems can be formulated as follows:

(32)

and

(33)
Theorem 7 ([10]).

Let be the SVD of the data matrix . The optimal solution to

(34)

is given by

(35)

where each entry of is obtained from one entry of as the solution to

(36)

that minimizes the cost, and the matrices , , and are partitioned according to the sets and .

Theorem 8.

Let be the skinny SVD of , where denotes the rank of and the diagonal entries of is in descending order. The optimal solutions to

(37)

are given by

(38)

and

(39)

where and are the diagonal entries on and , respectively.

(40)
Proof.

Letting denote the loss, we have

(41)

Next, we will bridge and . Let be the skinny SVD of . From Theorem 4, we have . Substituting this into (41) and letting , we obtain

(42)

which is a valid SVD of . ∎

Theorems 58 establish the relationships between FNR and NNR in the case of the limited representative capacity. Although FNR and NNR are not identical in such settings, they can be unified into a framework, i.e., both two methods obtain a solution from the column space of . The major difference between them is the adopted scaling factor. Moreover, NNR and FNR will truncates the trivial entries of coefficients in the case of uncorrupted data. With respect to corrupted case, two methods only scales the self-expressive coefficients by performing shrinkage.

Iii-E Exact Constraint and Data Corrupted by Laplacian Noise

The above analysis are based on the noise-free or the Gaussian noise assumptions. In this section, we investigate the Laplacian noise situation with the exact constraint. More specifically, we will prove that the optimal solutions to

(43)

and

(44)

have the same form.

As is unknown and -norm has no closed-form solution, we can solve Eqs.(43) and (44) using the augmented Lagrange multiplier method (ALM) [28].

Proposition 1 ([10]).

The optimal solution to Eq.(43) is given by

(45)

where , consists of the first column vectors of , and is iteratively computed via the following updated rules:

(46)
(47)
(48)
(49)
(50)

where is the learning rate of ALM and is a shrinkage-thresholding operator

(51)
Proposition 2.

The optimal solution to Eq.(44) is given by

(52)

where consists of the first column vectors of , and the updated rules are

(53)
(54)
(55)
(56)
(57)
Proof.

Using the augmented Lagrangian formulation, Eq.(44) can be rewritten as

(58)

By fixing others, we obtain by solving

(59)

According to Theorem 4, the optimal solutions to Eq.(59) is given by and , where consists of the first right singular vectors of , . Therefore, the optimal solutions to Eq.(44) can be iteratively computed via Eqs.(53)–(57). ∎

From Propositions 1–2, ones can find that the updated rules of NNR and FNR are identical under the framework of ALM. This would lead to the same minimizer to NNR and FNR.

Iii-F Exact Constraint and Data Corrupted by Sample-specified Noise

Besides the Gaussian noise and the Laplacian noise, we investigate sample-specified corruptions such as outliers [9, 29, 15] by adopting the norm. The formulations are as follows:

(60)

and

(61)

Similar to Propositions 1 and 2, it is easy to show that the optimal solutions to Eqs.(60)–(61) can be calculated via

(62)
(63)
(64)
(65)
(66)

where the operator is defined on the column of , i.e.,

(67)

where denotes the th column of .

Thus, ones can find that the optimal solutions of FNR and NNR are with the same form. The only one difference between them is the value of , i.e., for NNR and for FNR.

With respect to the relax constraint, ones can also establish the connections between FNR and NNR considering the Laplacian noise and sample-specified corruption. The analysis will be based on Theorem 4 and the form of is similar to the case of the exact constraint.

Iv Discussions

In this section, we first give the computational complexity analysis for FNR in different settings and then discuss the advantages of FNR over NNR in application scenario.

The above analysis shows that FNR and NNR are with the same form of solution. Thus, we can easily conclude that their computational complexity are the same under the same setting. More specifically, 1) when the input is free to corruption or contaminated by Gaussian noise, FNR and NNR will take to perform SVD on the input and then use to obtain the representation; 2) when the input contains Laplacian noise or sample-specified corruption, FNR and NNR will take to iteratively obtain the SVD of the input and then use to obtain the representation.

Our analysis explicitly gives the connections between NNR and FNR in theory. Thus, ones may hope to further understand them in the context of application scenario based on the theoretical analysis. Referring to experimental studies in existing works, we could conclude that: 1) for face recognition task, FNR would be more competitive since it could achieve comparable performance with over hundred times speedup as shown in [16, 17, 18, 30]

; 2) when dictionary can exactly reconstruct the input, both our theoretical and experimental analysis show that FNR and NNR perform comparable in feature extraction 

[20], image clustering and motion segmentation [10, 11]; 3) otherwise, FNR is better than NNR for feature extraction [19], image clustering and motion segmentation [23, 22, 9].

V Conclusion

In this paper, we investigated the connections between FNR and NNR in the case of the exact and the relax constraint. When the objective function is with the exact constraint, FNR is exactly NNR even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption. In the case of the relax constraint, FNR and NNR are two solutions on the column space of inputs. Under such a setting, the only one difference between FNR and NNR is the value of the thresholding parameter . Our theoretical results is complementary and a small step forward to existing compressive sensing. The major difference is that this work establishes the connections between the convex problem caused by -norm and the strictly convex problem caused by -norm in matrix space, while compressive sensing focuses on the equivalence between the non-convex problem caused by -norm and convex problem caused by -norm.

References

  • [1] D. L. Donoho and M. Elad, “Optimally sparse representation in general (nonorthogonal) dictionaries via minimization,” Proc. Natl. Acad. Sci., vol. 100, no. 5, pp. 2197–2202, 2003.
  • [2]

    J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. Huang, and S. Yan, “Sparse representation for computer vision and pattern recognition,”

    Proc. IEEE, vol. 98, no. 6, pp. 1031–1044, Jun. 2010.
  • [3] M. Aharon, M. Elad, and A. Bruckstein, “The K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE T. Signal Process., vol. 54, no. 11, pp. 4311–4322, Nov 2006.
  • [4] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210–227, 2009.
  • [5] X. Xu, Z. Hou, C. Lian, and H. He, “Online learning control using adaptive critic designs with sparse kernel machines,” IEEE T. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 762–775, May 2013.
  • [6] B. Cheng, J. Yang, S. Yan, Y. Fu, and T. Huang, “Learning with L1-graph for image analysis,” IEEE T. Image Process., vol. 19, no. 4, pp. 858–866, 2010.
  • [7] E. Elhamifar and R. Vidal, “Sparse subspace clustering: Algorithm, theory, and applications,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2765–2781, 2013.
  • [8] X. Peng, H. Tang, L. Zhang, Z. Yi, and S. Xiao, “A unified framework for representation-based subspace clustering of out-of-sample and large-scale data,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–14, 2015.
  • [9] G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by low-rank representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 171–184, 2013.
  • [10]

    P. Favaro, R. Vidal, and A. Ravichandran, “A closed form solution to robust subspace estimation and clustering,” in

    Proc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado Springs, CO, Jun. 2011, pp. 1801–1807.
  • [11] R. Vidal and P. Favaro, “Low rank subspace clustering (LRSC),” Pattern Recognit. Lett., vol. 43, no. 0, pp. 47 – 61, 2014.
  • [12] P. Sprechmann, A. Bronstein, and G. Sapiro, “Learning efficient sparse and low rank models,” IEEE T. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1821–1833, Sept 2015.
  • [13] S. Xiao, M. Tan, and D. Xu, “Robust kernel low rank representation,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–1, 2015.
  • [14] S. Xiao, D. Xu, and J. Wu, “Automatic face naming by learning discriminative affinity matrices from weakly labeled images,” IEEE T. Neural Netw. Learn. Syst., vol. 26, no. 10, pp. 2440–2452, Oct. 2015.
  • [15] J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, 2010.
  • [16]

    I. Naseem, R. Togneri, and M. Bennamoun, “Linear regression for face recognition,”

    IEEE T. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 2106–2112, Nov. 2010.
  • [17] Q. Shi, A. Eriksson, A. Van Den Hengel, and C. Shen, “Is face recognition really a compressive sensing problem?” in Proc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado, Springs, Jun. 2011, pp. 553–560.
  • [18] L. Zhang, M. Yang, and X. Feng, “Sparse representation or collaborative representation: Which helps face recognition?” in