I Introduction
Many problems in machine learning and computer vision begin with the processing of linearly inseparable data. The goal of processing is to distinct linearly inseparable data with linear methods. To achieve this, the inputs are always projected from the original space into another space. This is socalled representation learning and three methods have been extensively investigated in the community of computer vision,
i.e., sparse representation (SR), low rank representation (LRR), and Frobeniusnorm based representation (FNR).During the past decade, sparse representation [1, 2] has been one of the most popular representation learning methods. It linearly reconstructs each sample using a few of basis and has shown the effectiveness in a lot of applications, e.g., image repairing [3]
[4], online learning control [5], dimension reduction [6], and subspace clustering [7, 8].As another popular method, low rank representation [9, 10, 11, 12, 13, 14] has been proposed for subspace learning and subspace clustering. Different from SR, LRR computes the representation of a data set rather than a data point by solving a nuclear norm minimization problem. Thus, LRR is also known as nuclear norm based representation (NNR). Both LRR and SR benefit from the compressive sensing theory [15] which establishes the equivalence between  (rankminimization w.r.t. matrix space) and norm (nuclearnorm w.r.t. matrix space) based optimization problems. More specifically, compressive sensing provides the theoretical foundation to transform the nonconvex problem caused by norm into a convex problem using norm.
Recently, several works have shown that the Frobenius norm based representation (FNR) is competitive to SR and NNR in face recognition [16, 17, 18], subspace learning [19, 20]
[21], and subspace clustering [22, 23]. The advantage of FNR is that the objective only involves a strictly convex problem and thus the trap of local minimal is avoided.Although more and more experimental evidences have been provided to show the effectiveness of FNR, the success of FNR is counterintuitive as FNR is generally considered to be inferior to SR and NNR. Furthermore, fewer theoretical studies have been done to explore what makes FNR effective. Motivated by two NNR works [10, 11], this paper provides a novel theoretical explanation by bridging FNR and NNR. In other words, we show that under some mild conditions, the convex problem caused by nuclear norm can be converted to a strictly convex problem based on the Frobenius norm. More specifically, we prove that: 1) when the dictionary has enough representative capacity, FNR is equivalent to the NNR [10, 11] even though the data set contains the Gaussian noise, Laplacian noise, or samplespecified corruption; 2) when the dictionary has limited representative capacity, FNR and NNR are two solutions of the column space spanned by inputs. Our theoretical results unify FNR and NNR into a framework, i.e., FNR and NNR are in the form of , where
is the singular value decomposition (SVD) of a given data matrix and
denotes the shrinkagethresholding operator. The difference between FNR and NNR lies in the different choices of the shrinkagethresholding operator. To the best of our knowledge, this is one of the first several works to establish the connections between FNR and NNR.Ii Background
For a given data set (each column denotes a data point), it can be decomposed as the linear combination of by
(1) 
where denotes the constraint enforced over the representation . The main difference among most existing works is their objective functions, basically, the choice of . Different assumptions motivate different and this work focuses on the discussion of two popular objective functions, i.e., nuclearnorm and Frobeniusnorm.
By assuming is low rank and the input contains noise, Liu et al. [9] propose solving the following nuclear norm based minimization problem:
(2) 
where , is the th singular value of , and could be chosen as , , or Frobeniusnorm.
norm is usually adopted to depict the samplespecific corruptions such as outliers,
norm is used to characterize the Laplacian noise, and Frobenius norm is used to describe the Gaussian noise.Although Eq.(2) can be easily solved by the Augmented Lagrangian method (ALM) [24], its computational complexity is still very high. Recently, Favaro and Vidal [10, 11] proposed a new formulation of LRR which can be calculated very fast. The proposed objective function is as follows:
(3) 
where denotes the clean dictionary and denotes the Frobeniusnorm of a given data matrix. Different from Eq.(2), Eq.(3) calculates the low rank representation using a clean dictionary instead of the original data . Moreover, Eq.(3) has a closedform solution. In this paper, we mainly investigate this formulation of NNR.
Another popular representation is based on norm or its induced matrix norm (i.e., the Frobenius norm). The basic formulation of FNR is as follows:
(4) 
In our previous work [25], we have shown that the optimal solution to Eq.(4) is also the lowest rank solution, i.e.,
Theorem 1 ([25]).
Assume and has feasible solution(s), i.e., . Then
(5) 
is the unique minimizer to Eq.(4), where is the pseudoinverse of .
Considering nuclear norm based minimization problem, Liu et al. [9] have shown that
Theorem 2 ([9]).
Assume and has feasible solution(s), i.e., . Then
(6) 
is the unique minimizer to
(7) 
where is the pseudoinverse of .
Theorems 1 and 2 actually imply the equivalence between NNR and FNR when the dictionary can exactly reconstruct inputs and the data set is immune from corruptions. In this paper, we will further investigate the connections between NNR and FNR by considering more complex situations, e.g., the data set is corrupted by Gaussian noise.
Objective Function  or  or  

Nil  Nil  
Nil  Nil  
Nil  
Nil  
Objective Function  or  or  

Iii Connections Between Nuclear Norm and Frobenius Norm Based Representation
For a data matrix , let and be the full SVD and skinny SVD of , where and are in descending order and denotes the rank of . , and consist of the top (i.e., largest)
singular vectors and singular values of
. Similar to [4, 7, 9, 10, 11], we assume , where denotes the clean data set and denotes the errors.Our theoretical results will show that the optimal solutions of Frobeniusnorm and nuclearnorm based objective functions are in the form of , where denotes the shrinkagethresholding operator. In other words, FNR and NNR are two solutions on the column space of and they are identical in some situations. This provides a unified framework to understand FNR and NNR. The analysis will be performed considering several popular cases including exact/relax constraint and noncorrupted/corrupted data. When the dictionary has enough representative capacity, the objective function can be formulated with the exact constraint. Otherwise, the objective function is with the relax constraint. Noticed that, the exact constraint is considerably mild since most of data sets can be naturally reconstructed by itself in practice. With the exact constraint, many methods [10, 11] have been proposed and shown competitive performance comparing with the relax case. Besides the situation of noisefree, we will also investigate the connections between FNR and NNR when the data set contains the Gaussian noise, the Laplacian noise, or samplespecified corruption. Fig. 1, Tables I and II summary our results. Noticed that, in another independent work [26], Pan et al. proposed a subspace clustering method based on Frobenius norm and and reported some similar conclusions with this work. Different from this work, we mainly devote to build the theoretical connections between NNR and FNR involving different settings rather than developing new algorithm.
Iiia Exact Constraint and Uncorrupted Data
In the following analysis, we mainly focus on the case of selfexpression because almost all works on NNR are carried out under such settings.
When the data set is uncorrupted and the dictionary has enough representative capacity, Liu et al. [9] have shown that:
Corollary 1 ([9]).
Assume and have feasible solution(s), i.e., . Then
(8) 
is the unique minimizer to
(9) 
where is the skinny SVD of .
Considering the Frobenius norm, we can obtain the following result:
Corollary 2.
Let be the skinny SVD of the data matrix . The unique solution to
(10) 
is given by , where is the rank of and denotes a given data set without corruptions.
Proof.
Let be the skinny SVD of . The pseudoinverse of is . By Theorem 1, we obtain , as desired. ∎
IiiB Exact Constraint and Data Corrupted by Gaussian Noise
When the data set contains Gaussian noises (i.e., and is characterized by the Frobenius norm), we prove that
(11) 
and
(12) 
have the same minimizer in the form of , where is a parameter. By a simple transformation, we have the following results.
Theorem 3 ([10]).
Let be the SVD of the data matrix . The optimal solution to
(13) 
is given by , where , , and correspond to the top singular values and singular vectors of , respectively.
Theorem 4.
Let be the full SVD of , where the diagonal entries of are in descending order, and are the left and right singular vectors of , respectively. Suppose there exists a clean data set and errors, denoted by and , respectively. The optimal to
(14) 
is given by
(15) 
where the operator performs hard thresholding on the diagonal entries of by
(16) 
is a balanced parameter, , and denotes the th diagonal entry of . i.e., consists of the first column vectors of .
Proof.
Let be the optimal solution to Eq.(14) and its skinny SVD be , where is the rank of . Let and be the basis that orthogonal to and , respectively. Clearly, . By Corollary 1, we have . Next, we will bridge and .
Use the method of Lagrange multipliers, we obtain
(17) 
where is the Lagrange multiplier.
Letting , it gives that
(18) 
Letting , it gives that
(19) 
Thus, must be in the form of for some . Substituting this into (18), it given that
(20) 
Then, we have . Since , is minimized when is a diagonal matrix and can be chosen as . Then, . Consequently, the SVD of can be rewritten as
(21) 
Thus, the minimal cost of (14) is given by
where is the th largest singular value of . Let be the optimal , then, . ∎
IiiC Relaxed Constraint and Uncorrupted Data
In this section, we discuss the connections between FNR and NNR when the dictionary is uncorrupted and has limited representative capacity. The objective functions are
(22) 
and
(23) 
In a lot works such as [22, 18], (22) is minimized at . In this paper, we will give another form of the solution to (22
) and the new solution is performing shrinkage operation on the right eigenvectors of
, like NNR.Theorem 5 ([10]).
Let be the SVD of a given matrix . The optimal solution to
(24) 
is
(25) 
where , , and are partitioned according to the sets and .
Theorem 6.
Let be the full SVD of , where the diagonal entries of are in descending order, and are corresponding left and right singular vectors, respectively. The optimal to
(26) 
is given by
(27) 
where is a balanced factor and the operator performs shrinkagethresholding on the diagonal entries of by
(28) 
and is the rank of and denotes the th diagonal entry of .
IiiD Relax Constraint and Data Corrupted by Gaussian Noise
Suppose the data set is corrupted by and has limited representative capacity, the problems can be formulated as follows:
(32) 
and
(33) 
Theorem 7 ([10]).
Let be the SVD of the data matrix . The optimal solution to
(34) 
is given by
(35) 
where each entry of is obtained from one entry of as the solution to
(36) 
that minimizes the cost, and the matrices , , and are partitioned according to the sets and .
Theorem 8.
Let be the skinny SVD of , where denotes the rank of and the diagonal entries of is in descending order. The optimal solutions to
(37) 
are given by
(38) 
and
(39) 
where and are the diagonal entries on and , respectively.
(40) 
Proof.
Letting denote the loss, we have
(41) 
Theorems 5–8 establish the relationships between FNR and NNR in the case of the limited representative capacity. Although FNR and NNR are not identical in such settings, they can be unified into a framework, i.e., both two methods obtain a solution from the column space of . The major difference between them is the adopted scaling factor. Moreover, NNR and FNR will truncates the trivial entries of coefficients in the case of uncorrupted data. With respect to corrupted case, two methods only scales the selfexpressive coefficients by performing shrinkage.
IiiE Exact Constraint and Data Corrupted by Laplacian Noise
The above analysis are based on the noisefree or the Gaussian noise assumptions. In this section, we investigate the Laplacian noise situation with the exact constraint. More specifically, we will prove that the optimal solutions to
(43) 
and
(44) 
have the same form.
As is unknown and norm has no closedform solution, we can solve Eqs.(43) and (44) using the augmented Lagrange multiplier method (ALM) [28].
Proposition 1 ([10]).
The optimal solution to Eq.(43) is given by
(45) 
where , consists of the first column vectors of , and is iteratively computed via the following updated rules:
(46) 
(47) 
(48) 
(49) 
(50) 
where is the learning rate of ALM and is a shrinkagethresholding operator
(51) 
Proposition 2.
The optimal solution to Eq.(44) is given by
(52) 
where consists of the first column vectors of , and the updated rules are
(53) 
(54) 
(55) 
(56) 
(57) 
Proof.
Using the augmented Lagrangian formulation, Eq.(44) can be rewritten as
(58) 
By fixing others, we obtain by solving
(59) 
From Propositions 1–2, ones can find that the updated rules of NNR and FNR are identical under the framework of ALM. This would lead to the same minimizer to NNR and FNR.
IiiF Exact Constraint and Data Corrupted by Samplespecified Noise
Besides the Gaussian noise and the Laplacian noise, we investigate samplespecified corruptions such as outliers [9, 29, 15] by adopting the norm. The formulations are as follows:
(60) 
and
(61) 
Similar to Propositions 1 and 2, it is easy to show that the optimal solutions to Eqs.(60)–(61) can be calculated via
(62) 
(63) 
(64) 
(65) 
(66) 
where the operator is defined on the column of , i.e.,
(67) 
where denotes the th column of .
Thus, ones can find that the optimal solutions of FNR and NNR are with the same form. The only one difference between them is the value of , i.e., for NNR and for FNR.
With respect to the relax constraint, ones can also establish the connections between FNR and NNR considering the Laplacian noise and samplespecified corruption. The analysis will be based on Theorem 4 and the form of is similar to the case of the exact constraint.
Iv Discussions
In this section, we first give the computational complexity analysis for FNR in different settings and then discuss the advantages of FNR over NNR in application scenario.
The above analysis shows that FNR and NNR are with the same form of solution. Thus, we can easily conclude that their computational complexity are the same under the same setting. More specifically, 1) when the input is free to corruption or contaminated by Gaussian noise, FNR and NNR will take to perform SVD on the input and then use to obtain the representation; 2) when the input contains Laplacian noise or samplespecified corruption, FNR and NNR will take to iteratively obtain the SVD of the input and then use to obtain the representation.
Our analysis explicitly gives the connections between NNR and FNR in theory. Thus, ones may hope to further understand them in the context of application scenario based on the theoretical analysis. Referring to experimental studies in existing works, we could conclude that: 1) for face recognition task, FNR would be more competitive since it could achieve comparable performance with over hundred times speedup as shown in [16, 17, 18, 30]
; 2) when dictionary can exactly reconstruct the input, both our theoretical and experimental analysis show that FNR and NNR perform comparable in feature extraction
[20], image clustering and motion segmentation [10, 11]; 3) otherwise, FNR is better than NNR for feature extraction [19], image clustering and motion segmentation [23, 22, 9].V Conclusion
In this paper, we investigated the connections between FNR and NNR in the case of the exact and the relax constraint. When the objective function is with the exact constraint, FNR is exactly NNR even though the data set contains the Gaussian noise, Laplacian noise, or samplespecified corruption. In the case of the relax constraint, FNR and NNR are two solutions on the column space of inputs. Under such a setting, the only one difference between FNR and NNR is the value of the thresholding parameter . Our theoretical results is complementary and a small step forward to existing compressive sensing. The major difference is that this work establishes the connections between the convex problem caused by norm and the strictly convex problem caused by norm in matrix space, while compressive sensing focuses on the equivalence between the nonconvex problem caused by norm and convex problem caused by norm.
References
 [1] D. L. Donoho and M. Elad, “Optimally sparse representation in general (nonorthogonal) dictionaries via minimization,” Proc. Natl. Acad. Sci., vol. 100, no. 5, pp. 2197–2202, 2003.

[2]
J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. Huang, and S. Yan, “Sparse representation for computer vision and pattern recognition,”
Proc. IEEE, vol. 98, no. 6, pp. 1031–1044, Jun. 2010.  [3] M. Aharon, M. Elad, and A. Bruckstein, “The KSVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE T. Signal Process., vol. 54, no. 11, pp. 4311–4322, Nov 2006.
 [4] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210–227, 2009.
 [5] X. Xu, Z. Hou, C. Lian, and H. He, “Online learning control using adaptive critic designs with sparse kernel machines,” IEEE T. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 762–775, May 2013.
 [6] B. Cheng, J. Yang, S. Yan, Y. Fu, and T. Huang, “Learning with L1graph for image analysis,” IEEE T. Image Process., vol. 19, no. 4, pp. 858–866, 2010.
 [7] E. Elhamifar and R. Vidal, “Sparse subspace clustering: Algorithm, theory, and applications,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2765–2781, 2013.
 [8] X. Peng, H. Tang, L. Zhang, Z. Yi, and S. Xiao, “A unified framework for representationbased subspace clustering of outofsample and largescale data,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–14, 2015.
 [9] G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by lowrank representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 171–184, 2013.

[10]
P. Favaro, R. Vidal, and A. Ravichandran, “A closed form solution to robust subspace estimation and clustering,” in
Proc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado Springs, CO, Jun. 2011, pp. 1801–1807.  [11] R. Vidal and P. Favaro, “Low rank subspace clustering (LRSC),” Pattern Recognit. Lett., vol. 43, no. 0, pp. 47 – 61, 2014.
 [12] P. Sprechmann, A. Bronstein, and G. Sapiro, “Learning efficient sparse and low rank models,” IEEE T. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1821–1833, Sept 2015.
 [13] S. Xiao, M. Tan, and D. Xu, “Robust kernel low rank representation,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–1, 2015.
 [14] S. Xiao, D. Xu, and J. Wu, “Automatic face naming by learning discriminative affinity matrices from weakly labeled images,” IEEE T. Neural Netw. Learn. Syst., vol. 26, no. 10, pp. 2440–2452, Oct. 2015.
 [15] J.F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, 2010.

[16]
I. Naseem, R. Togneri, and M. Bennamoun, “Linear regression for face recognition,”
IEEE T. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 2106–2112, Nov. 2010.  [17] Q. Shi, A. Eriksson, A. Van Den Hengel, and C. Shen, “Is face recognition really a compressive sensing problem?” in Proc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado, Springs, Jun. 2011, pp. 553–560.
 [18] L. Zhang, M. Yang, and X. Feng, “Sparse representation or collaborative representation: Which helps face recognition?” in Proc. the 13th Int. Conf. on Comput. Vis., Barcelona, Spain, Nov. 2011, pp. 471–478.
 [19] X. Peng, Z. Yu, Z. Yi, and H. Tang, “Constructing the l2graph for robust subspace learning and subspace clustering,” IEEE T. Cybern., vol. PP, no. 99, pp. 1–14, 2016.
 [20] X. Peng, J. Lu, Z. Yi, and R. Yan, “Automatic subspace learning via principal coefficients embedding,” IEEE T. Cybern., vol. PP, no. 99, pp. 1–14, 2016.

[21]
X. Xu, Z. Huang, L. Zuo, and H. He, “Manifoldbased reinforcement learning via locally linear reconstruction,”
IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–14, 2016.  [22] C.Y. Lu, H. Min, Z.Q. Zhao, L. Zhu, D.S. Huang, and S. Yan, “Robust and efficient subspace segmentation via least squares regression,” in Proc. of 12th Eur. Conf. Computer Vis., Firenze, Italy, Oct. 2012, pp. 347–360.

[23]
X. Peng, Z. Yi, and H. Tang, “Robust subspace clustering vi
Comments
There are no comments yet.