Many problems in machine learning and computer vision begin with the processing of linearly inseparable data. The goal of processing is to distinct linearly inseparable data with linear methods. To achieve this, the inputs are always projected from the original space into another space. This is so-called representation learning and three methods have been extensively investigated in the community of computer vision,i.e., sparse representation (SR), low rank representation (LRR), and Frobenius-norm based representation (FNR).
During the past decade, sparse representation [1, 2] has been one of the most popular representation learning methods. It linearly reconstructs each sample using a few of basis and has shown the effectiveness in a lot of applications, e.g., image repairing 4], online learning control , dimension reduction , and subspace clustering [7, 8].
As another popular method, low rank representation [9, 10, 11, 12, 13, 14] has been proposed for subspace learning and subspace clustering. Different from SR, LRR computes the representation of a data set rather than a data point by solving a nuclear norm minimization problem. Thus, LRR is also known as nuclear norm based representation (NNR). Both LRR and SR benefit from the compressive sensing theory  which establishes the equivalence between - (rank-minimization w.r.t. matrix space) and -norm (nuclear-norm w.r.t. matrix space) based optimization problems. More specifically, compressive sensing provides the theoretical foundation to transform the non-convex problem caused by -norm into a convex problem using -norm.
Although more and more experimental evidences have been provided to show the effectiveness of FNR, the success of FNR is counter-intuitive as FNR is generally considered to be inferior to SR and NNR. Furthermore, fewer theoretical studies have been done to explore what makes FNR effective. Motivated by two NNR works [10, 11], this paper provides a novel theoretical explanation by bridging FNR and NNR. In other words, we show that under some mild conditions, the convex problem caused by nuclear norm can be converted to a strictly convex problem based on the Frobenius norm. More specifically, we prove that: 1) when the dictionary has enough representative capacity, FNR is equivalent to the NNR [10, 11] even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption; 2) when the dictionary has limited representative capacity, FNR and NNR are two solutions of the column space spanned by inputs. Our theoretical results unify FNR and NNR into a framework, i.e., FNR and NNR are in the form of , where
is the singular value decomposition (SVD) of a given data matrix anddenotes the shrinkage-thresholding operator. The difference between FNR and NNR lies in the different choices of the shrinkage-thresholding operator. To the best of our knowledge, this is one of the first several works to establish the connections between FNR and NNR.
For a given data set (each column denotes a data point), it can be decomposed as the linear combination of by
where denotes the constraint enforced over the representation . The main difference among most existing works is their objective functions, basically, the choice of . Different assumptions motivate different and this work focuses on the discussion of two popular objective functions, i.e., nuclear-norm and Frobenius-norm.
By assuming is low rank and the input contains noise, Liu et al.  propose solving the following nuclear norm based minimization problem:
where , is the th singular value of , and could be chosen as -, -, or Frobenius-norm.
-norm is usually adopted to depict the sample-specific corruptions such as outliers,-norm is used to characterize the Laplacian noise, and Frobenius norm is used to describe the Gaussian noise.
Although Eq.(2) can be easily solved by the Augmented Lagrangian method (ALM) , its computational complexity is still very high. Recently, Favaro and Vidal [10, 11] proposed a new formulation of LRR which can be calculated very fast. The proposed objective function is as follows:
where denotes the clean dictionary and denotes the Frobenius-norm of a given data matrix. Different from Eq.(2), Eq.(3) calculates the low rank representation using a clean dictionary instead of the original data . Moreover, Eq.(3) has a closed-form solution. In this paper, we mainly investigate this formulation of NNR.
Another popular representation is based on -norm or its induced matrix norm (i.e., the Frobenius norm). The basic formulation of FNR is as follows:
Theorem 1 ().
Assume and has feasible solution(s), i.e., . Then
is the unique minimizer to Eq.(4), where is the pseudo-inverse of .
Considering nuclear norm based minimization problem, Liu et al.  have shown that
Theorem 2 ().
Assume and has feasible solution(s), i.e., . Then
is the unique minimizer to
where is the pseudo-inverse of .
Theorems 1 and 2 actually imply the equivalence between NNR and FNR when the dictionary can exactly reconstruct inputs and the data set is immune from corruptions. In this paper, we will further investigate the connections between NNR and FNR by considering more complex situations, e.g., the data set is corrupted by Gaussian noise.
Iii Connections Between Nuclear Norm and Frobenius Norm Based Representation
For a data matrix , let and be the full SVD and skinny SVD of , where and are in descending order and denotes the rank of . , and consist of the top (i.e., largest)
singular vectors and singular values of. Similar to [4, 7, 9, 10, 11], we assume , where denotes the clean data set and denotes the errors.
Our theoretical results will show that the optimal solutions of Frobenius-norm and nuclear-norm based objective functions are in the form of , where denotes the shrinkage-thresholding operator. In other words, FNR and NNR are two solutions on the column space of and they are identical in some situations. This provides a unified framework to understand FNR and NNR. The analysis will be performed considering several popular cases including exact/relax constraint and non-corrupted/corrupted data. When the dictionary has enough representative capacity, the objective function can be formulated with the exact constraint. Otherwise, the objective function is with the relax constraint. Noticed that, the exact constraint is considerably mild since most of data sets can be naturally reconstructed by itself in practice. With the exact constraint, many methods [10, 11] have been proposed and shown competitive performance comparing with the relax case. Besides the situation of noise-free, we will also investigate the connections between FNR and NNR when the data set contains the Gaussian noise, the Laplacian noise, or sample-specified corruption. Fig. 1, Tables I and II summary our results. Noticed that, in another independent work , Pan et al. proposed a subspace clustering method based on Frobenius norm and and reported some similar conclusions with this work. Different from this work, we mainly devote to build the theoretical connections between NNR and FNR involving different settings rather than developing new algorithm.
Iii-a Exact Constraint and Uncorrupted Data
In the following analysis, we mainly focus on the case of self-expression because almost all works on NNR are carried out under such settings.
When the data set is uncorrupted and the dictionary has enough representative capacity, Liu et al.  have shown that:
Corollary 1 ().
Assume and have feasible solution(s), i.e., . Then
is the unique minimizer to
where is the skinny SVD of .
Considering the Frobenius norm, we can obtain the following result:
Let be the skinny SVD of the data matrix . The unique solution to
is given by , where is the rank of and denotes a given data set without corruptions.
Let be the skinny SVD of . The pseudo-inverse of is . By Theorem 1, we obtain , as desired. ∎
Iii-B Exact Constraint and Data Corrupted by Gaussian Noise
When the data set contains Gaussian noises (i.e., and is characterized by the Frobenius norm), we prove that
have the same minimizer in the form of , where is a parameter. By a simple transformation, we have the following results.
Theorem 3 ().
Let be the SVD of the data matrix . The optimal solution to
is given by , where , , and correspond to the top singular values and singular vectors of , respectively.
Let be the full SVD of , where the diagonal entries of are in descending order, and are the left and right singular vectors of , respectively. Suppose there exists a clean data set and errors, denoted by and , respectively. The optimal to
is given by
where the operator performs hard thresholding on the diagonal entries of by
is a balanced parameter, , and denotes the -th diagonal entry of . i.e., consists of the first column vectors of .
Let be the optimal solution to Eq.(14) and its skinny SVD be , where is the rank of . Let and be the basis that orthogonal to and , respectively. Clearly, . By Corollary 1, we have . Next, we will bridge and .
Use the method of Lagrange multipliers, we obtain
where is the Lagrange multiplier.
Letting , it gives that
Letting , it gives that
Thus, must be in the form of for some . Substituting this into (18), it given that
Then, we have . Since , is minimized when is a diagonal matrix and can be chosen as . Then, . Consequently, the SVD of can be rewritten as
Thus, the minimal cost of (14) is given by
where is the -th largest singular value of . Let be the optimal , then, . ∎
Iii-C Relaxed Constraint and Uncorrupted Data
In this section, we discuss the connections between FNR and NNR when the dictionary is uncorrupted and has limited representative capacity. The objective functions are
) and the new solution is performing shrinkage operation on the right eigenvectors of, like NNR.
Theorem 5 ().
Let be the SVD of a given matrix . The optimal solution to
where , , and are partitioned according to the sets and .
Let be the full SVD of , where the diagonal entries of are in descending order, and are corresponding left and right singular vectors, respectively. The optimal to
is given by
where is a balanced factor and the operator performs shrinkage-thresholding on the diagonal entries of by
and is the rank of and denotes the th diagonal entry of .
Iii-D Relax Constraint and Data Corrupted by Gaussian Noise
Suppose the data set is corrupted by and has limited representative capacity, the problems can be formulated as follows:
Theorem 7 ().
Let be the SVD of the data matrix . The optimal solution to
is given by
where each entry of is obtained from one entry of as the solution to
that minimizes the cost, and the matrices , , and are partitioned according to the sets and .
Let be the skinny SVD of , where denotes the rank of and the diagonal entries of is in descending order. The optimal solutions to
are given by
where and are the diagonal entries on and , respectively.
Letting denote the loss, we have
Theorems 5–8 establish the relationships between FNR and NNR in the case of the limited representative capacity. Although FNR and NNR are not identical in such settings, they can be unified into a framework, i.e., both two methods obtain a solution from the column space of . The major difference between them is the adopted scaling factor. Moreover, NNR and FNR will truncates the trivial entries of coefficients in the case of uncorrupted data. With respect to corrupted case, two methods only scales the self-expressive coefficients by performing shrinkage.
Iii-E Exact Constraint and Data Corrupted by Laplacian Noise
The above analysis are based on the noise-free or the Gaussian noise assumptions. In this section, we investigate the Laplacian noise situation with the exact constraint. More specifically, we will prove that the optimal solutions to
have the same form.
Proposition 1 ().
The optimal solution to Eq.(43) is given by
where , consists of the first column vectors of , and is iteratively computed via the following updated rules:
where is the learning rate of ALM and is a shrinkage-thresholding operator
The optimal solution to Eq.(44) is given by
where consists of the first column vectors of , and the updated rules are
Using the augmented Lagrangian formulation, Eq.(44) can be rewritten as
By fixing others, we obtain by solving
From Propositions 1–2, ones can find that the updated rules of NNR and FNR are identical under the framework of ALM. This would lead to the same minimizer to NNR and FNR.
Iii-F Exact Constraint and Data Corrupted by Sample-specified Noise
where the operator is defined on the column of , i.e.,
where denotes the th column of .
Thus, ones can find that the optimal solutions of FNR and NNR are with the same form. The only one difference between them is the value of , i.e., for NNR and for FNR.
With respect to the relax constraint, ones can also establish the connections between FNR and NNR considering the Laplacian noise and sample-specified corruption. The analysis will be based on Theorem 4 and the form of is similar to the case of the exact constraint.
In this section, we first give the computational complexity analysis for FNR in different settings and then discuss the advantages of FNR over NNR in application scenario.
The above analysis shows that FNR and NNR are with the same form of solution. Thus, we can easily conclude that their computational complexity are the same under the same setting. More specifically, 1) when the input is free to corruption or contaminated by Gaussian noise, FNR and NNR will take to perform SVD on the input and then use to obtain the representation; 2) when the input contains Laplacian noise or sample-specified corruption, FNR and NNR will take to iteratively obtain the SVD of the input and then use to obtain the representation.
Our analysis explicitly gives the connections between NNR and FNR in theory. Thus, ones may hope to further understand them in the context of application scenario based on the theoretical analysis. Referring to experimental studies in existing works, we could conclude that: 1) for face recognition task, FNR would be more competitive since it could achieve comparable performance with over hundred times speedup as shown in [16, 17, 18, 30]
; 2) when dictionary can exactly reconstruct the input, both our theoretical and experimental analysis show that FNR and NNR perform comparable in feature extraction, image clustering and motion segmentation [10, 11]; 3) otherwise, FNR is better than NNR for feature extraction , image clustering and motion segmentation [23, 22, 9].
In this paper, we investigated the connections between FNR and NNR in the case of the exact and the relax constraint. When the objective function is with the exact constraint, FNR is exactly NNR even though the data set contains the Gaussian noise, Laplacian noise, or sample-specified corruption. In the case of the relax constraint, FNR and NNR are two solutions on the column space of inputs. Under such a setting, the only one difference between FNR and NNR is the value of the thresholding parameter . Our theoretical results is complementary and a small step forward to existing compressive sensing. The major difference is that this work establishes the connections between the convex problem caused by -norm and the strictly convex problem caused by -norm in matrix space, while compressive sensing focuses on the equivalence between the non-convex problem caused by -norm and convex problem caused by -norm.
-  D. L. Donoho and M. Elad, “Optimally sparse representation in general (nonorthogonal) dictionaries via minimization,” Proc. Natl. Acad. Sci., vol. 100, no. 5, pp. 2197–2202, 2003.
J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. Huang, and S. Yan, “Sparse representation for computer vision and pattern recognition,”Proc. IEEE, vol. 98, no. 6, pp. 1031–1044, Jun. 2010.
-  M. Aharon, M. Elad, and A. Bruckstein, “The K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE T. Signal Process., vol. 54, no. 11, pp. 4311–4322, Nov 2006.
-  J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 210–227, 2009.
-  X. Xu, Z. Hou, C. Lian, and H. He, “Online learning control using adaptive critic designs with sparse kernel machines,” IEEE T. Neural Netw. Learn. Syst., vol. 24, no. 5, pp. 762–775, May 2013.
-  B. Cheng, J. Yang, S. Yan, Y. Fu, and T. Huang, “Learning with L1-graph for image analysis,” IEEE T. Image Process., vol. 19, no. 4, pp. 858–866, 2010.
-  E. Elhamifar and R. Vidal, “Sparse subspace clustering: Algorithm, theory, and applications,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2765–2781, 2013.
-  X. Peng, H. Tang, L. Zhang, Z. Yi, and S. Xiao, “A unified framework for representation-based subspace clustering of out-of-sample and large-scale data,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–14, 2015.
-  G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, “Robust recovery of subspace structures by low-rank representation,” IEEE T. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 171–184, 2013.
P. Favaro, R. Vidal, and A. Ravichandran, “A closed form solution to robust subspace estimation and clustering,” inProc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado Springs, CO, Jun. 2011, pp. 1801–1807.
-  R. Vidal and P. Favaro, “Low rank subspace clustering (LRSC),” Pattern Recognit. Lett., vol. 43, no. 0, pp. 47 – 61, 2014.
-  P. Sprechmann, A. Bronstein, and G. Sapiro, “Learning efficient sparse and low rank models,” IEEE T. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1821–1833, Sept 2015.
-  S. Xiao, M. Tan, and D. Xu, “Robust kernel low rank representation,” IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–1, 2015.
-  S. Xiao, D. Xu, and J. Wu, “Automatic face naming by learning discriminative affinity matrices from weakly labeled images,” IEEE T. Neural Netw. Learn. Syst., vol. 26, no. 10, pp. 2440–2452, Oct. 2015.
-  J.-F. Cai, E. J. Candès, and Z. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, 2010.
I. Naseem, R. Togneri, and M. Bennamoun, “Linear regression for face recognition,”IEEE T. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 2106–2112, Nov. 2010.
-  Q. Shi, A. Eriksson, A. Van Den Hengel, and C. Shen, “Is face recognition really a compressive sensing problem?” in Proc. of 24th IEEE Conf. Comput. Vis. and Pattern Recognit., Colorado, Springs, Jun. 2011, pp. 553–560.
-  L. Zhang, M. Yang, and X. Feng, “Sparse representation or collaborative representation: Which helps face recognition?” in Proc. the 13th Int. Conf. on Comput. Vis., Barcelona, Spain, Nov. 2011, pp. 471–478.
-  X. Peng, Z. Yu, Z. Yi, and H. Tang, “Constructing the l2-graph for robust subspace learning and subspace clustering,” IEEE T. Cybern., vol. PP, no. 99, pp. 1–14, 2016.
-  X. Peng, J. Lu, Z. Yi, and R. Yan, “Automatic subspace learning via principal coefficients embedding,” IEEE T. Cybern., vol. PP, no. 99, pp. 1–14, 2016.
X. Xu, Z. Huang, L. Zuo, and H. He, “Manifold-based reinforcement learning via locally linear reconstruction,”IEEE T. Neural Netw. Learn. Syst., vol. PP, no. 99, pp. 1–14, 2016.
-  C.-Y. Lu, H. Min, Z.-Q. Zhao, L. Zhu, D.-S. Huang, and S. Yan, “Robust and efficient subspace segmentation via least squares regression,” in Proc. of 12th Eur. Conf. Computer Vis., Firenze, Italy, Oct. 2012, pp. 347–360.
X. Peng, Z. Yi, and H. Tang, “Robust subspace clustering vi