 # Condition numbers for the truncated total least squares problem and their estimations

In this paper, we present explicit expressions for the mixed and componentwise condition numbers of the truncated total least squares (TTLS) solution of Ax≈b under the genericity condition, where A is a m× n real data matrix and b is a real m-vector. Moreover, we reveal that normwise, componentwise and mixed condition numbers for the TTLS problem can recover the previous corresponding counterparts for the total least squares (TLS) problem when the truncated level of for the TTLS problem is n. When A is a structured matrix, the structured perturbations for the structured truncated TLS (STTLS) problem are investigated and the corresponding explicit expressions for the structured normwise, componentwise and mixed condition numbers for the STTLS problem are obtained. Furthermore, the relationships between the structured and unstructured normwise, componentwise and mixed condition numbers for the STTLS problem are studied. Based on small sample statistical condition estimation (SCE), reliable condition estimation algorithms for both unstructured and structured normwise, mixed and componentwise are devised, which utilize the SVD of the augmented matrix [A b ]. The efficient proposed condition estimation algorithms can be integrated into the SVD-based direct solver for the small and medium size TTLS problem to give the error estimation for the numerical TTLS solution. Numerical experiments are reported to illustrate the reliability of the proposed estimation algorithms, which coincide with our theoretical results.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1. Introduction

In this paper, we consider the following linear model

 Ax≈b, (1.1)

where the data matrix and the observation vector are both perturbed. When , the linear model (1.1) is overdetermined. To find a solution to (1.1), one may solve the following minimization problem:

 min∥∥[ΔA Δb]∥∥Fsubject to (s.t.)(A+ΔA)x=b+Δb, (1.2)

where means the Frobenius matrix norm. This is the classical total least square (TLS) problem, which was originally proposed by Golub and Van Loan [9, 11].

The TLS problem is often used for the linear model (1.1) when the augmented matrix is rank deficient, i.e., the small singular values of are assumed to be separated from the others. More interestingly, the truncated total least square (TTLS) method aims to solve the linear model (1.1) in the sense that the small singular values of are set to be zeros. For the discussion of the TTLS, one may refer to [16, §3.6.1] and [6, 7]

. The TTLS problem arises in various applications such as linear system theory, computer vision, image reconstruction, system identification, speech and audio processing, modal and spectral analysis, and astronomy, etc. The overview of the TTLS can be found in

.

Let be the predefined truncated level, where . The TTLS problem aims to solve the following problem:

 xk=argmin∥x∥2, subject to Akx=bk, (1.3)

where denotes the Euclidean vector norm or its induced matrix norm and is the best rank- approximation of in the Frobenius norm.

In order to solve (1.3), we first recall the singular value decomposition (SVD) of which is given by

 [A b]=UΣV⊤, (1.4)

where and are orthogonal matrices and is a real matrix with a vector on its diagonal and , where . Here, is a diagonal matrix with a vector on its diagonal and the superscript “” takes the transpose of a matrix or vector. Then we have , where . Suppose the truncation level satisfies the condition

 σk>σk+1. (1.5)

Define

 V=[V11V12V21V22],V11∈Rn×k. (1.6)

If

 V22≠0, (1.7)

then the TTLS problem is generic and the TTLS solution is given by 

 xk=−1∥V22∥22V12V⊤22. (1.8)

Condition numbers measure the worst-case sensitivity of an input data to small perturbations. Normwise condition numbers for the TLS problem (1.2) under the genericity condition were studied in [1, 17], where SVD-base explicit formulas for normwise condition numbers were derived. The normwise condition number of the truncated SVD solution to a linear model as (1.1) was introduced in . When the data is sparse or badly scaled, it is more suitable to consider the componentwise perturbations since normwise perturbations only measure the perturbation for the data by means of norm and may ignore the relative size of the perturbation on its small (or zero) entries (cf. ). There are two types of condition numbers in componentwise perturbations. The mixed condition number measures the errors in the output using norms and the input perturbations componentwise, while the componentwise condition number measures both the error in the output and the perturbation in the input componentwise (cf. ). The Kronecker product based formulas for the mixed and componentwise condition numbers to the TLS problem (1.2) were derived in [30, 4]. The corresponding componentwise perturbation analysis for the multidimensional TLS problem and mixed least squares-TLS problem can be found in [28, 29].

Gratton et al. in  investigated the normwise condition number for the TTLS problem (1.3). The normwise condition number formula and its easier computable upper bounds for the TTLS solution (1.3) were derived (cf. [14, Theorems 2.4-2.6]), which rely on the SVD of the augmented matrix . Since the normwise condition number formula for the TTLS problem (1.3) involves Kronecker product, which is not easy to compute or evaluate even for the medium size TTLS problem, the condition estimation method based on the power method  or the Golub-Kahan-Lanczos (GKL) bidiagonalization algorithm  was proposed to estimate the spectral norm of Fréchet derivative matrix related to (1.3).

As mentioned before, when the TTLS problem (1.3) is sparse or badly scaled, which often occurs in scientific computing, the conditioning based on normwise perturbation analysis may severely overestimate the exact error of the numerical solution to (1.3). Indeed, from the numerical results for Example 5.1 in Section 5, the TTLS problem (1.3) with respect to the specific data and is well-conditioned under componentwise perturbation analysis while it is very ill-conditioned under normwise perturbation, which implies that the normwise relative errors for the numerical solution to (1.3) are pessimistic. In this paper, we propose the mixed and componentwise condition number for the TTLS problem (1.3) and the corresponding explicit expressions are derived, which can capture the exact conditioning of (1.3) with respect to the sparsity and scaling for the input data. Furthermore, when the truncated level in (1.3) is selected to be , (1.3) reduces to (1.2). The normwised, mixed and componentwise condition numbers for the TTLS problem (1.3) are shown to be mathematically equivalent to the corresponding ones [1, 17, 30] for the untruncated case from their explicit expressions.

Structured TLS problems [18, 23, 26] had been studied extensively in the past decades. For structured TLS problems, it is suitable to investigate structured perturbations on the input data, because structure-preserving algorithms that preserve the underlying matrix structure can enhance the accuracy and efficiency of the TLS solution computation. Structured condition numbers for structured TLS problems can be found in [24, 4, 5] and references therein. In this paper, we introduce structured perturbation analysis for the structured TTLS (STTLS) problem. The explicit structured normwise, mixed and componentwise condition numbers for the STTLS problem are obtained, and their relationships corresponding to the unstructured ones are investigated.

The Kronecker product based expressions for both unstructured and structured normwise, mixed and componentwise condition numbers of the TTLS solution involve higher dimensions thus prevent the efficient calculations of these condition numbers. In practice, it is important to estimate condition numbers efficiently since the forward error for the numerical solution can be obtained via combining condition numbers with backward errors. In this paper, based on the small sample statistical condition estimation (SCE) , we propose reliable condition estimation algorithms for both unstructured and structured normwise, mixed and componentwise condition numbers of the TTLS solution, which utilize the SVD of to reduce the computational cost. Furthermore, the proposed condition estimation algorithms can be integrated into the SVD-based direct solver for the small or medium size TTLS problem (1.3). Therefore, one can obtain the reliable forward error estimations for the numerical TTLS solution after implementing the proposed condition estimation algorithms. The main computational cost in condition number estimations for (1.3) is to evaluate the directional derivatives with respect to the generated direction during the loops in condition number estimations algorithms. We point out that the power method  for estimating the normwise condition number in  needs to evaluate the directional derivatives twice in one loop. However, only evaluating direction derivative once is needed in the loop of Algorithms 1 to 3. Therefore, compared with the normwise condition number estimation algorithm proposed in , our proposed condition number estimations algorithms in this paper are more efficient in terms of the computational complexity, which are also applicable for estimating the componentwise and structured perturbations for (1.3). For recent SCE’s developments for (structured) linear systems, linear least squares and TLS problem, we refer to [21, 20, 22, 5] and references therein.

The rest of this paper is organized as follows. In Section 2 we review pervious perturbation results on the TTLS problem and derive explicit expressions of the mixed and componentwise condition numbers. The structured normwise, mixed and componentwise condition numbers are also investigated in Section 2, where the relationships between the unstructured normwise, mixed and componentwise condition numbers for (1.3) with the corresponding structured counterparts are investigated. In Section 4 we are devoted to propose several condition estimation algorithms for the normwise, mixed and componentwise condition numbers of the TTLS problem. Moreover, the structured condition estimation is considered. In Section 5, numerical examples are shown to illustrate the efficiency and reliability of the proposed algorithms and report the perturbation bounds based on the proposed condition number. Finally, some concluding remarks are drawn in the last section.

## 2. Condition numbers for the TTLS problem

In this section we review previous perturbation results on the TTLS problem. The explicit expressions of the mixed and componentwise condition numbers for the TTLS problem are derived. Furthermore, for the structured TTLS problem, we propose the normwise, mixed and componentwise condition numbers, where explicit formulas for the corresponding counterparts are obtain. The relationships between the unstructured normwise, mixed and componentwise condition numbers for (1.3) with the corresponding structured counterparts are investigated. We first introduce some conventional notations in the subsequent paper.

Throughout this paper, we use the following notation. Let be the vector -norm or its induced matrix norm. Let

be the identity matrix of order

. Let be the -th column vector of an identity matrix of an appropriate dimension. The superscript “” means the Moore-Penrose inverse of a matrix. The symbol “” means componentwise multiplication of two conformal dimensional matrices. For any matrix , let , where denote the absolute value of . For any two matrices , represents for all and . For any , we define by

 zi=⎧⎪⎨⎪⎩xi/yi,if yi≠0,0,if xi=yi=0,∞,otherwise.

Let be a column vector obtained by stacking the columns of on top of one another. For a vector , let , where for and . The symbol “” means the Kronecker product and is a permutation matrix defined by

 vec(B⊤)=Πm,nvec(B),∀B∈Rm×n. (2.1)

Given the matrices , , and , and with appropriate dimensions, we have the following propertes of the Kronecker product and vec operator :

 ⎧⎪⎨⎪⎩vec(XDY)=(Y⊤⊗X)vec(D),(X1⊗X2)(Y1⊗Y2)=(X1Y1)⊗(X2Y2),Πp,m(Y⊗X)=(X⊗Y)Πn,q. (2.2)

### 2.1. Preliminaries

In this subsection, we recall the definition of absolute normwise condition number of the TTLS solution defined by (1.3) (cf. ). The absolute normwise condition number of in (1.3) is defined by

 κ(A,b)=limϵ→0sup∥ΔH∥F≤ϵ∥∥ψk([A b]+ΔH)−ψk([A b])∥∥2∥ΔH∥F, (2.3)

where the function is given by

 ψk([A b]):Rm×n×Rm→Rn:[A b]↦xk. (2.4)

Let the SVD of be given by (1.4). If the truncation level satisfies the conditions (1.5) and (1.7), then the explicit expression of is given by [14, Theorem 2.4]

 κ(A,b)=∥Mk∥2, (2.5)

where

 Mk=1∥V22∥22[Inxk]VKD−1[Ik⊗Σ⊤2Σ1⊗In−k+1]W (2.6)

with

 Σ1 =diag([σ1,…,σk])∈Rk×k,Σ2=diag([σk+1,…,σp])∈R(m−k)×(n−k+1), σ1 ≥⋯≥σk>σk+1≥⋯≥σp≥0,k

Please be noted the the dimension of may be large even for medium size TTLS problems. The explicit formula given by (2.5) involves the computation of the spectral norm of . Hence, upper bounds for is obtain in [14, §2.4], which only rely on the singular values of and . When the data is sparse or badly scaled, the normwise condition number may not reveal the conditioning of (1.3), since normwise perturbations ignore the relative size of the perturbation on its small (or zero) entries. Therefore, it is more suitable to consider the componentwise perturbation analysis for (1.3). In the next subsection, we shall introduce the mixed and componentwise condition number for (1.3).

In [14, §2.3], if both and the full SVD of are available, then one may compute by using the power method [15, Chap. 15] to or the Golub-Kahan-Lanczos (GKL) bidiagonalization algorithm  to or , where only the matrix-vector product is needed. However, as pointed in the introduction part, the normwise condition number estimation algorithm in  are devised based on the power method , which needs to evaluate and in one loop for some suitable dimensional vectors and . In Section 4, SCE-based condition estimation algorithms for (1.3) shall be proposed, where in one loop we only need to compute the directional derivative and is not involved. Therefore, compared with normwise condition number estimation algorithm in , SCE-based condition estimation algorithms are more efficient.

### 2.2. Mixed and componentwise condition numbers

In Lemma 2.1 below, the first order perturbation expansion of with respect to the perturbations of the data and is reviewed, which involves the Kronecker product. In order to avoid the Kronecker product in the explicit expression for the directional derivative of , we derive the corresponding equivalent formula (2.7) in Lemma 2.2. Furthermore, (2.7) can be used to save computation memory of SCE-base condition estimation algorithms in Section 4.

###### Lemma 2.1.

[14, Theorem 2.4] Let the SVD of the augmented matrix be given by (1.4). Suppose is a truncation level such that and . If with sufficiently small, then, for the TTLS solution of and the TTLS solution of , we have

 ~xk=xk+Mkvec(ΔH)+O(∥ΔH∥2F).
###### Lemma 2.2.

Under the same assumptions as in Lemma 2.1, if with sufficiently small, then the directional derivative of at in the direction is given by

 ψ′k([A b];[ΔA Δb]) =1∥V22∥22(V11(Z⊤1+Z2)V⊤22+V12(Z1+Z⊤2)V⊤21+xk4∑j=1cj), (2.7)

where

 Z1 =(Σ⊤2U⊤2ΔHV1)⊡D∈R(n−k+1)×k,Z2=(Σ⊤1U⊤1ΔHV2)⊡D⊤∈Rk×(n−k+1), c1 =V21Z⊤1V⊤22,c2=V21Z2V⊤22,c3=V22Z1V⊤21,c4=V22Z⊤2V⊤21,

with

 D(:,i)=⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩⎡⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢⎣(σ2i−σ2k+1)−1⋮(σ2i−σ2m)−1σ−2i⋮σ−2i⎤⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥⎦∈R(n−k+1), if m

Proof.   From Lemma 2.1 we have

 ψ′k([A b];[ΔA Δb])=Mkvec(ΔH),

where is defined by (2.6). Using (2.2), it is easy to verify that

 [Ik⊗Σ⊤2Σ1⊗In−k+1]W =(Ik⊗Σ⊤2)(V⊤1⊗U⊤2)+(Σ⊤1⊗In−k+1)Πn−k+1,k(V⊤2⊗U⊤1) =V⊤1⊗(Σ⊤2U⊤2)+(Σ⊤1U⊤1⊗V⊤2)Πm,n+1, (2.9)

Using the fact that and (2.2) we have

 [Ik⊗Σ⊤2Σ1⊗In−k+1]Wvec(ΔH) (2.10) = (V⊤1⊗(Σ⊤2U⊤2))vec(ΔH)+(Σ⊤1U⊤1⊗V⊤2)Πm,n+1vec(ΔH) = vec(Σ⊤2U⊤2ΔHV1)+vec(V⊤2ΔH⊤U1Σ1).

From (2.6), we see that the -th diagonal block of is given by

 D(i)=⎧⎪⎨⎪⎩diag([σ2i−σ2k+1,…,σ2i−σ2m,σ2i,…,σ2i]⊤)∈R(n−k+1)×(n−k+1),if m

for . By the definition of we have

 {D−1vec(Σ⊤2U⊤2ΔHV1)=vec((Σ⊤2U⊤2ΔHV1)⊡D),D−1vec(V⊤2ΔH⊤U1Σ1)=vec((V⊤2ΔH⊤U1Σ1)⊡D). (2.12)

Then, using the partition of given by (1.6) we have

 [In xk]VK=[In xk][V11V12V21V22][(V22⊗Ik)Πn−k+1,kV21⊗In−k+1] =V11(V22⊗Ik)Πn−k+1,k+V12(V21⊗In−k+1)+xkV21(V22⊗Ik)Πn−k+1,k+xkV22(V21⊗In−k+1).

This, together with (2.10) and (2.12), yields

 [In xk]VKD−1[Ik⊗Σ⊤2Σ1⊗In−k+1]Wvec(ΔH) = (V11(V22⊗Ik)Πn−k+1,k+V12(V21⊗In−k+1)+xkV21(V22⊗Ik)Πn−k+1,k+xkV22(V21⊗In−k+1)) (vec((Σ⊤2U⊤2ΔHV1)⊡D)+vec((V⊤2ΔH⊤U1Σ1)⊡D)) = V11((V⊤1ΔH⊤U2Σ2)⊡D⊤)V⊤22+V11((Σ⊤1U⊤1ΔHV2)⊡D⊤)V⊤22 +V12((Σ⊤2U⊤2ΔHV1)⊡D]V⊤21+V12((V⊤2ΔH⊤U1Σ1)⊡D)V⊤21 +xkV21((V⊤1ΔH⊤U2Σ2)⊡D⊤)V⊤22+xkV21((Σ⊤1U⊤1ΔHV2)⊡D⊤)V⊤22 +xkV22((Σ⊤2U⊤2ΔHV1)⊡D]V⊤21+xkV22((V⊤2ΔH⊤U1Σ1)⊡D)V⊤21.

This completes the proof. ∎

When the data is sparse or badly-scaled, it is more suitable to adopt the componentwise perturbation analysis to investigate the conditioning of the TTLS problem. In the following definition, we introduce the relative mixed and componentwise condition number for the TTLS problem.

###### Definition 2.1.

Suppose the truncation level is chosen such that and . The mixed and componentwise condition numbers for the TTLS problem (1.3) are defined as follows:

 m(A,b) c(A,b)

In the following theorem, we give the explicit expressions of and .

###### Theorem 2.1.

Suppose the truncation level is chosen such that and . Then the mixed and componentwise condition numbers and defined in Definition 2.1 for the TTLS problem (1.3) can be characterized by

 m(A,b) =∥∥|Mk|vec([|A| |b|])∥∥∞∥xk∥∞, (2.13a) c(A,b) =∥∥∥|Mk|vec([|A| |b|])xk∥∥∥∞. (2.13b)

Proof.   Let , where and . For any , it follows from that

 |ΔA|≤ϵ|A|and|Δb|≤ϵ|b|.

Define

 ΘA=diag(vec(A))andΘb=diag(b).

By Lemma 2.1 we have for sufficiently small,

 ψk([A b]+ΔH)−ψk([A b])=Mkvec(ΔH)+O(∥ΔH∥2F) = Mk[ΘAΘb][Θ†Avec(ΔA)Θ†bvec(Δb)]+O(∥[ΔA Δb]∥2F),

and taking infinity norms we have

 ∥∥ψk([A b]+ΔH)−ψk([A b])∥∥∞ = ∥∥ ∥∥Mk[ΘAΘb][Θ†Avec(ΔA)Θ†bvec(Δb)]∥∥ ∥∥∞+O(∥[ΔA Δb]∥2F) ≤ ϵ∥∥∣∣Mk∣∣[|ΘA||Θb|]∥∥∞+O(ϵ2),

where the fact that is used. Thus,

 m(A,b) ≤ ∥∥∥|Mk|[|ΘA||Θb|]∥∥∥∞∥xk∥∞=∥∥∥|Mk|[|ΘA||Θb|]1mn+m∥∥∥∞∥xk∥∞ =

where .

On the other hand, let the index is such that

 ∥∥|Mk|vec([|A| |b|])∥∥∞=|Mk(a,:)|vec([|A| |b|]),

where denotes the -th row of . We choose

 vec(ΔH)=ϵΘvec([|A||b|]),

where is a diagonal matrix such that =sign for . Using (2.2) we have

 m(A,b) ≥ limϵ→0∥∥ϵMkΘvec([|A||b|])+O(ϵ∥vec([|A||b|])∥22)∥∥∞ϵ∥xk∥∞ = ∥MkΘvec([|A||b|])∥∞∥xk∥∞ =

Therefore, we derive (2.13a). One can use the similar argument to obtain (2.13b). ∎

###### Remark 2.1.

Based on (2.3) and (2.5), the relative normwise condition number for the TTLS problem (1.3) can be defined and has the following expression

 κrel(A,b)=limϵ→0sup∥ΔH∥F≤ϵ∥∥[Ab]∥∥F∥ψk([Ab]+ΔH)−ψk([Ab])∥2ϵ∥xk∥2=∥Mk∥2 ∥[Ab]∥F∥xk∥2. (2.15)

Using the fact that

 ⎧⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪⎩∥∥|Mk|∥∥2=∥Mk∥2,∥Mk∥∞≤√m(n+1)∥Mk∥2,∥xk∥2≤√n∥xk∥∞,∥∥vec([A b])∥∥∞≤∥∥[A b]∥∥F,

it is easy to see that

 m(A,b)≤√(n+1)nm κrel(A,b). (2.16)

From Example 5.1, we can see can be much smaller than when the data is sparse and badly scaled. Therefore, one should adopt the mixed and componentwise condition number to measure the conditioning of (1.3) instead of the normwise condition number when is spare or badly scaled. Furthermore, in practice it is necessary to propose efficient and reliable condition estimations for and , which will be investigated in Section 4.

In [3, 18, 23], the structured TLS (STTLS) problem has been studied extensivley. Hence, it is interesting to study the structured perturbation analysis for the STTLS problem. In the following, we propose the structured normwise, mixed and componentwise condition numbers for the STTLS problem, where is a linear structured data matrix. Assume that is a linear subspace which consists of a class of basis matrices. Suppose there are () linearly independent matrices in , where are matrices of constants, typically 0s and 1s. For any , there is a uniques vector such that

 A=t∑i=1aiSi. (2.17)

In the following, we study the sensitivity of the STTLS solution to perturbations on the data and , which is defined by

 ψs,k(a,b):Rt×Rm→Rn:(a,b)↦xk, (2.18)

where is the unique solution to the STTLS problem (1.3) and (2.17).

###### Definition 2.2.

Suppose the truncation level is chosen such that and . The absolute structured normwise, mixed and componentwise condition number for the STTLS problem (1.3) and (2.17) are defined as follows:

 κs(a,b) ms(a,b) =limϵ→0sup|Δa|≤ϵ|a||Δb|≤ϵ|b|∥ψs,k((a,b)+(Δa,Δb))−ψs,k(a,b)∥∞ϵ∥xk∥∞, cs(a,b) =limϵ→0sup|Δa|≤ϵ|a||Δb|≤ϵ|b|1ϵ∥∥∥ψs,k((a,b)+(Δa,Δb))−ψs,k(a,b)xk∥∥∥∞.

In the following lemma, we provide the first order expansion of the STTLS solution with respect to the structured perturbations on and on , which help us to derive the structured condition number expressions for the STTLS problem (1.3) and (2.17). In view of the fact that , we can prove the following lemma from Lemma 2.1. The detailed proof is omitted here.

###### Lemma 2.3.

Under the same assumptions of Lemma 2.1, if with sufficiently small, then, for the STTLS solution of and the STTLS solution of , we have

 ~xk=xk+Mk[M00Im][ΔaΔb]+O(∥∥∥[ΔaΔb]∥∥∥22),

where .

The following theorem concerns with the explicit expressions for the structured normwise, mixed and componentwise condition numbers , , and defined in Definition 2.2 when can be expressed by (2.17). Since the proof is similar to Theorem 2.1, we omit it here.

###### Theorem 2.2.

Suppose the truncation level is chosen such that and . The absolute structured normwise, mixed and componentwise condition numbers , , and defined in Definition 2.2 for the STTLS problem (1.3) and (2.17) can be characterized by

 κs(a,b)=∥∥∥Mk[M00Im]∥∥∥2,ms(a,b)=∥∥∥ ∣∣∣Mk[M00Im