Low Rank Vectorized Hankel Lift for Matrix Recovery via Fast Iterative Hard Thresholding

09/23/2021
by   Zengying Zhu, et al.
FUDAN University
0

We propose a VHL-FIHT algorithm for matrix recovery in blind super-resolution problems in a nonconvex schema. Under certain assumptions, we show that VHL-FIHT converges to the ground truth with linear convergence rate. Numerical experiments are conducted to validate the linear convergence and effectiveness of the proposed approach.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

10/06/2021

Blind Super-resolution via Projected Gradient Descent

Blind super-resolution can be cast as low rank matrix recovery problem b...
08/31/2020

Super-linear convergence in the p-adic QR-algorithm

The QR-algorithm is one of the most important algorithms in linear algeb...
12/02/2021

Blind Super-resolution of Point Sources via Projected Gradient Descent

Blind super-resolution can be cast as a low rank matrix recovery problem...
03/02/2019

Matrix Completion via Nonconvex Regularization: Convergence of the Proximal Gradient Algorithm

Matrix completion has attracted much interest in the past decade in mach...
11/18/2013

Minimum n-Rank Approximation via Iterative Hard Thresholding

The problem of recovering a low n-rank tensor is an extension of sparse ...
04/02/2017

Provable Inductive Robust PCA via Iterative Hard Thresholding

The robust PCA problem, wherein, given an input data matrix that is the ...
08/12/2020

Vectorized Hankel Lift: A Convex Approach for Blind Super-Resolution of Point Sources

We consider the problem of resolving r point sources from n samples at t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Matrix recovery considers the problem of reconstructing a data matrix from a small number of measurements. This problem has many applications, such as recommedation system [12], X-ray crystallography [13], quantum tomography [14] and blind deconvolution [19], etc.

Though it is an ill-posed problem, tractable recovery is possible when the matrix admits certain low dimensional structures. Typical example is low rank matrix recovery where the target data matrix is assumed to be low rank. Many computationally efficient approaches for low rank matrix recovery have been developed in recent decade, including convex methods as well as non-convex methods, see [11, 4, 7, 9] and reference therein.

Motivated by blind super-resolution of point sources, we study a different low rank structured matrix recovery problem, where the target matrix can be modeled as . Here is normalized vector such that and is a vector defined as

The linear measurements of is given by

(I.1)

where is a linear operator that performs the linear observation and is a random vector. Let be the vectorized Hankel lift operator [6] which maps a matrix into an vectorized Hankel matrix,

where is the -th column of and . We will investigate matrix recovery problem based on the low rank structure of vectorized Hankel matrix . Specifically, assuming

(I.2)

We are particularly interested in under what condition can the data matrix be recovered from a small number of linear measurements, given the following non-convex recovery problem

(I.3)

We tend to solve it in a fast and provable way and this leads us to the Fast Iterative Hard Thresholding algorithm.

I-a Related work

The blind super-resolution problem is explored in many literatures [6, 10, 15, 18]. More specifically, an atomic norm minimization method has been proposed in [15, 18] to solve the blind super-resolution problem. Recently, Chen et al. [6] proposed a nuclear minimization method called Vectorized Hankel Lift to recover . Indeed, all of these recovery methods are for convex optimization problems which can be solved by many sophisticated algorithms such as interior point methods, gradient descent etc. However, one common drawback of the convex optimization approach is that the high computational complexity of solving the equivalent semi-definite programming.

I-B Our contribution

In this paper, we consider a non-convex recovery approach for blind super-resolution problem (I.3). Here, the objective function is a standard least squares function which minimizes the distance from the observed point sources to the recovered ones while the constraint function enforces low rank structure in the transform domain. We proposed a non-convex algorithm called Vectorized Hankel Lifted Fast Iterative Hard Thresholding (VHL-FIHT for short) to solve this low rank structured matrix recovery procedure (I.3). The algorithm is presented in Algorithm . We also establish the linear convergence rate of this method provided that number of measurements is of the order and given a suitable initialization.

I-C Notations and preliminaries

The vectorized Hankel matrix admits the following low rank decomposition [6]

(I.4)

where is the Khatri-Rao product of matrix and the matrices as well as are full column rank matrices.

The adjoint of , denoted , is a linear mapping from matrices to matrices of size . In particular, for any matrix , the -th column of is given by

where and is the set

Let be an operator such that

for any , where the scalar is defined as for . The Moore-Penrose pseudoinverse of is given by which satisfies . The adjoint of the operator , denoted , is defined as . Denote and . The adjoint of , denoted , is given by .

I-D Organization

The remainder of this letter is organized as follows. In Section 2, we will introduce VHL-FIHT algorithm. In Section 3, we will introduce two assumptions and establish our main results. The performance of VHL-FIHT is evaluated from numerical experiments in Section 4. In Section 5, we give the detailed proofs of main result. We close with a conclusion in Section 6.

Ii Vectorized Hankel Lifted Fast Iterative Hard Thresholding

Let

be the compact singular value decomposition of a rank-

matrix, where , and . It is known that the tangent space of the fixed rank- matrix manifold at is given by

Given any matrix , the projection of onto can be computed using the formula

The VHL-FIHT method is shown in Algorithm when used to solve (I.3

).An initial guess was obtained with spectrum method. In each iteration of VHL-FIHT, the current estimate

is first updated along the gradient descent direction of the objective in (I.3). Then, the Hankel matrix corresponding to the update is formed via the application of the vectorized Hankel lift operator , followed by a projection operator onto the space. Finally, it imposes a hard thresholding operator to by truncated SVD process and then apply on the low rank matrix .

Input: Initialization .
Output:

1:for  do
2:     
3:     
4:     
5:     
Algorithm 1 VHL-FIHT

Iii Main Result

In this section, we establish our main results. To this end, we make two assumptions.

Assumption III.1.

The column vectors of the subspace matrix are i.i.d random vectors which obey

for some constant . Here, denotes the -th entry of .

Assumption III.2.

There exists a constant such that

where the columns of and are the left and right singular vectors of separately, and is the -th block of .

Assumption III.1 is introduced in [5] and has been used in blind super-resolution [6, 10, 15, 18]. Assumption III.2 is commonly used in spectrally sparse signal recovery [1, 2, 8] and blind super-resolution [6]. Now, we are in the position to state our main result.

Theorem III.1.

Under Assumption III.1 and III.2

, with probability at least

, the iterations generated by FIHT (Alg. 1) with the initial guess satisfies

(III.1)

provided

where and are absolute constants and .

Remark III.1.

The sample complexity established in [6] for the Vectorized Hankel Lift is . While the sample complexity is sub-optimal dependence on and , our recovery method requires low per iteration computational complexity. Similar to [3], the per iteration computational cost of FIHT is about . To improve the dependence on , we will investigate other initialization procedures in future work.

Iv Numerical results

Numerical experiments are conducted to evaluate the performance of FIHT. In the numerical experiments, the target matrix is generated by and the measurements are obtained by (LABEL:measurements), where the location

are generated from a standard uniform distribution (i.e., U

), and the amplitudes are generated via where follows U and follows U. Each entry of subspace matrix

is independently and identically generated from a standard normal distribution (i.e.,

). The coefficient vectors

are generated from a standardized multivariate Gaussian distribution (i.e.,

, where

is the identity matrix). We set the dimension of signal

and the number of point sources . Fig. 1 presents the logarithmic recovery error with respect to the number of iteration. Numerical experiment shows that FIHT converges linearly.

Fig. 1: with respect to the number of iteration.

V Proof of main result

We first introduce three auxiliary lemmas that will be used in our proof.

Lemma V.1 ([6, Corollary III.9]).

Suppose . The event

(V.1)

occurs with probability exceeding .

Lemma V.2 ([16, Lemma 5.2]).

Suppose that . Then with probability at least , the initialization obeys

Lemma V.3.

Suppose

(V.2)

Conditioned on (V.1), one has

(V.3)
(V.4)

We can rewrite the iteration as

(V.5)

Notice that . Our proof follows the line of [1]. We first assume that in the -th iteration obeys

(V.6)

Denote . We have that and can be bounded as follows:

Applying Lemma [17, Lemma 4.1] yields that

A simple computation obtains that . Finally, Lemma V.3 implies that

Combining these terms together, we have

(V.7)
(V.8)

where (V.7) has used (V.2) and (V.8) is due to . Finally, it remains to verify (V.6). By Lemma V.2, the inequality (V.6) is valid for . Since is a contractive sequence following from (V.8), the inequality (V.6) holds for all by induction.

V-a Proof of Lemma v.3

Proof.

For any , we have

where the last inequality is due to Lemma III in [6]. So it follows that and

Finally, a straightforward computation yields that

(V.9)
(V.10)

where (V.9) is due to Lemma [17, Lemma 4.1] and the fact that and , step (V.10) follows (V.2).∎

Vi conclusion

We propose a VHL-FIHT method to solve the blind super-resolution problem in a non-convex schema. The convergence analysis has been established for VHL-FIHT, showing that the algorithm will linearly converge to the target matrix given suitable initialization and provided the number of samples is large enough. The numerical experiments validate our theoretical results.

Vii acknowledgement

We would like to thank professor Ke Wei for useful discussion.

References

  • [1] J. Cai, S. Liu, and W. Xu (2015) A fast algorithm for reconstruction of spectrally sparse signals in super-resolution. In Wavelets and Sparsity XVI, Vol. 9597, pp. 95970A. Cited by: §III, §V.
  • [2] J. Cai, T. Wang, and K. Wei (2018) Spectral compressed sensing via projected gradient descent. SIAM Journal on Optimization 28 (3), pp. 2625–2653. Cited by: §III.
  • [3] J. Cai, T. Wang, and K. Wei (2019) Fast and provable algorithms for spectrally sparse signal reconstruction via low-rank hankel matrix completion. Applied and Computational Harmonic Analysis 46 (1), pp. 94–121. External Links: ISSN 1063-5203, Document, Link Cited by: Remark III.1.
  • [4] J. Cai and K. Wei (2018) Exploiting the structure effectively and efficiently in low rank matrix recovery. Handbook of Numerical Analysis abs/1809.03652. Cited by: §I.
  • [5] E. J. Candes and Y. Plan (2011) A probabilistic and ripless theory of compressed sensing. IEEE transactions on information theory 57 (11), pp. 7235–7254. Cited by: §III.
  • [6] J. Chen, W. Gao, S. Mao, and K. Wei (2020) Vectorized hankel lift: a convex approach for blind super-resolution of point sources. External Links: 2008.05092 Cited by: §I-A, §I-C, §I, Remark III.1, §III, §V-A, Lemma V.1.
  • [7] Y. Chen and Y. Chi (2018) Harnessing structures in big data via guaranteed low-rank matrix estimation: recent theory and fast algorithms via convex and nonconvex optimization. IEEE Signal Processing Magazine 35 (4), pp. 14–31. Cited by: §I.
  • [8] Y. Chen and Y. Chi (2014-10) Robust spectral compressed sensing via structured matrix completion. IEEE Transactions on Information Theory 60 (10), pp. 6576–6601. Cited by: §III.
  • [9] Y. Chi, Y. M. Lu, and Y. Chen (2019) Nonconvex optimization meets low-rank matrix factorization: an overview. IEEE Transactions on Signal Processing 67 (20), pp. 5239–5269. Cited by: §I.
  • [10] Y. Chi (2016) Guaranteed blind sparse spikes deconvolution via lifting and convex optimization. IEEE Journal of Selected Topics in Signal Processing 10 (4), pp. 782–794. Cited by: §I-A, §III.
  • [11] M. A. Davenport and J. Romberg (2016) An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing 10 (4), pp. 608–622. Cited by: §I.
  • [12] D. Goldberg, D. Nichols, B. M. Oki, and D. Terry (1992) Using collaborative filtering to weave an information tapestry. Communications of the ACM 35, pp. 61–70. Cited by: §I.
  • [13] R. W. Harrison (1993-05) Phase problem in crystallography. J. Opt. Soc. Am. A 10 (5), pp. 1046–1055. External Links: Link, Document Cited by: §I.
  • [14] M. Kliesch, R. Kueng, J. Eisert, and D. Gross (2017-01) Guaranteed recovery of quantum processes from few measurements. Quantum 3, pp. . External Links: Document Cited by: §I.
  • [15] S. Li, M. B. Wakin, and G. Tang (2019) Atomic norm denoising for complex exponentials with unknown waveform modulations. IEEE Transactions on Information Theory 66 (6), pp. 3893–3913. Cited by: §I-A, §III.
  • [16] S. Mao and J. Chen (2021) Fast blind super-resolution of point sources via projected gradient descent. In preparation. Cited by: Lemma V.2.
  • [17] K. Wei, J. Cai, T. F. Chan, and S. Leung (2016) Guarantees of riemannian optimization for low rank matrix recovery. SIAM Journal on Matrix Analysis and Applications 37 (3), pp. 1198–1222. Cited by: §V-A, §V.
  • [18] D. Yang, G. Tang, and M. B. Wakin (2016) Super-resolution of complex exponentials from modulations with unknown waveforms. IEEE Transactions on Information Theory 62 (10), pp. 5809–5830. Cited by: §I-A, §III.
  • [19] S. Zhang, Y. Hao, M. Wang, and J. H. Chow (2018) Multichannel Hankel matrix completion through nonconvex optimization. IEEE Journal of Selected Topics in Signal Processing 12 (4), pp. 617–632. Cited by: §I.