Low-Rank Approximation with 1/ε^1/3 Matrix-Vector Products
share
We study iterative methods based on Krylov subspaces for low-rank approximation under any Schatten-p norm. Here, given access to a matrix A through matrix-vector products, an accuracy parameter ϵ, and a target rank k, the goal is to find a rank-k matrix Z with orthonormal columns such that A(I -ZZ^⊤)_S_p≤ (1+ϵ)min_U^⊤ U = I_kA(I - U U^⊤)_S_p, where M_S_p denotes the ℓ_p norm of the the singular values of M. For the special cases of p=2 (Frobenius norm) and p = ∞ (Spectral norm), Musco and Musco (NeurIPS 2015) obtained an algorithm based on Krylov methods that uses Õ(k/√(ϵ)) matrix-vector products, improving on the naïve Õ(k/ϵ) dependence obtainable by the power method, where Õ suppresses poly(log(dk/ϵ)) factors. Our main result is an algorithm that uses only Õ(kp^1/6/ϵ^1/3) matrix-vector products, and works for all p ≥ 1. For p = 2 our bound improves the previous Õ(k/ϵ^1/2) bound to Õ(k/ϵ^1/3). Since the Schatten-p and Schatten-∞ norms are the same up to a 1+ ϵ factor when p ≥ (log d)/ϵ, our bound recovers the result of Musco and Musco for p = ∞. Further, we prove a matrix-vector query lower bound of Ω(1/ϵ^1/3) for any fixed constant p ≥ 1, showing that surprisingly Θ̃(1/ϵ^1/3) is the optimal complexity for constant k. To obtain our results, we introduce several new techniques, including optimizing over multiple Krylov subspaces simultaneously, and pinching inequalities for partitioned operators. Our lower bound for p ∈ [1,2] uses the Araki-Lieb-Thirring trace inequality, whereas for p>2, we appeal to a norm-compression inequality for aligned partitioned operators.
READ FULL TEXT