Performance of the low-rank tensor-train SVD (TT-SVD) for large dense tensors on modern multi-core CPUs

by   Melven Röhrig-Zöllner, et al.

There are several factorizations of multi-dimensional tensors into lower-dimensional components, known as `tensor networks'. We consider the popular `tensor-train' (TT) format and ask: How efficiently can we compute a low-rank approximation from a full tensor on current multi-core CPUs? Compared to sparse and dense linear algebra, kernel libraries for multi-linear algebra are rare and typically not as well optimized. Linear algebra libraries like BLAS and LAPACK may provide the required operations in principle, but often at the cost of additional data movements for rearranging memory layouts. Furthermore, these libraries are typically optimized for the compute-bound case (e.g. square matrix operations) whereas low-rank tensor decompositions lead to memory bandwidth limited operations. We propose a `tensor-train singular value decomposition' (TT-SVD) algorithm based on two building blocks: a `Q-less tall-skinny QR' factorization, and a fused tall-skinny matrix-matrix multiplication and reshape operation. We analyze the performance of the resulting TT-SVD algorithm using the Roofline performance model. In addition, we present performance results for different algorithmic variants for shared-memory as well as distributed-memory architectures. Our experiments show that commonly used TT-SVD implementations suffer severe performance penalties. We conclude that a dedicated library for tensor factorization kernels would benefit the community: Computing a low-rank approximation can be as cheap as reading the data twice from main memory. As a consequence, an implementation that achieves realistic performance will move the limit at which one has to resort to randomized methods that only process part of the data.



There are no comments yet.


page 1

page 2

page 3

page 4


Computing low-rank approximations of large-scale matrices with the Tensor Network randomized SVD

We propose a new algorithm for the computation of a singular value decom...

Tensor Completion via a Low-Rank Approximation Pursuit

This paper considers the completion problem for a tensor (also referred ...

Efficient algorithms for computing a rank-revealing UTV factorization on parallel computing architectures

The randomized singular value decomposition (RSVD) is by now a well esta...

H2OPUS-TLR: High Performance Tile Low Rank Symmetric Factorizations using Adaptive Randomized Approximation

Tile low rank representations of dense matrices partition them into bloc...

Out-of-core singular value decomposition

Singular value decomposition (SVD) is a standard matrix factorization te...

Performance Modeling and Prediction for Dense Linear Algebra

This dissertation introduces measurement-based performance modeling and ...

Strassen's Algorithm for Tensor Contraction

Tensor contraction (TC) is an important computational kernel widely used...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.