Sublinear Time Approximation of Text Similarity Matrices

12/17/2021
by   Archan Ray, et al.
8

We study algorithms for approximating pairwise similarity matrices that arise in natural language processing. Generally, computing a similarity matrix for n data points requires Ω(n^2) similarity computations. This quadratic scaling is a significant bottleneck, especially when similarities are computed via expensive functions, e.g., via transformer models. Approximation methods reduce this quadratic complexity, often by using a small subset of exactly computed similarities to approximate the remainder of the complete pairwise similarity matrix. Significant work focuses on the efficient approximation of positive semidefinite (PSD) similarity matrices, which arise e.g., in kernel methods. However, much less is understood about indefinite (non-PSD) similarity matrices, which often arise in NLP. Motivated by the observation that many of these matrices are still somewhat close to PSD, we introduce a generalization of the popular Nyström method to the indefinite setting. Our algorithm can be applied to any similarity matrix and runs in sublinear time in the size of the matrix, producing a rank-s approximation with just O(ns) similarity computations. We show that our method, along with a simple variant of CUR decomposition, performs very well in approximating a variety of similarity matrices arising in NLP tasks. We demonstrate high accuracy of the approximated similarity matrices in the downstream tasks of document classification, sentence similarity, and cross-document coreference.

READ FULL TEXT
research
11/15/2020

Efficient construction of an HSS preconditioner for symmetric positive definite ℋ^2 matrices

In an iterative approach for solving linear systems with ill-conditioned...
research
09/16/2021

Sparse Factorization of Large Square Matrices

Square matrices appear in many machine learning problems and models. Opt...
research
06/27/2018

Sublinear-Time Quadratic Minimization via Spectral Decomposition of Matrices

We design a sublinear-time approximation algorithm for quadratic functio...
research
07/31/2019

Sublinear Subwindow Search

We propose an efficient approximation algorithm for subwindow search tha...
research
10/27/2022

Efficient Similarity-based Passive Filter Pruning for Compressing CNNs

Convolution neural networks (CNNs) have shown great success in various a...
research
02/06/2017

Learning similarity preserving representations with neural similarity encoders

Many dimensionality reduction or manifold learning algorithms optimize f...
research
05/29/2016

Recycling Randomness with Structure for Sublinear time Kernel Expansions

We propose a scheme for recycling Gaussian random vectors into structure...

Please sign up or login with your details

Forgot password? Click here to reset