On Sampling Random Features From Empirical Leverage Scores: Implementation and Theoretical Guarantees

03/20/2019
by   Shahin Shahrampour, et al.
0

Random features provide a practical framework for large-scale kernel approximation and supervised learning. It has been shown that data-dependent sampling of random features using leverage scores can significantly reduce the number of features required to achieve optimal learning bounds. Leverage scores introduce an optimized distribution for features based on an infinite-dimensional integral operator (depending on input distribution), which is impractical to sample from. Focusing on empirical leverage scores in this paper, we establish an out-of-sample performance bound, revealing an interesting trade-off between the approximated kernel and the eigenvalue decay of another kernel in the domain of random features defined based on data distribution. Our experiments verify that the empirical algorithm consistently outperforms vanilla Monte Carlo sampling, and with a minor modification the method is even competitive to supervised data-dependent kernel learning, without using the output (label) information.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/11/2019

A General Scoring Rule for Randomized Kernel Approximation with Application to Canonical Correlation Analysis

Random features has been widely used for kernel approximation in large-s...
research
12/19/2017

On Data-Dependent Random Features for Improved Generalization in Supervised Learning

The randomized-feature approach has been successfully employed in large-...
research
05/23/2017

Data-driven Random Fourier Features using Stein Effect

Large-scale kernel approximation is an important problem in machine lear...
research
06/12/2020

Fourier Sparse Leverage Scores and Approximate Kernel Learning

We prove new explicit upper bounds on the leverage scores of Fourier spa...
research
11/20/2019

Random Fourier Features via Fast Surrogate Leverage Weighted Sampling

In this paper, we propose a fast surrogate leverage weighted sampling st...
research
10/16/2020

Fast Graph Kernel with Optical Random Features

The graphlet kernel is a classical method in graph classification. It ho...
research
05/21/2018

Relating Leverage Scores and Density using Regularized Christoffel Functions

Statistical leverage scores emerged as a fundamental tool for matrix ske...

Please sign up or login with your details

Forgot password? Click here to reset