High-Dimensional Gaussian Process Inference with Derivatives

by   Filip de Roos, et al.

Although it is widely known that Gaussian processes can be conditioned on observations of the gradient, this functionality is of limited use due to the prohibitive computational cost of 𝒪(N^3 D^3) in data points N and dimension D. The dilemma of gradient observations is that a single one of them comes at the same cost as D independent function evaluations, so the latter are often preferred. Careful scrutiny reveals, however, that derivative observations give rise to highly structured kernel Gram matrices for very general classes of kernels (inter alia, stationary kernels). We show that in the low-data regime N<D, the Gram matrix can be decomposed in a manner that reduces the cost of inference to 𝒪(N^2D + (N^2)^3) (i.e., linear in the number of dimensions) and, in special cases, to 𝒪(N^2D + N^3). This reduction in complexity opens up new use-cases for inference with gradients especially in the high-dimensional regime, where the information-to-cost ratio of gradient observations significantly increases. We demonstrate this potential in a variety of tasks relevant for machine learning, such as optimization and Hamiltonian Monte Carlo with predictive gradients.


Scaling Gaussian Processes with Derivative Information Using Variational Inference

Gaussian processes with derivative information are useful in many settin...

Bézier Gaussian Processes for Tall and Wide Data

Modern approximations to Gaussian processes are suitable for "tall data"...

Scalable First-Order Bayesian Optimization via Structured Automatic Differentiation

Bayesian Optimization (BO) has shown great promise for the global optimi...

Mixtures of Gaussian Process Experts with SMC^2

Gaussian processes are a key component of many flexible statistical and ...

Randomly Projected Additive Gaussian Processes for Regression

Gaussian processes (GPs) provide flexible distributions over functions, ...

On Learning High Dimensional Structured Single Index Models

Single Index Models (SIMs) are simple yet flexible semi-parametric model...

Infinite-Horizon Gaussian Processes

Gaussian processes provide a flexible framework for forecasting, removin...