The Rényi Gaussian Process

1 Introduction

In Gaussian process ( $G P$ ), people mainly focus on two types of inferences: the exact inference and the variational inference. The exact inference (Wang et al., 2019) directly optimizes the data likelihood

L_{e x a c t} = log N (0, σ_{ϵ}^{2} I + K_{f, f}),

while the variational inference using Kullback-Leibler (KL) divergence (Burt et al., 2019) optimizes a tractable lower bound of data likelihood

L_{V I} = log N (0, σ_{ϵ}^{2} I + Q) - \frac{1}{2 σ_{ϵ}^{2}} Tr (K_{f, f} - Q),

where $Q$ is a Nyström approximation of the exact covariance matrix $K_{f, f}$ . This lower bound significantly reduces computational burden and avoids model overfitting as it enforces regularization on the likelihood. In this article we introduce an alternative closed form lower bound on the likelihood based on the Rényi $α$ -divergence (Rényi and others, 1961)

Main Results The new lower bound can be expressed in a closed form as follows:

\begin{matrix} L_{α} (q; y) & = log N (0, σ_{ϵ}^{2} I + (1 - α) K_{f, f} + α Q) + log | I + \frac{1 - α}{σ_{ϵ}^{2}} (K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} . \end{matrix}

This new lower bound can be viewed as a convex combination of the sparse $G P$ and the exact $G P$ . The key advantage of this bound is it capability to control and tune the enforced regularization on the model and thus is a generalization of the traditional sparse variational $G P$ regression. Please refer to Sec. 2 for more information.

From the theoretical perspective, we show that with probability at least $1 - δ$ , the Rényi $α$ -divergence between the variational distribution and the true posterior converges to 0 as the number of data points increase. Specifically,

\begin{matrix} D_{α} [q | | p] & \leq \frac{1}{δ} \frac{α}{2 (1 - α)} log [1 + \frac{1 - α}{σ_{ϵ}^{2}} \frac{[(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ]}{N}]^{N} + α \frac{(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ}{2 δ σ_{ϵ}^{2}} \frac{{∥ y ∥}^{2}}{σ_{ϵ}^{2}} . \end{matrix}

As shown in this equation, $α$ plays an important role in controlling rate of convergence. The role of $α$ and rates of convergence of different kernels are given in in Sec. 3 and 4.

2 The Rényi Gaussian Processes and Variational Inference

Traditional variational inference is seeking to minimize the Kullback-Leibler (KL) divergence between the variational density $q (θ)$ and the intractable posterior $p (θ | D)$ , where $θ$

is a vector of parameters and

D

is the dataset. This minimization problem in turns yields a tractable evidence lower bound (ELBO) of the marginal log-likelihood function of data

log p (D)

. The Rényi’s

α

-divergence is a more general distance measure than the KL divergence. In this work, we want to explore the Rényi divergence based

G P

2.1 Rényi Divergence

The Rényi’s $α$ -divergence between two distributions $p$ and $q$

on a random variable

θ

is defined as

D_{α} [p | | q] = \frac{1}{α - 1} log \int p (θ)^{α} q (θ)^{1 - α} d θ, α \in (0, 1) .

This divergence contains a rich family of distance measure such as KL-divergence. Besides, the domain of $α$ can be extended to $α < 0$ and $α > 1$ .

Claim 1.

${lim}_{α \to 1} D_{α} [p | | q] = K L [p | | q]$ .

Therefore, KL-divergence is a special case of $α$ -divergence. It is well-known that KL-divergence yields a popular ELBO. Therefore, it would be interesting to derive a similar bound using the $α$ -divergence. Let $D = y$ (our data). Starting from $VR(q||p)\coloneqqlogp(y)−Dα[q(f,U|Z)||p(f,U,y|Z)]$ , we will reach the variational Rényi (VR) bound (Li and Turner, 2016). This form is defined as

Lα(q;y)\coloneqq11−αlogEq[(p(f,U,y|Z)q(f,U|Z))1−α]=VR(q||p),

where $f$ is a Gaussian process, $Z$ is the pseudo-input and $U$ is the latent variable.

Claim 2.

$L_{0} (q; y) = log P (y)$ .

Denote by the $L_{V I}$ as the ELBO, we have the following claim.

Claim 3.

$L_{V I} = {lim}_{α \to 1} L_{α} (q; y) \leq L_{α_{+}} (q; y) \leq log P (y) \leq L_{α -} (q; y), \forall α_{+} \in (0, 1), α_{-} < 0$ .

The proof is given in the appendix.

2.2 The Variational Rényi Lower Bound

Our bound is

\begin{matrix} L_{α} (q; y) & = log N (0, σ_{ϵ}^{2} I + (1 - α) [K_{f, f}] + α Q) + log C_{x}, \end{matrix}

where

\begin{matrix} C_{x} & = | I + \frac{1 - α}{σ_{ϵ}^{2}} (K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} \approx {1 + \frac{1 - α}{σ_{ϵ}^{2}} Tr% (K_{f, f} - Q) + O (\frac{(1 - α)^{2}}{σ_{ϵ}^{4}})}^{\frac{- α}{2 (1 - α)}}, \end{matrix}

$K$ is the covariance matrix and $Q = K_{f, U} K_{U, U}^{- 1} K_{U, f}$ . It can be seen that the new lower bound is the convex combination of components from sparse $G P$ ( $Q$ ) and components from exact $G P$ ( $K_{f, f}$ ). We can also see that $α$ plays an important role in model regularization.

We also derive a data-dependent upper bound similar to Titsias (2014). See appendix for details.

3 Convergence Analysis

In this section, we will derive some convergence results based on recent works from Titsias (2014); Huggins et al. (2018); Burt et al. (2019). We will provide some extensions to those works. Due to space limit, we move all proofs into appendix.

Theorem 4.

Suppose $N$ data points are drawn i.i.d from input distribution $p (x)$ and $k (x, x) \leq v, \forall x \in X$ . Sample $M$ inducing points from the training data with the probability assigned to any set of size $M$ equal to the probability assigned to the corresponding subset by an $ϵ$ k-Determinantal Point Process (k-DPP) (Belabbas and Wolfe, 2009) with $k = M$ . If $y$ is distributed according to a sample from the prior generative model, with probability at least $1 - δ$ ,

\begin{matrix} VR [q | | p] & \leq α \frac{(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ}{2 δ σ_{ϵ}^{2}} + \frac{1}{δ} \frac{α}{2 (1 - α)} log [1 + \frac{1 - α}{σ_{ϵ}^{2}} \frac{[(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ]}{N}]^{N} . \end{matrix}

where $λ_{m}$

are the eigenvalues of the integral operator

K

associated to kernel,

k

and

p (x)

As $α \to 1$ , we obtain the bound for the KL divergence.

Theorem 5.

\begin{matrix} D_{α} [q | | p] & \leq \frac{1}{δ} \frac{α}{2 (1 - α)} log [1 + \frac{1 - α}{σ_{ϵ}^{2}} \frac{[(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ]}{N}]^{N} + α \frac{(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ}{2 δ σ_{ϵ}^{2}} \frac{{∥ y ∥}^{2}}{σ_{ϵ}^{2}} \end{matrix}

where $C = N \sum_{m = M + 1}^{\infty} λ_{m}$ and $λ_{m}$ are the eigenvalues of the integral operator $K$ associated to kernel, $k$ and $p (x)$ .

As $α \to 1$ , we reach the bound for the KL divergence.

4 Consequences

4.1 Smooth Kernel

We will provide a convergence result with SE kernel. For SE kernel, we have $λ_{m} = v \sqrt{2 a / A} B^{m - 1}$ , where $a = 1 / (4 σ_{ϵ}^{2})$ , $b = 1 / (2 ℓ^{2})$ , $c = \sqrt{a^{2} + 2 a b}$ , $A = a + b + c$ and $B = b / A$ . $ℓ$ is the length parameter, $v$

is signal variance and

σ_{ϵ}

is the noise parameter. We can obtain

\sum_{m = M + 1}^{\infty} λ_{m} = \frac{v \sqrt{2 a}}{(1 - B) \sqrt{A}} B^{M}

(Burt et al., 2019).

Corollary 6.

Suppose ${∥ y ∥}^{2} \leq R N$ , where $R$ is a constant. Fix $γ > 0$ and take $ϵ = \frac{δ σ_{ϵ}^{2}}{v N^{γ + 2}}$

. Assume the input data is normally distributed and regression in performed with a SE kernel. With probability

1 - δ

V R [q | | p] \leq 2 α \frac{R}{σ_{ϵ}^{2}} \frac{1}{N^{γ}} + \frac{1}{δ} \frac{α}{2 (1 - α)} log [1 + (1 - α) (\frac{4 δ}{N^{γ + 2}})]^{N},

when inference is performed with $M = \frac{(3 + γ) log N + log D}{log (B^{- 1})}$ . where $D = \frac{v \sqrt{2 a}}{a \sqrt{A} σ_{ϵ}^{2} δ (1 - B)}$ .

Proof.

From Burt et al. (2019), we know $\frac{C (M + 1)}{2 δ σ_{ϵ}^{2}} < \frac{1}{N^{γ + 1}}$ . By Theorem 19, we will obtain

\begin{matrix} VR [q | | p] & \leq 2 α \frac{R}{σ_{ϵ}^{2}} \frac{1}{N^{γ}} + \frac{1}{δ} \frac{α}{2 (1 - α)} log [1 + (1 - α) (\frac{2 δ}{N^{γ + 2}} + \frac{2 δ}{N^{γ + 2}})]^{N} < 2 α \frac{R}{σ_{ϵ}^{2}} \frac{1}{N^{γ}} + α (\frac{2}{N^{γ + 1}}) = \frac{α}{N^{γ}} (\frac{2 R}{} σ_{ϵ}^{2} + \frac{2}{N}), \end{matrix}

∎

As $N \to \infty$ , $VR [q | | p] \to 0$ . As $α \to 1$ , we obtain $\frac{1}{N^{γ}} (\frac{2 R}{σ_{ϵ}^{2}} + \frac{2}{N}$ ).

4.2 Non-smooth Kernel

For the Matérn $k + \frac{1}{2}$ , $λ_{m} ≍ \frac{1}{m^{2 k + 2}}$ . We can obtain $\sum_{m = M + 1}^{\infty} λ_{m} = O (\frac{1}{M^{2 k + 1}})$ by the following claim.

Claim 7.

$\sum_{m = M + 1}^{\infty} λ_{m} = O (\frac{1}{M^{2 k + 1}})$ .

Proof.

It is easy to see that $\sum_{m = 1}^{\infty} λ_{m} = ζ (2 k + 2)$ , where $ζ$ is a Riemann zeta function. By the Euler-Maclaurin sum formula, we have the generalized harmonic number (Woon, 1998)

M \sum m = 1 (\frac{1}{m})^{2 k + 2} = ζ (2 k + 2) + \frac{1}{- 2 k - 1} M^{- 2 k - 1} + \frac{1}{2} M^{- 2 k - 2} - \frac{2 k + 2}{12} M^{- 2 k - 3} + O (M^{- 2 k - 4}) .

Therefore,

\infty \sum m = M + 1 λ_{m} = O (\frac{1}{M^{2 k + 1}}) = - \frac{1}{- 2 k - 1} M^{- 2 k - 1} - \frac{1}{2} M^{- 2 k - 2} + \frac{2 k + 2}{12} M^{- 2 k - 3} - O (M^{- 2 k - 4}) = O (\frac{1}{M^{2 k + 1}}) .

∎

Let $\sum_{m = M + 1}^{\infty} λ_{m} \leq A \frac{1}{M^{2 k + 1}}$ . Then by Theorem 19, we have

\begin{matrix} α \frac{(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ}{2 δ σ_{ϵ}^{2}} \frac{{∥ y ∥}^{2}}{σ_{ϵ}^{2}} \leq α \frac{(M + 1) N A \frac{1}{M^{2 k + 1}} + 2 N v ϵ}{2 δ σ_{ϵ}^{2}} \frac{R N}{σ_{ϵ}^{2}} = \frac{α R}{2 δ σ_{ϵ}^{4}} (\frac{(M + 1) N^{2} A}{M^{2 k + 1}} + 2 N^{2} v ϵ) . \end{matrix}

In order to let ${lim}_{N \to \infty} \frac{(M + 1) N^{2}}{M^{2 k + 1}} \to 0$ , we require $M = N^{p}$ . Therefore,

\begin{matrix} \frac{(M + 1) N^{2} A}{M^{2 k + 1}} = \frac{(N^{p} + 1) N^{2} A}{N^{(2 k + 1) p}} \leq \frac{A}{N^{2 k p - 2}} . \end{matrix}

Let $2 k p - 2 \geq γ$ , then $p \geq \frac{γ + 2}{2 k}$ . Therefore, we have

\frac{α R}{2 σ_{ϵ}^{4}} (\frac{(M + 1) N^{2} A}{M^{2 k + 1}} + 2 N^{2} v ϵ) \leq \frac{α R}{N^{γ} σ_{ϵ}^{2}} + \frac{α R A}{2 δ σ_{ϵ}^{4} N^{γ}} .

Another term in the bound can also be simplified as

\begin{matrix} \frac{α}{2 (1 - α)} log [1 + \frac{1 - α}{σ_{ϵ}^{2}} \frac{[(M + 1) N \sum_{m = M + 1}^{\infty} λ_{m} + 2 N v ϵ]}{N}]^{N} \leq \frac{α N}{2 (1 - α)} log [1 + (1 - α) (\frac{A}{σ_{ϵ}^{2} N^{γ + 2}} + \frac{2 δ}{σ_{ϵ}^{2} N^{γ + 2}})] . \end{matrix}

Appendix

Proof of Claims in Sec. 2.1

Claim 8.

${lim}_{α \to 1} D_{α} [p | | q] = K L [p | | q]$ .

Proof.

Applying the L’Hopital rule, we have

\begin{matrix} lim α \to 1 D_{α} [p | | q] & = lim α \to 1 \frac{1}{α - 1} log \int p (θ)^{α} q (θ)^{1 - α} d θ = lim α \to 1 \frac{1}{\frac{d}{d α} (α - 1)} \frac{d}{} d α log \int p (θ)^{α} q (θ)^{1 - α} d θ = lim α \to 1 \frac{d}{d α} log \int p (θ)^{α} q (θ)^{1 - α} d θ \end{matrix}

By the Leibniz’s rule, we have

\begin{matrix} lim α \to 1 D_{α} [p | | q] & = lim α \to 1 \frac{d}{d α} log \int p (θ)^{α} q (θ)^{1 - α} d θ = lim α \to 1 \frac{\int p (θ)^{α} q (θ)^{1 - α} [log p (θ) - log q (θ)] d θ}{\int p (θ)^{α} q (θ)^{1 - α} d θ} = \frac{\int p (θ) [log p (θ) - log q (θ)] d θ}{\int p (θ) d θ} = \int p (θ) log \frac{p (θ)}{q (θ)} d θ = K L [p | | q] . \end{matrix}

∎

Claim 9.

$L_{0} (q; y) = log P (y)$ .

Proof.

This is trivial, just let $α = 0$ . ∎

Claim 10.

$L_{V I} = {lim}_{α \to 1} L_{α} (q; y) \leq L_{α_{+}} (q; y) \leq log P (y) \leq L_{α -} (q; y), \forall α_{+} \in (0, 1), α_{-} < 0$ .

Proof.

The first equality follows from the Claim 1. The left inequality can be obtained by the Jensen inequality. ∎

The Variational Rényi Lower Bound

When we apply the VR bound to $G P$ and assume that $q (f, U | Z) = p (f | U, Z) q (U)$ , we can further obtain

It has been shown that $p (f | U, Z) = N (K_{f, U} K_{U, U}^{- 1} U, K_{f, f} - Q)$ , where $Q = K_{f, U} K_{U, U}^{- 1} K_{U, f}$ . Besides, we have $p (y | f) = N (f, σ_{ϵ}^{2} I)$ . Therefore,

\begin{matrix} \int p (f | U, Z) p (y | f)^{1 - α} d f = \int p (f | U, Z) (| 2 π σ_{ϵ}^{2} I |^{- 0.5} e^{- \frac{1}{2} (y - f)^{T} (σ_{ϵ}^{2} I)^{- 1} (y - f)})^{1 - α} d f = \frac{| 2 π σ_{ϵ}^{2} I |^{- 0.5 (1 - α)}}{| 2 π σ_{ϵ}^{2} I / (1 - α) |^{- 0.5}} \int p (f | U, Z) N (f, \frac{σ_{ϵ}^{2} I}{1 - α}) d f = \frac{| 2 π σ_{ϵ}^{2} I |^{- 0.5 (1 - α)}}{| 2 π σ_{ϵ}^{2} I / (1 - α) |^{- 0.5}} N (K_{f, U} K_{U, U}^{- 1} U, \frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) = (2 π σ_{ϵ}^{2})^{\frac{α N}{2}} (\frac{1}{1 - α})^{\frac{N}{2}} N (K_{f, U} K_{U, U}^{- 1} U, \frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) = p (y | U, Z) . \end{matrix}

Instead of treating $q (U)$ as a pool of free parameters, it is desirable to find the optimal $q^{*} (U)$ to maximize the lower bound. This can be achieved by the special case of the Hölder inequality (i.e., Lyapunov inequality).

Then we have,

\begin{matrix} L_{α} (q; y) = \frac{1}{1 - α} log \int p (y | U, Z) q (U)^{α} p (U | Z)^{1 - α} d U = \frac{1}{1 - α} log \int q (U) (\frac{p (y | U, Z)^{1 / (1 - α)} p (U | Z)}{q (U)})^{1 - α} d U = \frac{1}{1 - α} log E_{q} (\frac{p (y | U, Z)^{1 / (1 - α)} p (U | Z)}{q (U)})^{1 - α} \leq \frac{1}{1 - α} log [E_{q} (\frac{p (y | U, Z)^{1 / (1 - α)} p (U | Z)}{q (U)})]^{1 - α} = log E_{q} (\frac{p (y | U, Z)^{1 / (1 - α)} p (U | Z)}{q (U)}) = log \int p (y | U, Z)^{1 / (1 - α)} p (U | Z) d U . \end{matrix}

The optimal $q (U)$ is

q^{*} (U) \propto p (y | U, Z)^{1 / (1 - α)} p (U | Z) .

Specifically,

q^{*} (U) = \frac{p (y | U, Z)^{1 / (1 - α)} p (U | Z)}{\int p (y | U, Z)^{1 / (1 - α)} p (U | Z) d U} .

It can be shown that

\begin{matrix} p (y | U, Z)^{\frac{1}{1 - α}} = [(2 π σ_{ϵ}^{2})^{\frac{α N}{2}} (\frac{1}{1 - α})^{\frac{N}{2}}]^{\frac{1}{1 - α}} N (K_{f, U} K_{U, U}^{- 1} U, \frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q)^{\frac{1}{1 - α}} = [(2 π σ_{ϵ}^{2})^{\frac{α N}{2 (1 - α)}} (\frac{1}{1 - α})^{\frac{N}{2 (1 - α)}}] C N (K_{f, U} K_{U, U}^{- 1} U, σ_{ϵ}^{2} I + (1 - α) [K_{f, f} - Q]), \end{matrix}

where $C = \frac{| 2 π (\frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) |^{- 0.5 / (1 - α)}}{| 2 π (σ_{ϵ}^{2} I + (1 - α) [K_{f, f} - Q]) |^{- 0.5}} = | 2 π (\frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} (1 - α)^{N / 2}$ . Since $p (U | Z) = N (0, K_{U, U})$ , we have

\begin{matrix} L_{α} (q; y) & = log \int p (y | U, Z)^{1 / (1 - α)} p (U | Z) d U = log C_{x} N (0, σ_{ϵ}^{2} I + (1 - α) [K_{f, f} - Q] + K_{f, U} K_{U, U}^{- 1} K_{U, f}) = log C_{x} N (0, σ_{ϵ}^{2} I + (1 - α) [K_{f, f} - Q] + Q) = log N (0, σ_{ϵ}^{2} I + (1 - α) [K_{f, f}] + α Q) + log C_{x}, \end{matrix}

where

\begin{matrix} C_{x} & = [(2 π σ_{ϵ}^{2})^{\frac{α N}{2 (1 - α)}} (\frac{1}{1 - α})^{\frac{N}{2 (1 - α)}}] [| 2 π (\frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} (1 - α)^{N / 2}] = (2 π σ_{ϵ}^{2})^{\frac{α N}{2 (1 - α)}} (1 - α)^{\frac{- α N}{2 (1 - α)}} | 2 π (\frac{σ_{ϵ}^{2}}{1 - α} I + K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} = | I + \frac{1 - α}{σ_{ϵ}^{2}} (K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} \approx {1 + \frac{1 - α}{σ_{ϵ}^{2}} Tr% (K_{f, f} - Q) + O (\frac{(1 - α)^{2}}{σ_{ϵ}^{4}})}^{\frac{- α}{2 (1 - α)}} \end{matrix}

The last equality comes from the variation of Jacobi’s formula. The $\approx$ approximates well only when $\frac{1 - α}{σ_{ϵ}^{2}}$ is “small”. Therefore, the lower bound can be expressed as

\begin{matrix} L_{α} (q; y) \approx log N (0, σ_{ϵ}^{2} I + (1 - α) [K_{f, f}] + α Q) + log {1 + \frac{1 - α}{σ_{ϵ}^{2}} Tr (K_{f, f} - Q) + O (\frac{(1 - α)^{2}}{σ_{ϵ}^{4}})}^{\frac{- α}{2 (1 - α)}}, \end{matrix}

given that $α \in (0, 1)$ . While this form is attractive, it is not practically useful since when $1 - α$ is “large”, the approximation does not work well. In the analysis section, we will instead use $| I + \frac{1 - α}{σ_{ϵ}^{2}} (K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}}$ to prove the convergence result.

The Data-dependent Upper Bound

Lemma 11.

Suppose we have two positive semi-definite (PSD) matrices $A$ and $B$ such that $A - B$ is also a PSD matrix, then $| A | \geq | B |$ . Furthermore, if $A$ and $B$ are positive definite (PD), then $B^{- 1} \geq A^{- 1}$ .

This lemma has been proved in (Horn and Johnson, 2012). Based on this lemma, we can compute a data-dependent upper bound on the log-marginal likelihood (Titsias, 2014).

Claim 12.

$logp(y)≤log1|2π((1−α)Kf,f+αQ+σ2ϵI)|12e−12yT((1−α)Kf,f+αQ+αTr(Kf,f−Q)I+σ2ϵI)−1y\coloneqqLupper$ .

Proof.

Since

K_{f, f} + σ_{ϵ}^{2} I = (1 - α) K_{f, f} + α K_{f, f} + σ_{ϵ}^{2} I ⪰ (1 - α) K_{f, f} + α Q + σ_{ϵ}^{2} I ⪰ 0,

where $A ⪰ B$ means $x^{T} A x \geq x^{T} B x \geq 0, \forall x$ . Then, we can obtain $| K_{f, f} + σ_{ϵ}^{2} I | \geq | (1 - α) K_{f, f} + α Q + σ_{ϵ}^{2} I |$ since they are both PSD matrix. Therefore,

\frac{1}{| 2 π (K_{f, f} + σ_{ϵ}^{2} I) |^{\frac{1}{2}}} \leq \frac{1}{| 2 π ((1 - α) K_{f, f} + α Q + σ_{ϵ}^{2} I) |^{\frac{1}{2}}} .

Let $U Λ U^{T}$ be the eigen-decomposition of $K_{f, f} - Q$ . This decomposition exists since the matrix is PD. Then

where $z = U^{T} y$ , ${λ_{i}}_{i = 1}^{N}$ are eigenvalues of $K_{f, f} - Q$ and $λ_{m a x} = max (λ_{1}, \dots, λ_{N})$ . Therefore, we have $y^{T} (K_{f, f} - Q) y \leq Tr (K_{f, f} - Q) {∥ y ∥}^{2} = Tr (K_{f, f} - Q) y^{T} y$ . Apparently, $α y^{T} (K_{f, f} - Q) y \leq α Tr (K_{f, f} - Q) y^{T} y$ . Therefore, we can obtain.

\begin{matrix} y^{T} (K_{f, f} + σ_{ϵ}^{2} I) y \leq y^{T} ((1 - α) K_{f, f} + α Q + σ_{ϵ}^{2} I) y + α Tr (K_{f, f} - Q) y^{T} y = y^{T} ((1 - α) K_{f, f} + α Q + α Tr (K_{f, f} - Q) I + σ_{ϵ}^{2} I) y . \end{matrix}

Based on this inequality, it is easy to show that

e^{- \frac{1}{2} y^{T} (K_{f, f} + σ_{ϵ}^{2} I)^{- 1} y} \leq e^{- \frac{1}{2} y^{T} ((1 - α) K_{f, f} + α Q + α Tr (K_{f, f} - Q) I + σ_{ϵ}^{2} I)^{- 1} y} .

Finally, we obtain

\begin{matrix} \frac{1}{| 2 π (K_{f, f} + σ_{ϵ}^{2} I) |^{\frac{1}{2}}} e^{- \frac{1}{2} y^{T} (K_{f, f} + σ_{ϵ}^{2} I)^{- 1} y} \leq \frac{1}{| 2 π ((1 - α) K_{f, f} + α Q + σ_{ϵ}^{2} I) |^{\frac{1}{2}}} e^{- \frac{1}{2} y^{T} ((1 - α) K_{f, f} + α Q + α Tr (K_{f, f} - Q) I + σ_{ϵ}^{2} I)^{- 1} y} . \end{matrix}

∎

We will use this upper bound to prove our main theorem.

Detailed Proof of Convergence Result

Let $q\coloneqqq(f,U|Z)$ and $p\coloneqqp(f,U,y|Z)$ .

Claim 13.

$- log | I + \frac{1 - α}{σ_{ϵ}^{2}} (K_{f, f} - Q) |^{\frac{- α}{2 (1 - α)}} \leq \frac{α}{2 (1 - α)} log (Tr (I + \frac{1 - α}{σ^{2}}$

The Rényi Gaussian Process

Authors

Related Research

Deep Gaussian Processes

Rates of Convergence for Sparse Variational Gaussian Process Regression

Deep Recurrent Gaussian Process with Variational Sparse Spectrum Approximation

Variational Gaussian Approximation for Poisson Data

Informative Planning for Worst-Case Error Minimisation in Sparse Gaussian Process Regression

Bounded Regression with Gaussian Process Projection

Identification of Gaussian Process State Space Models

This week in AI

1 Introduction

2 The Rényi Gaussian Processes and Variational Inference

2.1 Rényi Divergence

Claim 1.

Claim 2.

Claim 3.

2.2 The Variational Rényi Lower Bound

3 Convergence Analysis

Theorem 4.

Theorem 5.

4 Consequences

4.1 Smooth Kernel

Corollary 6.

Proof.

4.2 Non-smooth Kernel

Claim 7.

Proof.

Appendix

Proof of Claims in Sec. 2.1

Claim 8.

Proof.

Claim 9.

Proof.

Claim 10.

Proof.

The Variational Rényi Lower Bound

The Data-dependent Upper Bound

Lemma 11.

Claim 12.

Proof.

Detailed Proof of Convergence Result

Claim 13.