Training generative neural networks via Maximum Mean Discrepancy optimization

We consider training a deep neural network to generate samples from an unknown distribution given i.i.d. data. We frame learning as an optimization minimizing a two-sample test statistic---informally speaking, a good generator network produces samples that cause a two-sample test to fail to reject the null hypothesis. As our two-sample test statistic, we use an unbiased estimate of the maximum mean discrepancy, which is the centerpiece of the nonparametric kernel two-sample test proposed by Gretton et al. (2012). We compare to the adversarial nets framework introduced by Goodfellow et al. (2014), in which learning is a two-player game between a generator network and an adversarial discriminator network, both trained to outwit the other. From this perspective, the MMD statistic plays the role of the discriminator. In addition to empirical comparisons, we prove bounds on the generalization error incurred by optimizing the empirical MMD.

Comments

There are no comments yet.

Authors

• 11 publications
• 31 publications
• 86 publications
• k-Sample problem based on generalized maximum mean discrepancy

In this paper we deal with the problem of testing for the quality of k p...
11/22/2018 ∙ by Armando Sosthene Kali Balogoun, et al. ∙ 0

read it

• B-tests: Low Variance Kernel Two-Sample Tests

A family of maximum mean discrepancy (MMD) kernel two-sample tests is in...
07/08/2013 ∙ by Wojciech Zaremba, et al. ∙ 0

read it

• Classification Logit Two-sample Testing by Neural Networks

The recent success of generative adversarial networks and variational le...
09/25/2019 ∙ by Xiuyuan Cheng, et al. ∙ 0

read it

• Regularized Finite Dimensional Kernel Sobolev Discrepancy

We show in this note that the Sobolev Discrepancy introduced in Mroueh e...
05/16/2018 ∙ by Youssef Mroueh, et al. ∙ 0

read it

• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy

We propose a method to optimize the representation and distinguishabilit...
11/14/2016 ∙ by Dougal J. Sutherland, et al. ∙ 0

read it

• Exemplar-based synthesis of geology using kernel discrepancies and generative neural networks

We propose a framework for synthesis of geological images based on an ex...
09/20/2018 ∙ by Shing Chan, et al. ∙ 0

read it

• FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Test

The maximum mean discrepancy (MMD) is a recently proposed test statistic...
05/12/2014 ∙ by Ji Zhao, et al. ∙ 0

read it

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In this paper, we consider the problem of learning generative models from i.i.d. data with unknown distribution . We formulate the learning problem as one of finding a function , called the generator, such that, given an input drawn from some fixed noise distribution , the distribution of the output is close to the data’s distribution . Note that, given and , we can easily generate new samples despite not having an explicit representation for the underlying density.

We are particularly interested in the case where the generator is a deep neural network whose parameters we must learn. Rather than being used to classify or predict, these networks transport input randomness to output randomness, thus inducing a distribution. The first direct instantiation of this idea is due to MacKay94, although MacKay draws connections even further back to the work of Saund89 and others on autoencoders, suggesting that generators can be understood as decoders. MacKay’s proposal, called

density networks

, uses multi-layer perceptrons (MLP) as generators and learns the parameters by approximating Bayesian inference.

Since MacKay’s proposal, there has been a great deal of progress on learning generative models, especially over high-dimensional spaces like images. Some of the most successful approaches have been based on restricted Boltzmann machines SalakhutdinovH09 and deep Boltzmann networks HintonSalakhutdinov2006b. A recent example is the Neural Autoregressive Density Estimator due to UriaML13a. An indepth survey, however, is beyond the scope of this article.

This work builds on a proposal due to Goodfellow:2014. Their adversarial nets framework takes an indirect approach to learning deep generative neural networks: a discriminator network is trained to recognize the difference between training data and generated samples, while the generator is trained to confuse the discriminator. The resulting two-player game is cast as a minimax optimization of a differentiable objective and solved greedily by iteratively performing gradient descent steps to improve the generator and then the discriminator.

Given the greedy nature of the algorithm, Goodfellow:2014 give a careful prescription for balancing the training of the generator and the discriminator. In particular, two gradient steps on the discriminator’s parameters are taken for every iteration of the generator’s parameters. It is not clear at this point how sensitive this balance is as the data set and network vary. In this paper, we describe an approximation to adversarial learning that replaces the adversary with a closed-form nonparametric two-sample test statistic based on the Maximum Mean Discrepancy (MMD), which we adopted from the kernel two sample test Gretton:2012. We call our proposal MMD nets.111In independent work reported in a recent preprint, Li, Swersky, and Zemel LSZ also propose to use MMD as a training objective for generative neural networks. We leave a comparison to future work. We give bounds on the estimation error incurred by optimizing an empirical estimator rather than the true population MMD and give some illustrations on synthetic and real data.

2 Learning to sample as optimization

It is well known that, for any distribution and any continuous distribution on sufficiently regular spaces and , respectively, there is a function , such that when . (See, e.g., [][Lem. 3.22]FMP2.) In other words, we can transform an input from a fixed input distribution through a deterministic function, producing an output whose distribution is . For a given family of functions , called generators, we can cast the problem of learning a generative model as an optimization

 argminθδ(P,Gθ(N)),

where is some measure of discrepancy and is the distribution of when . In practice, we only have i.i.d. samples from , and so we optimize an empirical estimate of .

2.1 Adversarial nets

Adversarial nets Goodfellow:2014 can be cast within this framework: Let be a family of functions , called discriminators. We recover the adversarial nets objective with the discrepancy

 δAN(P,Gθ(N))=maxϕE[logDϕ(X)+log(1−Dϕ(Y))],

where and . In this case, Section 2 becomes

 minθmaxϕV(Gθ,Dϕ)

where

 V(Gθ,Dϕ)=E[logDϕ(X)+log(1−Dϕ(Gθ(W)))]

for and . The output of the discriminator

can be interpreted as the probability it assigns to its input being drawn from

, and so is the expected log loss incurred when classifying the origin of a point equally likely to have been drawn from or . Therefore, optimizing maximizes the probability of distinguishing samples from and . Assuming that the optimal discriminator exists for every , the optimal generator is that whose output distribution is closest to , as measured by the Jensen–Shannon divergence, which is minimized when .

In Goodfellow:2014, the generators and discriminators

are chosen to be multilayer perceptrons (MLP). In order to find a minimax solution, they propose taking alternating gradient steps along

and . Note that the composition that appears in the value function is yet another (larger) MLP. This fact permits the use of the back-propagation algorithm to take gradient steps.

2.2 MMD as an adversary

In their paper introducing adversarial nets, Goodfellow:2014 remark that a balance must be struck between optimizing the generator and optimizing the discriminator. In particular, the authors suggest maximization steps for every one minimization step to ensure that is well synchronized with during training. A large value for , however, can lead to overfitting. In their experiments, for every step taken along the gradient with respect to , they take two gradient steps with respect to to bring closer to the desired optimum (Goodfellow, pers. comm.).

It is unclear how sensitive this balance is. Regardless, while adversarial networks deliver impressive sampling performance, the optimization takes approximately 7.5 hours to train on the MNIST dataset running on a nVidia GeForce GTX TITAN GPU with 6GB RAM. Can we potentially speed up the process with a more tractable choice of adversary?

Our proposal is to replace the adversary with the kernel two-sample test introduced by Gretton:2012. In particular, we replace the family of discriminators with a family of test functions , closed under negation, and use the maximum mean discrepancy between and over , given by

 δMMDH(P,Gθ(N))=supf∈HE[f(X)]−E[f(Y)],

where and . See Fig. 1 for a comparison of the architectures of adversarial and MMD nets.

While Section 2.2 involves a maximization over a family of functions, Gretton:2012 show that it can be solved in closed form when is a reproducing kernel Hilbert space (RKHS).

More carefully, let be a reproducing kernel Hilbert space (RKHS) of real-valued functions on and let denote its inner product. By the reproducing property it follows that there exists a reproducing kernel such that every can be expressed as

 f(x)=⟨f,k(⋅,x)⟩H=∑αik(x,xi) (1)

The functions induced by a kernel are those functions in the closure of the span of the set , which is necessarily an RKHS. Note, that for every positive definite kernel there is a unique RKHS such that every function in satisfies Eq. 1.

Assume that is a nonempty compact metric space and a class of functions . Let and be Borel probability measures on , and let and

be random variables with distribution

and , respectively. The maximum mean discrepancy (MMD) between and is

 MMD(F,p,q)=supf∈FE[f(X)]−E[f(Y)]

If is chosen to be an RKHS , then

 MMD2(F,p,q)=∥μp−μq∥2H

where is the mean embedding of , given by

 μp=∫Xk(x,⋅)p(dx)∈H

and satisfying, for all ,

 E[f(X)]=⟨f,μp⟩H.

The properties of depend on the underlying RKHS . For our purposes, it suffices to say that if we take to be and consider the RKHS induced by Gaussian or Laplace kernels, then MMD is a metric, and so the minimum of our learning objective is achieved uniquely by , as desired. (For more details, see Fukumizu2008.)

In practice, we often do not have access to or . Instead, we are given independent i.i.d. data and fom and , respectively, and would like to estimate the MMD. Gretton:2012 showed that

 MMD2[H,p,q]=E[k(X,X′)−2k(X,Y)+k(Y,Y′)] (2)

and then proposed an unbiased estimator

 MMD2u[H,X,Y]=1N(N−1)∑n≠n′k(xn,xn′)+1M(M−1)∑m≠m′k(ym,ym′)−2MNM∑m=1N∑n=1k(xn,ym). (3)

3 MMD Nets

With an unbiased estimator of the MMD objective in hand, we can now define our proposal, MMD nets: Fix a neural network , where represents the parameters of the network. Let denote noise inputs drawn from , let with denote the noise inputs transformed by the network , and let denote the training data in . Given a positive definite kernel on , we minimize as a function of , where

 C(Yθ,X)=1M(M−1)∑m≠m′k(ym,ym′)−2MNM∑m=1N∑n=1k(ym,xn). (4)

Note that is comprised of only those parts of the unbiased estimator that depend on .

In practice, the minimization is solved by gradient descent, possibly on subsets of the data. More carefully, the chain rule gives us

 ∇C(Yθ,X) =1NN∑n=1M∑m=1∂Cn(Yθ,Xn)∂ym∂Gθ(wm)∂θ, (5)

where

 Cn(Yθ,Xn)=1M(M−1)∑m≠m′k(ym,ym′)−2MM∑m=1k(ym,xn). (6)

Each derivative is easily computed for standard kernels like the RBF kernel. Our gradient depends on the partial derivatives of the generator with respect to its parameters, which we can compute using back propagation.

4 Generalization bounds for MMD

MMD nets operate by minimizing an empirical estimate of the MMD. This estimate is subject to Monte Carlo error and so the network weights (parameters) that are found to minimize the empirical MMD may do a poor job at minimizing the exact population MMD. We show that, for sufficiently large data sets, this estimation error is bounded, despite the space of parameters being continuous and high dimensional.

Let denote the space of possible parameters for the generator , let be the distribution on for the noisy inputs, and let be the distribution of when for . Let be the value optimizing the unbiased empirical MMD estimate, i.e.,

 MMD2u(H,X,Y^θ)=infθMMD2u(H,X,Yθ),

and let be the value optimizing the population MMD, i.e.,

 MMD2(H,pdata,pθ∗)=infθMMD2(H,pdata,pθ).

We are interested in bounding the difference

 MMD2(H,pdata,p^θ)−MMD2(H,pdata,pθ∗).

To that end, for a measured space , write for the space of essentially bounded functions on and write for the unit ball under the sup norm, i.e.,

 B(L∞(X))={f:X→R:(∀x ∈X)f(x)∈[−1,1]}.

The bounds we obtain will depend on a notion of complexity captured by the fat-shattering dimension:

Definition 1 (Fat-shattering Mendelson03).

Let and . For every , is said to be -shattered by if there is some function , such that for every there is some for which

For every , the fat-shattering dimension of , written , is defined as

 fatε(F)=sup{|XN|:XN⊂ X,XN is ε-shattered by F}

We then have the following bound on the estimation error:

Theorem 1 (estimation error).

Assume the kernel is bounded by one. Define

 Gk+={g=k(Gθ(w),Gθ(⋅)):w∈W,θ∈Θ}

and

 GXk+={g=k(x,Gθ(⋅)):x∈X,θ∈Θ}.

Assume there exists and such that, for all , it holds that and . Then with probability at least ,

 MMD2(H,pdata,p^θ)

with

where the rate is

 r(p,γ,M)=Cp√γ⎧⎪ ⎪ ⎪⎨⎪ ⎪ ⎪⎩M−12if p<2,M−12log32(M)if p=2,M−1pif p>2,

for constants and depending on and alone.

The proof appears in the appendix. We can obtain simpler, but slightly more restrictive, hypotheses if we bound the fat-shattering dimension of the class of generators alone: Take the observation space to be a bounded subset of a finite-dimensional Euclidean space and the kernel to be Lipschitz continuous and translation invariant. For the RBF kernel, the Lipschitz constant is proportional to the inverse of the length-scale: the resulting bound loosens as the length scale shrinks.

5 Empirical evaluation

In this section, we demonstrate the approach on an illustrative synthetic example as well as the standard MNIST digits and Toronto Face Dataset (TFD) benchmarks. We show that MMD-based optimization of the generator rapidly delivers a generator that performs well in maximizing the density of a held-out test set under a kernel-density estimator.

5.1 Gaussian data, kernel, and generator

Under an RBF kernel and Gaussian generator with parameters , it is straightforward to find the gradient of by applying the chain rule. Using fixed random standard normal numbers , we have for . The result of these illustrative synthetic experiments can be found in Fig. 1. The dataset consisted of samples from a standard normal and noise input samples were generated from a standard normal with a fixed random seed. The algorithm was initialized at values . We fixed the learning rate to and ran gradient descent steps for iterations.

5.2 MNIST digits

We trained our generative network on the MNIST digits dataset Lecun98. The generator was chosen to be a fully connected, 3 hidden layers neural network with sigmoidal activation functions. Following Gretton:2012, we used a radial basis function (RBF) kernel, but also evaluated the rational quadratic (RQ) kernel CarlGPs and Laplacian kernel, but found that the RBF performed best in the parameter ranges we evaluated. We used Bayesian optimization (WHETLab) to set the bandwidth of the RBF and the number of neurons in each layer on initial test runs of 50,000 iterations. We used the median heuristic suggested by Gretton:2012 for the kernel two-sample test to choose the kernel bandwidth. The learning rate was adjusting during optimization by RMSPROP Tieleman2012.

Fig. 2 presents the digits learned after 1,000,000 iterations. We performed minibatch stochastic gradient descent, resampling the generated digits every 300 iterations, using minibatches of size 500, with equal numbers of training and generated points. It is clear that the digits produced have many artifacts not appearing in the MNIST data set. Despite this, the mean log density of the held-out test data is , as compared with the reported mean log density achieved by adversarial nets.

There are several possible explanations for this. First, kernel density estimation is known to perform poorly in high dimensions. Second, the MMD objective can itself be understood as the squared difference of two kernel density estimates, and so, in a sense, the objective being optimized is directly related to the subsequent mean test log density evaluation. There is no clear connection for adversarial networks, which might explain why it suffers under this test. Our experience suggests that the RBF kernel delivers base line performance but that an image-specific kernel, capturing, e.g., shift invariance, might lead to better images.

5.3 Toronto face dataset

We have also trained the generative MMD network on Toronto face dataset (TFD) [TFD]. The parameters were adapted from the MNIST experiment: we also used a 3-hidden-layer sigmoidal MLP with similar architecture (1000, 600, and 1000 units) and RBF kernel for the cost function with the same hyper parameter. The training dataset batch sizes were equal to the number of generated points (500). The generated points were resampled every 500 iterations. The network was optimized for 500,000 iterations.

The samples from the resulting network are plotted in Fig. 3. The mean log density of the held-out test set is 2283 39. Although this figure is higher than the mean log density of 2057 26 reported in [Goodfellow:2014], the samples from the MMD network are again clearly distinguishable from the training dataset. Thus the high test score suggests that kernel density estimation does not perform well at evaluating the performance for these high dimensional datasets.

6 Conclusion

MMD offers a closed form surrogate for the discriminator in adversarial nets framework. After using Bayesian optimization for the parameters, we found that the network outperformed the adversarial network in terms of the density of the held-out test set under kernel density estimation. On the other hand, there is a clear discrepancy between the digits produced by MMD Nets and the MNIST digits, which might suggest that KDE is not up to the task of evaluating these models. Given how quickly MMD Nets achieves this level of performance, it is worth considering its use as an initialization for more costly procedures.

Acknowledgments

The authors would like to thank Bharath Sriperumbudur for technical discussions.

Appendix A Proofs

We begin with some preliminaries and known results:

Definition 2 (Mendelson03).

A random variable is said to be a Rademacher random variable if it takes values in , each with probability .

Definition 3 (Mendelson03).

Let be a probability measure on , and let be a class of uniformly bounded functions on . Then the Rademacher complexity of (with respect to ) is

 RN(F)=EμEσ1,…,σN[1√Nsupf∈F∣∣N∑n=1σnf(Xn)∣∣],

where is a sequence of independent Rademacher random variables, and are independent, -distributed random variables, independent also from .

Theorem 2 (McDiarmid’s Inequality Mendelson03).

Let and assume there exists such that, for all , we have

 supx1,…,xk,x′k,…,xN|f(x1,…,xk,…,xN) −f(x1,…,x′k,…,xN)|≤ck.

Then, for all and independent random variables in ,

 Pr{f(ξ1,…,ξN)−E(f(ξ1,…,ξN))≥ε)}
Theorem 3 ([][Thm. 2.35]Mendelson03).

Let . Assume there exists , such that for all , for some . Then there exists constants depending on only, such that

 RN(F)≤Cpγ12⎧⎪ ⎪⎨⎪ ⎪⎩1if 02.
Theorem 4 (Gretton:2012).

Assume , . Then

 Pr[|MMD2u(H,X,Yθ)−MMD2(H,pdata,pθ)|>ε]≤δε

where

 δε=2exp(−ε2M16K2).

The case where is a finite set is elementary:

Theorem 5 (estimation error for finite parameter set).

Let be the distribution of , with taking values in some finite set . Then, with probability at least , where is defined as in Theorem 4, we have

 MMD2(H,pdata,p^θ)
Proof.

Let

 E(θ)=MMD2u(H,X,Yθ)

and

 T(θ)=MMD2(H,pdata,pθ).

Note, that the upper bound stated in Theorem 4 holds for the parameter value , i.e.,

 Pr[|E(θ∗)−T(θ∗)|>ε]≤δε.

Because depends on the training data and generator data , we use a uniform bound that holds over all . Specifically,

 Pr[|E(^θ)−T(^θ)|>ε] ≤Pr[supθ|E(θ)−T(θ)|>ε] ≤T∑t=1Pr[|E(^θ)−T(^θ)|>ε] ≤Tδε.

This yields that with probability at least ,

Since was chosen to minimize , we know that

 T(^θ)≥T(θ∗).

Similarly, by Section 4,

 E(θ∗)≥E(^θ).

Therefore it follows that

proving the theorem. ∎

Corollary 1.

With probability at least ,

 MMD2(H,pdata,p^θ)

where

 εδ=8K√1Mlog[2(T+1)δ].

In order to prove the general result, we begin with some technical lemmas. The development here owes much to Gretton:2012.

Lemma 1.

Let and

 F+={h=f(y,⋅):f∈F,y∈Y}∩B(L∞(Y)).

Let be -distributed independent random variables in . Assume for some and some , we have , for all . For , define to be

 supf∈F∣∣E(f(Y,Y′))−1N(N−1)∑n≠n′f(yn,yn′)∣∣.

Then there exists a constant that depends on , such that

 E(ρ(Y1,…,YN))≤Cγ12⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1√N−1if p<2√log3(N−1)N−1if p=21(N−1)1pif p>2.
Proof.

Let us introduce , where and have the same distribution and are independent for all . Then the following is true:

 E(f(Y,Y′)) =E(1N(N−1)∑n,n′:n≠n′f(ζn,ζn′))

Using Jensen’s inequality and the independence of and , we have

Introducing conditional expectations allows us to rewrite the equation with the sum over outside the expectations. I.e., Appendix A equals to

The second equality follows by symmetry of random variables . Note that we also added Rademacher random variables before each term in the sum since has the same distribution as for all and therefore the ’s do not affect the expectation of the sum.

Note that and are identically distributed. Thus the triangle inequality implies that Appendix A is less than or equal to

 2N−1E(E(Y)(supf∈F∣∣N−1∑n=1σnf(Y,Yn)∣∣)) ≤2√N−1RN−1(F+),

where is the Rademacher’s complexity of . Then by Theorem 3, we have

 E(ρ(Y1,…,YN))≤Cγ12⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1√N−1if p<2√log3(N−1)N−1if p=21(N−1)1pif p>2.

Lemma 2.

Let and

 F+={f:x×Y→R,x∈X}.

and . Let and be - and -distributed independent random variables in and , respectively. Assume for some , such that for all , , for some . For all , , and all , , define

 ρ(x1,…,xN,y1,…,yM) = supf∈F∣∣E(f(X,Y) −1NM∑n,mf(xn,ym)∣∣.

Then there exists that depends on , such that

 E(ρ(X1,…,XN,Y1,…,YM)) ≤Cγ12⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1√Mif p<2√log3(M)Mif p=21(M)1pif p>2.
Proof.

The proof is very similar to that of Lemma 1. ∎

Proof of Theorem 1.

The proof follows the same steps as the proof of Theorem 5 apart from a stronger uniform bound stated in Appendix A. I.e., we need to show:

 Pr[supθ∈Θ|E(θ)−T(θ)|≥ε]≤δ.

Expanding as defined by Eqs. 2 and 3, and substituting , yields

For all , does not depend on and therefore the first two terms of the equation above can be taken out of the supremum. Also, note that since , we have

 ∣∣ζ(x1,…,xn,…,xN)−ζ(x1,…,x′n,…,xN)∣∣≤2KN,

where

 ζ(x1,…,xN)=1N(N−1)∑n,n′:n′≠nk(xn,xn′),

and is an unbiased estimate of . Then from McDiarmid’s inequality on , we have

Therefore Appendix A is bounded by the sum of the bound on Appendix A and the following:

 supθ∈Θ∣∣E(k(Gθ(W),Gθ(W′)))−1M(M−1)∑m≠m′k(Gθ(Wm),Gθ(Wm′))−2E(k(X,Gθ(W)))+2MN∑m,nk(Xn,Gθ(Wm))∣∣.

Thus the next step is to find the bound for the supremum above.

Define

 f(W1,…,WM;pnoise)=f(W–––M) =supθ∈Θ∣∣E(k(Gθ(W),Gθ(W′))) −1M(M−1)∑m≠m′k(Gθ(Wm),Gθ(Wm′))∣∣

and

 h(X1,…,XN,W1,…,WM;pdata,pnoise) =h(X––N,W–––M) =supθ∈Θ∣∣1MN∑m,nk(Xn,Gθ(Wm))−E(k(X,Gθ(W)))∣∣.

Then by triangle inequality

We will first find the upper bound on , i.e., for every , we will show that there exists , such that

 Pr(f(W–––M)>ε)≤δf

For each ,

since the kernel is bounded by , and therefore is bounded by for all . The conditions of Theorem 2 are satisfied and thus we can use McDiarmid’s Inequality on :

 Pr(f(W–––M)−E(f(W–––M))≥ϵ)≤exp(−ε2M2K2).

Define

 Gk={k(Gθ(⋅),Gθ(⋅)):θ∈Θ}

To show Appendix A, we need to bound the expectation of . We can apply Lemma 1 on the function classes and . The resulting bound is

 E(f(W–––M))≤εp1=Cfγ121⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1√M−1if p1<2√log3(M−1)M−1if p1=21(M−1)1p1if p1>2.,

where and are parameters associated to fat shattering dimension of as stated in the assumptions of the theorem, and is a constant depending on .

Now we can write down the bound on :

 Pr(f(W–––M)≥εp1+ϵ)≤exp(−ε2M2K2)=δf.

Similarly, has bounded differences:

and

McDiarmid’s inequality then implies

 Pr(h(X––N,W–––M)−E(h(X––N,W–––M)≥ε)≤exp(−ε22K2NMN+M).

We can bound expectation of using Lemma 2 applied on and , where

 GXk={k(⋅,Gθ(⋅)):θ∈Θ}.

Then

 E(h(X––N,W–––M))≤εp2=Chγ122⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩1√Mif p2<2√log3(M)Mif p2=21M1p2if p2>2.

for some constant that depends on . The final bound on is then

 Pr(h(X––N,W–––M)≥εp2+ε)≤exp(−ε22K2NMN+M)=δh.

Summing up the bounds from Appendix A and Appendix A, it follows that

Using the bound in Appendix A, we have obtain the uniform bound we were looking for:

 Pr[supθ∈Θ|E(θ)−T(θ)|>εp1+2εp2+4ε]≤δh,

which by Appendix A yields

 Pr[|E(^θ)−T(^θ)|>εp1+2εp2+4ε]≤δh.

Since it was assumed that and , we get

 δh=exp(−ε2M4).

To finish, we proceed as in the proof of Theorem 5. We can rearrange some of the terms to get a different form of Appendix A:

 Pr[|E(