DPPNet: Approximating Determinantal Point Processes with Deep Networks

01/07/2019 ∙ by Zelda Mariet, et al. ∙ Google MIT 0

Determinantal Point Processes (DPPs) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items. For this reason, they have gained prominence in many machine learning applications that rely on subset selection. However, sampling from a DPP over a ground set of size N is a costly operation, requiring in general an O(N^3) preprocessing cost and an O(Nk^3) sampling cost for subsets of size k. We approach this problem by introducing DPPNets: generative deep models that produce DPP-like samples for arbitrary ground sets. We develop an inhibitive attention mechanism based on transformer networks that captures a notion of dissimilarity between feature vectors. We show theoretically that such an approximation is sensible as it maintains the guarantees of inhibition or dissimilarity that makes DPPs so powerful and unique. Empirically, we demonstrate that samples from our model receive high likelihood under the more expensive DPP alternative.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Selecting a representative sample of data from a large pool of available candidates is an essential step of a large class of machine learning problems: noteworthy examples include automatic summarization, matrix approximation, and minibatch selection. Such problems require sampling schemes that calibrate the tradeoff between the point-wise

qualitye.g. the relevance of a sentence to a document summary – of selected elements and the set-wise diversity111Here, we use diversity to mean useful coverage across dissimilar examples in a meaningful feature space, rather than other definitions diversity that may appear in ML fairness literature. of the sampled set as a whole.Determinantal Point Processes (Dpps) are probabilistic models over subsets of a ground set that elegantly model the tradeoff between these often competing notions of quality and diversity. Given a ground set of size , Dpps allow for sampling over all possible subsets of elements, assigning to any subset of a ground set

of elements the probability

222We are adopting here the -Ensemble construction (Borodin & Rains, 2005) of a Dpp.

(1)

where is the Dpp kernel and denotes the principal submatrix of indexed by items in . Intuitively, Dpps measure the volume spanned by the feature embedding of the items in feature space (Figure 1). First introduced by Macchi (1975) to model the distribution of possible states of fermions obeying the Pauli exclusion principle, the properties of Dpps have since then been studied in depth (Hough et al., 2006; Borodin, 2009, see e.g.). As Dpps capture repulsive forces between similar elements, they arise in many natural processes, such as the distribution of non-intersecting random walks (Johansson, 2004)

, spectra of random matrix ensembles 

(Mehta & Gaudin, 1960; Ginibre, 1965), and zero-crossings of polynomials with Gaussian coefficients (Hough et al., 2009). More recently, Dpps have become a prominent tool in machine learning due to their elegance and tractability: recent applications include video recommendation (Chen et al., 2018), minibatch selection (Zhang et al., 2017), and kernel approximation (Li et al., 2016b; Mariet et al., 2018). However, the sampling cost makes the practical application of Dpps intractable for large datasets, requiring additional work such as subsampling from , structured kernels (Gartrell et al., 2017; Mariet & Sra, 2016b), or approximate sampling methods (Anari et al., 2016a; Li et al., 2016a; Affandi et al., 2013). Nonetheless, even such methods require significant pre-processing time, and scale poorly with the size of the dataset. Furthermore, when dealing with ground sets with variable components, pre-processing costs cannot be amortized, significantly impeding the application of Dpps in practice. These setbacks motivate us to investigate the use of more scalable models to generate high-quality, diverse samples from datasets to obtain highly-scalable methods with flexibility to adapt to constantly changing datasets. Specifically, we use generative deep models to approximate the Dpp

distribution over a ground set of items with both fixed and variable feature representations. We show that a simple, carefully constructed neural network,

DppNet, can generate Dpp-like samples with very little overhead, while maintaining fundamental theoretical properties of Dpp measures. Furthermore, we show that DppNets can be trivially employed to sample from a conditional Dpp (i.e. sampling such that is predefined) and for greedy mode approximation.

(a)(b)(c)
Figure 1: Geometric intuition for Dpps: let be two feature vectors of such that the Dpp kernel verifies ; then . Increasing the norm of a vector (quality) or increasing the angle between the vectors (diversity) increases the volume spanned by the vectors (Kulesza & Taskar, 2012, Section 2.2.1).

Contributions.

  • [leftmargin=*]

  • We introduce DppNet, a deep network trained to generate Dpp-like samples based on static and variable ground sets of possible items.

  • We derive theoretical conditions under which the DppNets inherit the Dpp’s log-submodularity.

  • We show empirically that DppNets provide an accurate approximation to Dpps and drastically speed up Dpp sampling.

2 Related work

Dpps belong to the class of Strongly Rayleigh (SR) measures; these measures benefit from the strongest characterization of negative association between similar items; as such, SR measures have benefited from significant interest in the mathematics community (Pemantle, 2000; Borcea et al., 2009; Borcea & Brändén, 2008; Marcus et al., 2015) and more recently in machine learning (Anari et al., 2016b; Li et al., 2016a; Mariet et al., 2018). This, combined with their tractability, makes Dpps a particularly attractive tool for subset selection in machine learning, and is one of the key motivations for our work. The application of Dpps to machine learning problems spans fields from document and video summarization (Gong et al., 2014; Chao et al., 2015), recommender systems (Zhou et al., 2010; Chen et al., 2018) and object retrieval (Affandi et al., 2014) to kernel approximation (Li et al., 2016b), neural network pruning (Mariet & Sra, 2016a), and minibatch selection (Zhang et al., 2017). Zou & Adams (2012) developed DPP priors for encouraging di

1:
2:function Sample(, )
3:     
4:     while  do
5:          DppNet ()
6:         if sampling then
7:               Multinomial
8:         else if greedy mode then
9:                        
10:               
11:     return
Algorithm 1

Sampling and greedy mode estimation from a

DppNet

diversity in generative models and Snoek et al. (2013) showed that DPPs accurately model inhibition in neural spiking data. In the general case, sampling exactly from a Dpp requires an initial eigendecomposition of the kernel matrix , incurring a cost. In order to avoid this time-consuming step, several approximate sampling methods have been derived; Affandi et al. (2013) approximate the Dpp kernel during sampling; more recently, results by Anari et al. (2016a) followed by Li et al. (2016a) showed that DPPs are amenable to efficient MCMC-based sampling methods.Exact methods that significantly speed up sampling by leveraging specific structure in the Dpp kernel have also been developed (Mariet & Sra, 2016b; Gartrell et al., 2017). Of particular interest is the dual sampling method introduced in Kulesza & Taskar (2012): if the DPP kernel can be composed as an inner product over a finite basis, i.e. there exists a feature matrix such that the Dpp kernel is given by , exact sampling can be done in .However, MCMC sampling requires variable amounts of sampling rounds, which is unfavorable for parallelization; dual Dpp sampling requires an explicit feature matrix . Motivated by recent work on modeling set functions with neural networks (Zaheer et al., 2017; Cotter et al., 2018), we propose instead to generate approximate samples via a generative network; this allows for simple parallelization while simultaneously benefiting from recent improvements in specialized architectures for neural network models (e.g. parallelized matrix multiplications). We furthermore show that, extending the abilities of dual Dpp sampling, neural networks may take as input variable feature matrices and sample from non-linear kernels .

3 Generating Dpp samples with deep models

In this section, we build up a framework that allows the computational cost associated with Dpp sampling to be addressed via approximate sampling with a neural network. Given a positive semi-definite matrix , we take to represent the distribution modeled by a Dpp with kernel over the power set of . Given , we write and the submatrix of indexed by items in .

3.1 Motivating the model choice

Although the elegant quality/diversity tradeoff modeled by Dpps is a key reason for their recent success in many different applications, they benefit from other properties that make them particularly well-suited to machine learning problems. We now focus on how these properties can be maintained with a deep generative model with the right architecture.

Closure under conditioning.

Dpps are closed under conditioning: given , the conditional distribution over given by (for ) is also Dpp with kernel  (Borodin & Rains, 2005). Although conditioning comes at the cost of an expensive matrix inversion, this property make Dpps well-suited to applications requiring diversity in conditioned sets, such as basket completion for recommender systems. Standard deep generative models such as (Variational) Auto-Encoders (Kingma & Welling, 2014) (VAEs) and Generative Adversarial Networks (Goodfellow et al., 2014) (GANs) would not enable simple conditioning operations during sampling. Instead, we develop a model that given an input set , returns a prediction vector such that where : in other words, is the marginal probability of item being included in the final set, given that is a subset of the final set. Mathematically, we can compute as

(2)

for  (Kulesza & Taskar, 2012); for , we simply set . With this architecture, we sample a set via Algorithm 1, which allows for trivial basket-completion type conditioning operations. Furthermore, Algorithm 1 can be modified to implement a greedy sampling algorithm without any additional cost.

Log-submodularity.

As mentioned above, Dpps are included in the larger class of Strongly Rayleigh (SR) measures over subsets. Although being SR is a delicate property, which is maintained by only few operations (Borcea et al., 2009), log-submodularity333Recall that a function over subsets is log-submodular if and only if for any sets , we have . (which is implied by SR-ness) is more robust, as well as a fundamental property in discrete optimization (Zagoruyko & Komodakis, 2016; Buchbinder et al., 2012; Grötschel et al., 1981) and machine learning (Iyer & Bilmes, 2015; Wei et al., 2015). Crucially, we show in the following that (log)-submodularity can be inherited by a generative model trained on a log-submodular distribution:

Theorem 1.

Let be a strictly submodular function over subsets of , and be a function over the same space such that

(3)

where indicates the total variational distance. Then is also submodular.

Corollary 1.1.

Let be a strictly log-submodular Dpp over and DppNet be a network trained on the Dpp probabilities

, with a loss function of the form

where is a norm and (resp. ) is the probability vector assigned by the Dpp (resp. the DppNet) to each subset of . Let . If DppNet converges to a loss smaller than

its generative distribution is log-submodular.

The result follows directly from Thm. 3 and the equivalence of norms in finite dimensional spaces.

Remark 1.

Cor. 1.1 is generalizable to the KL divergence loss via Pinsker’s inequality.

For this reason, we train our models by minimizing the distance between the predicted and target probabilities, rather than optimizing the log-likelihood of generative samples under the true Dpp.

Leveraging the sampling path.

When drawing samples from a Dpp, the standard Dpp sampling algorithm (Kulesza & Taskar, 2012, Alg. 1) generates the sample as a sequence, adding items one after the other until reaching a pre-determined size444When not fixed by the user as in -Dpp(Kulesza & Taskar, 2011), the expected sampled set size under a Dpp depends on the eigenspectrum of the kernel ., similarly to Alg. 1. We take advantage of this by recording all intermediary subsets generated by the Dpp when sampling training data: in practice, instead of training on subsets of size , we train on subsets of size

. Thus, our model is very much like an unrolled recurrent neural network.

3.2 Sampling over fixed and varying ground sets

In the simplest setting, we may wish to draw many samples over a ground set with a fixed feature embedding. In this case, we wish to model a Dpp with a fixed kernel via a generative neural network. Specifically, we consider a fixed Dpp with kernel and wish to obtain a generative model such that its distribution is close (under a chosen metric) to the Dpp distribution .

timesFeed-forwardconnections




Figure 2: Transformer network architecture for sampling from a variable ground set.

More generally, in many cases we may care about sampling from a Dpp over a ground set of items that varies: this may be the case for example with a pool of products that are available for sale at a given time, or social media posts with a relevance that varies based on context. To leverage the speed-up provided by dual Dpp sampling, we can only sample from the Dpp with kernel given by ; for more complex kernels, we once again incur the cost. Furthermore, training a static neural network for each new feature embedding may be too costly. Instead, we augment the static DppNet to include the feature matrix representing the ground set of all items as input to the network. Specifically, we draw inspiration for the dot-product attention introduced in (Vaswani et al., 2017). In the original paper, the attention mechanism takes as input 3 matrices: the keys , the values , and the query . Attention is computed as where is the dimension of each query/key: the inner product acts as a proxy to the similarity between each query and each key. Finally, the reweighted value matrix is fed to the trainable neural network. Here, the feature representation of items in the input set acts as the query ; the feature representation of our ground set is both the keys and the values. In order for the attention mechanism to make sense in the framework of Dpp modeling, we make two modifications to the attention in (Vaswani et al., 2017):

  • [leftmargin=*]

  • We want our network to attend to items that are dissimilar to the query (input subset): for each item in the input subset , we compute its pairwise dissimilarity to each item in as the vector .

  • Instead of returning this matrix of dissimilarities , we return a vector in the probability simplex such that . This allows us to have a fixed-size input to the neural network, and simultaneously enforces the desirable property that similarity to a single item is enough to disqualify an element from the ground set. Note that we could also return in the form of a matrix, but this would be counterproductive to speeding up Dpp sampling.

Putting everything together, our attention vector is computed via the inhibitive attention mechanism

(4)

where represents the row-wise multiplication operator; this vector can be computed in time. The attention component of the neural network finally feeds the element-wise multiplication of each row of with to the feed-forward component. Given and a subset , the network is trained as in the static case to learn the marginal probabilities of adding any item in to under a Dpp with a kernel dependent on . In practice, we set to be an exponentiated quadratic kernel constructed with the features .

Remark 2.

Dual sampling for Dpps as introduced in (Kulesza & Taskar, 2012) is efficient only when sampling from a Dpp with kernel ; for non-linear kernels, a low-rank decomposition of must first be obtained, which in the worst case requires operations. In comparison, the dynamic DppNet can be trained on any Dpp kernel, while only requiring as input.

4 Experimental results

To evaluate DppNet, we look at its performance both as a proxy for a static Dpp (Section 4.1) and as a tool for generating diverse subsets of varying ground sets (Section 4.2

). Our models are trained with TensorFlow, using the Adam optimizer. Hyperparameters are tuned to maximize the normalized log-likelihood of generated subsets. We compare

DppNet to Dpp performance as well as two additional baselines:

  • [leftmargin=*]

  • Unif: Uniform sampling over the ground set.

  • -Medoids: The -medoids clustering algorithm (Hastie et al., 2001, 14.3.10), applied to items in the ground set, with distance between points computed as the same distance metric used by the Dpp. Conversely to -means, -Med uses data points as centers for each cluster.

We use the negative log-likelihood (NLL) of a subset under a Dpp constructed over the ground set to evaluate the subsets obtained by all methods. This choice is motivated by the following considerations: Dpps have become a standard way of measuring and enforcing diversity over subsets of data in machine learning, and b) to the extent of our knowledge, there is no other standard method to benchmark the diversity of a selected subset that depends on specific dataset encodings.

4.1 Sampling over the unit square

We begin by analyzing the performance of a DppNet trained on a Dpp with fixed kernel over the unit square. This is motivated by the need for diverse sampling methods on the unit hypercube, motivated by e.g. quasi-Monte Carlo methods, latin hypercube sampling (McKay et al., 1979) and low discrepancy sequences. The ground set consists of the 100 points lying at the intersections of the grid on the unit square. The Dpp is defined by setting its kernel to . As the Dpp kernel is fixed, these experiments exclude the inhibitive attention mechanism. We report the performance of the different sampling methods in Figure 3. Visually (Figure 2(a)) and quantitively (Figure 2(b)), DppNet improves significantly over all other baselines. The NLL of DppNet samples is almost identical to that of true Dpp samples. Furthermore, greedily sampling the mode from the DppNet achieves a better NLL than Dpp samples themselves. Numerical results are reported in Table 1.

(a) Sampling subsets of size 20 over the unit square.
(b) Normalized log-likelihood of samples drawn from all methods as a function of the sampled set size.
Figure 3: Sampling on the unit square with a DppNet

(1 hidden layer with 841 neurons) trained on a single

Dpp kernel. Visually, DppNet gives similar results to the full Dpp (left). As evaluated by Dpp NLL, the DppNet’s mode achieves superior performance to the full Dpp, and DppNet sampling overlaps completely with Dpp sampling (right).
Uniform -Medoids DppNet Dpp
Table 1: NLL under for sets of size sampled over the unit square. DppNet achieves comparable performance to the Dpp, outperforming the other baselines.

4.2 Sampling over variable ground sets

We evaluate the performance of DppNets on varying ground set sizes through the MNIST (LeCun & Cortes, 2010), CelebA (Liu et al., 2015), and MovieLens (Harper & Konstan, 2015) datasets. For MNIST and CelebA, we generate feature representations of length 32 by training a Variational Auto-Encoder (Kingma & Welling, 2014) on the dataset555Details on the encoders are provided in Appendix B.; for MovieLens, we obtain a feature vector for each movie by applying nonnegative matrix factorization the rating matrix, obtaining features of length 10. Experimental results presented below were obtained using feature representations obtained via the test instances of each dataset. The DppNet is trained based on samples from Dpps with a linear kernel for MovieLens and with an exponentiated quadratic kernel for the image datasets. Bandwidths were set to for MNIST and for CelebA in order to obtain a Dpp average sample size 20: recall that for a Dpp with kernel , the expected sample size is given by the formula

For MNIST, Figure 4 shows images selected by the baselines and the DppNet, chosen among 100 digits with either random labels or all identical labels; visually, DppNet and Dpp samples provide a wider coverage of writing styles. However, the NLL of samples from DppNet decay significantly, whereas the DppNet mode continues to maintain competitive performance with Dpp samples. Numerical results for MNIST are reported in Table 2; additionally to the previous baselines, we also consider two further ways of generating subsets. InhibAttn samples items from the multinomial distribution generated by the inhibitive attention mechanism only (without the subsequent neural network). NoAttn

is a pure feed-forward neural network without attention; after hyper-parameter tuning, we found that the best architecture for this model consisted in 6 layers of 585 neurons each. Table 

2 reveals that both the attention mechanism and the subsequent neural network are crucial to modeling Dpp samples. Strikingly, DppNet performs significantly better than other baselines even on feature matrices drawn from a single class of digits (Table 2), despite the training distribution over feature matrices being much less specialized. This implies that DppNet sampling for dataset summarization may be leveraged to focus on sub-areas of datasets that are identified as areas of interest. Numerical results for CelebA and MovieLens are reported in Table 3, confirming the modeling ability of DppNets.

Figure 4: Digits sampled from a DppNet (3 layer of 365 neurons) trained on MNIST encodings.
All 0 1 2 3 4 5 6 7 8 9
Dpp 49.2 0.1 52.2 0.1 60.5 0.1 49.8 0.0 50.7 0.1 51.0 0.1 50.4 0.1 51.6 0.1 51.5 0.1 50.9 0.1 52.7 0.1
Unif 51.6 0.1 54.9 0.1 65.1 0.1 51.5 0.1 52.9 0.1 53.3 0.1 52.4 0.1 54.6 0.1 55.1 0.1 53.3 0.1 56.2 0.1
Medoids 51.0 0.1 55.1 0.1 65.0 0.1 51.5 0.0 52.9 0.1 53.1 0.1 52.4 0.0 54.4 0.1 55.1 0.1 53.2 0.1 56.1 0.1
InhibAttn 51.3 0.1 54.7 0.1 65.0 0.1 51.4 0.1 52.8 0.1 53.0 0.1 52.1 0.1 54.5 0.1 54.9 0.1 53.2 0.1 55.9 0.1
NoAttn 51.4 0.1 54.9 0.1 65.4 0.1 51.5 0.1 52.9 0.1 53.3 0.1 52.2 0.1 54.6 0.1 55.2 0.1 53.3 0.1 56.1 0.1
DPPNet Mode 48.6 0.2 53.6 0.3 63.6 0.4 50.8 0.2 51.4 0.3 51.6 0.4 51.8 0.3 52.8 0.3 52.7 0.4 50.9 0.3 55.0 0.4
Table 2: NLL (mean standard error) under the true Dpp of samples drawn uniformly, according to the mode of the DppNet, and from the Dpp itself. We sample subsets of size 20; for each class of digits we build 25 feature matrices from encodings of those digits, and for each feature matrix we draw 25 different samples. Bolded numbers indicate the best-performing (non-Dpp) sampling method.
Dataset Kernel Dpp Goal Uniform -Medoids DppNet Mode
CelebA Exp. quadratic 49.04 2.03 50.84 1.53 51.18 1.34 49.28 1.57
MovieLens Linear 84.29 0.20 92.04 0.17 88.90 0.16 80.21 0.33
Table 3: NLLs on CelebA and MovieLens (mean standard error); 20 samples of size 20 were drawn across 20 different feature matrices each for a total of 100 samples per method; DppNet is the non-Dpp model achieves the best NLLs.

Finally, we verify that DppNet allows for significantly faster sampling by running Dpp and DppNet sampling for subsets of size 20 drawn from a ground set of size 100 with both a standard Dpp and DppNet (using the MNIST architecture). Both methods were implemented in graph-mode TensorFlow. Sampling batches of size 32, standard Dpp sampling costs 2.74 0.02 seconds; DppNet sampling takes 0.10 0.001 seconds, amounting to an almost 30-fold speed improvement.

4.3 DppNet for kernel reconstruction

(a) Error
(b) Wallclock time
Figure 5: Results for the Nyström approximation experiments, comparing DppNet to the fast MCMC sampling method of Li et al. (2016c) according to root mean squared error (RMSE) and wallclock time. We observe that subsets selected by DPPnet achieve comparable and lower RMSE than a DPP and the MCMC method respectively while being significantly faster.

As a final experiment, we evaluate DppNet’s performance on a downstream task for which Dpps have been shown to be useful: kernel reconstruction using the Nyström method (Nyström, 1930; Williams & Seeger, 2001). Given a positive semidefinite matrix , the Nyström method approximates by where denotes the pseudoinverse of and (resp. ) is the submatrix of formed by its rows (resp. columns) indexed by . The Nyström method is a popular method to scale up kernel methods and has found many applications in machine learning (see e.g. (Bac, ; She, ; Fow, ; Tal, )). Importantly, the approximation quality directly depends on the choice of subset . Recently, Dpps have been shown to be a competitive approach for selecting  (Mariet et al., 2018; Li et al., 2016b). Following the approach of Li et al. (2016b), we evaluate the quality of the kernel reconstruction by learning a regression kernel on a training set, and reporting the prediction error on the test set using the Nyström reconstructed kernel . Additionally to the full Dpp, we also compare DppNet to the MCMC sampling method with quadrature acceleration (Li et al., 2016b, c) Figure 5 reports our results on the Ailerons dataset666http://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html also used in Li et al. (2016b). We start with a ground set size of 1000 and compute the resulting root mean squared error (RMSE) of the regression using various sized subsets selected by sampling from a DPP, the MCMC method of Li et al. (2016c), using the full ground set and DPPnet. Figure 4(b) reports the runtimes for each method. We note that while all methods were run on CPU, DPPNet is more amenable to acceleration using GPUs.

5 Conclusion and discussion

We introduced DppNets, generative networks trained on Dpps over static and varying ground sets which enable fast and modular sampling in a wide variety of scenarios. We showed experimentally on several datasets and standard Dpp applications that DppNets obtain competitive performance as evaluated in terms of NLLs, while being amenable to the extensive recent advances in speeding up computation for neural network architectures. Although we trained our models on Dpps on exponentiated quadratic and linear kernels; we can train on any kernel type built from a feature representations of the dataset. This is not the case for dual Dpp exact sampling, which requires that the Dpp kernel be for faster sampling. DppNets are not exchangeable: that is, two sequences and where is a permutation of , which represent the same set of items, will not in general have the same probability under a DppNet. Exchangeability can be enforced by leveraging previous work (Zaheer et al., 2017); however, non-exchangeability can be an asset when sampling a ranking of items. Our models are trained to take as input a fixed-size subset representation; we aim to investigate the ability to take a variable-length encoding as input as future work. The scaling of the DppNet’s complexity with the ground set size also remains an open question. However, standard tricks to enforce fixed-size ground sets such as sub-sampling from the dataset may be applied to DppNets. Similarly, if further speedups are necessary, sub-sampling from the ground set – a standard approach for Dpp sampling over very large set sizes – can be combined with DppNet sampling. In light of our results on dataset sampling, the question of whether encoders can be trained to produce encodings conducive to dataset summarization via DppNets seems of particular interest. Assuming knowledge of the (encoding-independent) relative diversity of a large quantity of subsets, an end-to-end training of the encoder and the DppNet simultaneously may yield interesting results. Finally, although Corollary 1.1 shows the log-submodularity of the Dpp can be transferred to a generative model, understanding which additional properties of training distributions may be conserved through careful training remains an open question which we believe to be of high significance to the machine learning community in general.

References

  • (1)
  • (2)
  • (3)
  • (4)
  • Affandi et al. (2013) Raja Hafiz Affandi, Alex Kulesza, Emily Fox, and Ben Taskar. Nystrom approximation for large-scale determinantal processes. In Artificial Intelligence and Statistics, 2013.
  • Affandi et al. (2014) Raja Hafiz Affandi, Emily B. Fox, Ryan P. Adams, and Benjamin Taskar. Learning the parameters of Determinantal Point Process kernels. In International Conference on Machine Learning, 2014.
  • Anari et al. (2016a) Nima Anari, Shayan Oveis Gharan, and Alireza Rezaei.

    Monte Carlo Markov Chain algorithms for sampling Strongly Rayleigh distributions and Determinantal Point Processes.

    In Conference on Learning Theory, 2016a.
  • Anari et al. (2016b) Nima Anari, Shayan Oveis Gharan, and Alireza Rezaei. Monte carlo markov chain algorithms for sampling strongly rayleigh distributions and determinantal point processes. In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir (eds.), Conference on Learning Theory, 2016b.
  • Borcea & Brändén (2008) Julius Borcea and Petter Brändén. Applications of stable polynomials to mixed determinants: Johnson’s conjectures, unimodality, and symmetrized Fischer products. Duke Math. J., 143(2):205–223, 2008.
  • Borcea et al. (2009) Julius Borcea, Petter Brändén, and Thomas M. Liggett. Negative dependence and the geometry of polynomials. Journal of the American Mathematical Society, 22(2):521–567, 2009.
  • Borodin (2009) Alexei Borodin. Determinantal Point Processes, 2009.
  • Borodin & Rains (2005) Alexei Borodin and Eric M. Rains. Eynard–Mehta theorem, Schur process, and their Pfaffian analogs. Journal of Statistical Physics, 121(3):291–317, Nov 2005.
  • Buchbinder et al. (2012) Niv Buchbinder, Moran Feldman, Joseph (Seffi) Naor, and Roy Schwartz. A tight linear time (1/2)-approximation for unconstrained submodular maximization. In Symposium on Foundations of Computer Science, 2012.
  • Chao et al. (2015) Wei-Lun Chao, Boqing Gong, Kristen Grauman, and Fei Sha. Large-margin Determinantal Point Processes. In Uncertainty in Artificial Intelligence, 2015.
  • Chen et al. (2018) Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy MAP inference for Determinantal Point Process to improve recommendation diversity. In Advances in Neural Information Processing Systems, 2018.
  • Cotter et al. (2018) Andrew Cotter, Maya R. Gupta, Heinrich Jiang, James Muller, Taman Narayan, Serena Wang, and Tao Zhu. Interpretable set functions. CoRR, abs/1806.00050, 2018.
  • Gartrell et al. (2017) Mike Gartrell, Ulrich Paquet, and Noam Koenigstein. Low-rank factorization of Determinantal Point Processes. In AAAI Conference on Artificial Intelligence, 2017.
  • Ginibre (1965) J. Ginibre. Statistical ensembles of complex: Quaternion, and real matrices. Journal of Mathematical Physics (New York) (U.S.), 6, 3 1965.
  • Gong et al. (2014) Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha. Diverse sequential subset selection for supervised video summarization. In Advances in Neural Information Processing Systems, 2014.
  • Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
  • Grötschel et al. (1981) Martin Grötschel, László Lovász, and Alexander Schrijver.

    The ellipsoid method and its consequences in combinatorial optimization.

    Combinatorica, 1(2):169–197, 1981.
  • Harper & Konstan (2015) F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4):19:1–19:19, December 2015.
  • Hastie et al. (2001) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
  • Hough et al. (2006) J. Ben Hough, Manjunath Krishnapur, Yuval Peres, and Bálint Virág. Determinantal processes and independence. Probab. Surveys, 3:206–229, 2006.
  • Hough et al. (2009) J.B. Hough, M. Krishnapur, Y. Peres, and B. Vir´ag. Zeros of Gaussian Analytic Functions and Determinantal Point Processes. American Mathematical Society, 2009.
  • Iyer & Bilmes (2015) Rishabh Iyer and Jeffrey Bilmes. Submodular Point Processes with Applications to Machine learning. In International Conference on Artificial Intelligence and Statistics, 2015.
  • Johansson (2004) Kurt Johansson.

    Determinantal processes with number variance saturation.

    Communications in Mathematical Physics, 252(1):111–148, Dec 2004.
  • Kingma & Welling (2014) D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
  • Kulesza & Taskar (2011) Alex Kulesza and Ben Taskar. k-DPPs: Fixed-size Determinantal Point Processes. In Proceedings of the 28th International Conference on Machine Learning, 2011.
  • Kulesza & Taskar (2012) Alex Kulesza and Ben Taskar. Determinantal Point Processes for Machine Learning. Now Publishers Inc., Hanover, MA, USA, 2012.
  • LeCun & Cortes (2010) Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
  • Li et al. (2016a) Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Fast mixing Markov chains for Strongly Rayleigh measures, DPPs, and constrained sampling. In Advances in Neural Information Processing Systems, 2016a.
  • Li et al. (2016b) Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Fast DPP sampling for Nyström with application to kernel methods. In International Conference on Machine Learning, 2016b.
  • Li et al. (2016c) Chengtao Li, Suvrit Sra, and Stefanie Jegelka. Gaussian quadrature for matrix inverse forms with applications. In International Conference on Machine Learning, 2016c.
  • Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In

    International Conference on Computer Vision

    , December 2015.
  • Macchi (1975) Odile Macchi. The coincidence approach to stochastic point processes. Advances in Applied Probability, 7:83–122, 03 1975.
  • Marcus et al. (2015) Adam Marcus, Daniel Spielman, and Nikhil Srivastava. Interlacing families II: Mixed characteristic polynomials and the Kadison–Singer problem. Annals of Mathematics, pp. 327–350, July 2015.
  • Mariet & Sra (2016a) Zelda Mariet and Suvrit Sra. Diversity networks: Neural network compression using Determinantal Point Processes. In International Conference on Learning Representations, 2016a.
  • Mariet & Sra (2016b) Zelda Mariet and Suvrit Sra. Kronecker Determinantal Point Processes. In Advances in Neural Information Processing Systems. 2016b.
  • Mariet et al. (2018) Zelda Mariet, Suvrit Sra, and Stefanie Jegelka. Exponentiated Strongly Rayleigh distributions. In Advances in Neural Information Processing Systems 31. 2018.
  • McKay et al. (1979) M. D. McKay, R. J. Beckman, and W. J. Conover. Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2):239–245, 1979.
  • Mehta & Gaudin (1960) M.L. Mehta and M. Gaudin.

    On the density of eigenvalues of a random matrix.

    Nuclear Physics, 18:420 – 427, 1960.
  • Nyström (1930) E.J. Nyström. über die praktische auflösung von integralgleichungen mit anwendungen auf randwertaufgaben. Acta Mathematica, 54(1):185–204, 1930. ISSN 0001-5962.
  • Pemantle (2000) Robin Pemantle. Towards a theory of negative dependence. 41(3):1371–1390, March 2000. ISSN 0022-2488 (print), 1089-7658 (electronic), 1527-2427.
  • Snoek et al. (2013) Jasper Snoek, Richard S. Zemel, and Ryan Prescott Adams. A Determinantal Point Process latent variable model for inhibition in neural spiking data. In Advances in Neural Information Processing Systems, 2013.
  • Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. 2017.
  • Wei et al. (2015) Kai Wei, Rishabh Iyer, and Jeff Bilmes.

    Submodularity in data subset selection and active learning.

    In International Conference on Machine Learning, 2015.
  • Williams & Seeger (2001) Christopher K. I. Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In T. K. Leen, T. G. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Systems 13, pp. 682–688. MIT Press, 2001.
  • Zagoruyko & Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
  • Zaheer et al. (2017) Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan R. Salakhutdinov, and Alexander J. Smola. Deep sets. In Advances in Neural Information Processing Systems, 2017.
  • Zhang et al. (2017) Cheng Zhang, Hedvig Kjellström, and Stephan Mandt. Stochastic learning on imbalanced data: Determinantal Point Processes for mini-batch diversification. In Uncertainty in Artificial Intelligence, 2017.
  • Zhou et al. (2010) T. Zhou, Z. Kuscsik, J.G. Liu, M. Medo, J.R. Wakeling, and Y.C. Zhang. Solving the apparent diversity-accuracy dilemma of recommender systems. Proceedings of the National Academy of Sciences, 107(10):4511–4515, 2010.
  • Zou & Adams (2012) James Zou and Ryan Prescott Adams. Priors for diversity in generative latent variable models. In Advances in Neural Information Processing Systems, 2012.

Appendix A Maintaining log-submodularity in the generative model

Theorem 2.

Let be a strictly submodular distribution over subsets of a ground set , and be a distribution over the same space such that

(5)

Then is also submodular.

Proof.

In all the following, we assume that are subsets of a ground set such that and (the inequalities being immediate in these corner cases). Let

By the strict submodularity hypothesis, we know . Let such that and . To show the log-submodularity of , it suffices to show that

By definition of ,

From equation 5, we know that

It follows that

which proves the submodularity of . ∎

Appendix B Encoder details

For the MNIST encodings, the VAE encoder consists of a 2d-convolutional layer with 64 filters of height and width 4 and strides of 2, followed by a 2d convolution layer with 128 filters (same height, width and strides), then by a dense layer of 1024 neurons. The encodings are of length 32.

Figure 6: Digits and VAE reconstructions from the MNIST training set

CelebA encodings were generated by a VAE using a Wide Residual Network (Zagoruyko & Komodakis, 2016) encoder with 10 layers and filter-multiplier , a latent space of 32 full-covariance Gaussians, and a deconvolutional decoder trained end-to-end using an ELBO loss. In detail, the decoder architecture consists of a 16K dense layer followed by a sequence of convolutions with filters interleaved with upsampling layers and a final convolution with output channels for each of 5 components in a mixture of quantized logistic distributions representing the decoded image.