Active Mini-Batch Sampling using Repulsive Point Processes

04/08/2018 ∙ by Cheng Zhang, et al. ∙ KTH Royal Institute of Technology Microsoft Disney Research 0

The convergence speed of stochastic gradient descent (SGD) can be improved by actively selecting mini-batches. We explore sampling schemes where similar data points are less likely to be selected in the same mini-batch. In particular, we prove that such repulsive sampling schemes lowers the variance of the gradient estimator. This generalizes recent work on using Determinantal Point Processes (DPPs) for mini-batch diversification (Zhang et al., 2017) to the broader class of repulsive point processes. We first show that the phenomenon of variance reduction by diversified sampling generalizes in particular to non-stationary point processes. We then show that other point processes may be computationally much more efficient than DPPs. In particular, we propose and investigate Poisson Disk sampling---frequently encountered in the computer graphics community---for this task. We show empirically that our approach improves over standard SGD both in terms of convergence speed as well as final model performance.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 8

page 14

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Stochastic gradient descent (SGD) [3]

is key to modern scalable machine learning. Combined with back-propagation, it forms the foundation to train deep neural networks

[20]. Applied to variational inference [10, 33], it enables the use of probabilistic graphical models on massive data. SGD training has contributed to breakthroughs in many applications [14, 24].

A key limitation for the speed of convergence of SGD algorithms is the stochastic gradient noise. Smaller gradient noise allows for larger learning rates and therefore faster convergence. For this reason, variance reduction for SGD is an active research topic.

Several recent works have shown that the variance of SGD can be reduced when diversifying the data points in a mini-batch based on their features [35, 8, 34]. When data points are coming from similar regions in feature space, their gradient contributions are positively correlated. Diversifying data points by sampling from different regions in feature space de-correlates their gradient contributions, which leads to a better gradient estimation.

Another benefit of actively biasing the mini-batch sampling procedure relates to better model performance [4, 31, 34]. Zhang et al. [34]

biased the data towards a more uniform distribution, upsampling data-points in scarce regions and downsampling data points in dense regions, leading to a better performance during test time. Chen et al.

[5]

showed that training on simple classification tasks first, and later adding more difficult examples, leads to a clear performance gain compared to training on all examples simultaneously. We refer to such schemes which modify the marginal probabilities of each selected data point as

active bias. The above results suggest that utilizing an active bias in mini-batch sampling can result in improved performance without additional computational cost.

In this work, we present a framework for active mini-batch sampling based on repulsive point processes. The idea is simple: we specify a data selection mechanism that actively selects a mini-batch based on features of the data. This mechanism introduces repulsion between the data points, meaning that it suppresses data points with similar features to co-occur in the same mini-batch. We use repulsive point processes for this task. Finally, the chosen mini-batch is used to perform a stochastic gradient step, and this scheme is repeated until convergence.

Our framework generalizes the recently proposed mini-batch diversification based on determinantal point processes (DPP) [34] to a much broader class of repulsive processes, and allows users to encode any preferred active bias with efficient mini-batch sampling algorithms. In more detail, our contributions are as follows:

  1. We propose to use point processes for active mini-batch selection (Section 3).
    We provide a theoretical analysis which shows that mini-batch selection with repulsive point processes may reduce the variance of stochastic gradient descent. The proposed approach can accommodate point processes with adaptive densities and adaptive pair-wise correlations. Thus, we can use it for data subsampling with an active bias.

  2. Going beyond DPPs, we propose a group of more efficient repulsive point processes based on Poisson Disk Sampling (PDS) (Section 4).
    We propose PDS with dart throwing for mini-batch selection. Compared to DPPs, this improves the sampling costs from to merely , where is the number of data points and is the mini-batch size (Section 4.1).
    We propose a dart-throwing method with an adaptive disk size and adaptive densities to sample mini-batches with an active bias (Section 4.2).

  3. We test our proposed method on several machine learning applications (Section 5

    ) from the domains of computer vision and speech recognition. We find increased model performance and faster convergence due to variance reduction.

2 Related Work

In this section, we begin with reviewing the most relevant work on diversified mini-batch sampling. Then, we discuss the benefits of subsampling schemes with an active bias, where data are either reweighed or re-ordered. Finally, we review the relevant aspects of point processes.

Diversified Mini-Batch Sampling

Prior research [35, 8, 34, 32] has shown that sampling diversified mini-batches can reduce the variance of stochastic gradients. It is also the key to overcome the problem of the saturation of the convergence speed in the distributed setting [32]. Diversifying the data is also computationally efficient for large-scale learning problems [35, 8, 34].

Zhang et. al. [34] recently proposed to use DPPs for diversified mini-batch sampling and drew the connection to stratified sampling [35] and clustering-based preprocessing for SGD [8]. A disadvantage of the DPP-approach is the computational overhead. Besides presenting a more general theory, we provide more efficient point processes in this work.

Active Bias

Different types of active bias in subsampling the data can improve the convergence and lead to improved final performance in model training [1, 9, 4, 5, 31]. As summarized in [4]

, self-paced learning biases towards easy examples in the early learning phase. Active-learning, on the other hand, puts more emphasis on uncertain cases, and hard example mining focuses on the difficult-to-classify examples.

Chang et. al.  [4] investigate a supervised setup and sample data points, which have high prediction variance, more often. [9] maintain a desired class distribution during mini-batch sampling. Diversified mini-batch sampling with DPPs [34] re-weights the data towards a more uniform distribution, which improves the final performance when the data set is imbalanced.

The choice of a good active bias depends on the data set and problem at hand. Our proposed method is compatible with different active bias preferences.

Point Processes

Point processes have a long history in physics and mathematics, and are heavily used in computer graphics [23, 28, 11, 26, 17]. DPPs, as a group of point processes, have been introduced and used in the machine learning community in recent years [15, 21, 12].

Other types of point processes have been less explored and used in machine learning. There are many different repulsive point processes, such as PDS, or Gibbs processes, with properties similar to DPPs, but with significantly higher sampling efficiency [11]. Additionally, more flexible point processes with adaptive densities and interactions are well studied in computer graphics [22, 29, 13], but not explored much in the machine learning community. Our proposed framework is based on generic point processes. As one of the most efficient repulsive point processes, we advocate Poisson disk sampling in this paper.

3 Repulsive Point Processes for Variance Reduction

In this section, we first briefly introduce our main idea of using point processes for mini-batch sampling in the context of the problem setting (Section 3.1) and revisit point processes (Section 3.2). We prove that any repulsive point process can lead to reduced gradient variance in SGD (Section 3.3), and discuss the implications of this result. The theoretical analysis in this section leads to multiple practical algorithms in Section 4.

3.1 Problem Setting

(a) Non-Repulsive
(b) Stationary
(c) Non-Stationary
Figure 1: Examples of sampling a subset of data points. We sampled different subsets of 100 points (dark green) from a bigger set of points (light blue), using three different point processes. Panel  1(a) shows a uniformly randomly sampled subset. Panel 1(b) and panel  1(c) show two examples with different repulsive point processes.

Consider a loss function

, where are the model parameters, and indicates the data. In this paper, we consider a modified empirical risk minimization problem  [34]:

(1)

indicates a point process defining a distribution over subsets of the data, which will be specified below. Note that this leads to a potentially biased version of the standard empirical risk [3]..

We optimize Eq. 1 via stochastic gradient descent, which leads to the updates

(2)

, a set of data indices that define the mini-batch. is the gradient estimated from a mini-batch. The data points chosen for each mini-batch are drawn from a point process , which defines probability measures over different mini-batches. Therefore, our scheme generalizes SGD in that the data points in the mini-batch are selected actively, rather than uniformly.

Figure 1 shows examples of subset selection using different point processes. Drawing the data randomly without replacement corresponds to a point process as well, thus standard SGD trivially belongs to the class of algorithms considered here. In this paper, we investigate different point processes and analyze how they improve the performance of different models on empirical data sets.

3.2 Background on Point Processes

Point processes are generative processes of collections of points in some measure space [25, 11]. They can be used to sample subsets of data with various properties, either from continuous spaces or from discrete sets, such as a finite dataset. In this paper, we explore different point processes to sample mini-batches with different statistical properties.

More formally, a point process in can be defined by considering the joint probabilities of finding points generated by this process in infinitesimal volumes. One way to express these probabilities is via product densities. Let denote some arbitrary points in , and infinitesimal spheres centered at these points with volumes . Then the order product density is defined by

where is the joint probability of having a point of the point process in each of the infinitesimal spheres . We can use to generate infinitely many point configurations, each corresponding to e.g. a mini-batch.

For example, DPP defines this probability of sampling a subset as being proportional to the determinant of a kernel matrix. It is thus described by the order product density [18]: , where is the determinant of the sized sub-matrix of kernel matrix C with entries specified by .

For our analysis, we will just need the first and second order product density, which are commonly denoted by , . An important special case of point processes is stationary processes. For such processes, the point distributions generated are translation invariant, where the intensity is a constant.

3.3 Point Processes for Active Mini-Batch Sampling

Recently, Zhang et al.[34] investigated how to utilize a particular type of point process, DPP, for mini-batch diversification. Here, we generalize the theoretical results to arbitrary stochastic point processes, and elaborate on how the resulting formulations can be utilized for SGD based algorithms. This opens the door to exploiting a vast literature on the theory of point processes, and efficient algorithms for sampling.

SGD-based algorithms utilize the estimator for the gradient of the objective (Section 3.1). Each mini-batch, i.e. set of data points in this estimator, can be considered as an instance of an underlying point process . Our goal is to design sampling algorithms for improved learning performance by altering the bias and variance of this gradient estimator.

We first derive a closed form formula for the variance of the gradient estimator for general point processes. We then show that, under mild regularity assumptions, repulsive point processes generally imply variance reduction. For what follows, let denote the gradient of the loss function, and recall that , the mini-batch size.

Theorem 1.

The variance of the gradient estimate in SGD for a general stochastic point process is given by:

(3)
Proof.

In Appendix A.1

Remark 1.

This formula applies to general point processes and hence sampling strategies for mini-batches. It proves that variance only depends on first and second order correlations captured by and , respectively. This provides a simple and convenient tool for analyzing properties of sampling strategies with respect to dataset characteristics for variance control, once only these lower order sampling characteristics are known or estimated by simulation.

Remark 2.

For standard SGD, we have . This is due to the nature of random sampling, where sampling a point is independent of already sampled points. Note that this applies also to adaptive sampling with non-constant . Hence, the term vanishes in SGD. In contrast, we show next that this term may induce a variance reduction for repulsive point processes.

Remark 3.

Repulsive point processes may make the first term in Eq. 3 negative, implying variance reduction. For repulsive point processes, the probability of sampling points that are close to each other is low. Consequently, if and are close, , and the term is negative. This is due to points repelling each other (we will elaborate more on this in the next section). Furthermore, assuming that the loss function is sufficiently smooth in its data argument, the gradients are aligned for close points i.e . This combined implies that close points provide negative contributions to the first integral in Eq. 3. The contributions of points farther apart average out and become negligible due to gradients not being correlated with , which is the case for all current sampling algorithms and the ones we propose in the next section. The negative first term in Eq. 3 leads to variance reduction, for repulsive point processes.

Implications.

This proposed theory allows us to use any point process for mini-batch sampling, such as DPP, finite Gibbs processes, Poisson disk sampling (PDS), and many more [11]. It thus offers many new directions of possible improvement. Foremost, we can choose point processes with a different degree of repulsion [2], and computational efficiency. Furthermore, in this general theory, we are able to adapt the density and alter the pair-wise interactions to encode our preference. In the next section, we propose several practical algorithms utilizing these benefits.

4 Poisson Disk Sampling for Active Mini-batch Sampling

We adapt efficient dart throwing algorithms for fast repulsive and adaptive mini-batch sampling (Section 4.1).

We further extend the algorithm with an adaptive disk size and density (Section 4.2). For supervised setups, we shrink the disc size towards the decision boundary, using mingling indices [11]. This biases towards hard examples and improves classification accuracy.

4.1 Stationary Poisson Disk Sampling

Figure 2: Demonstration of PDS. The black circles of a certain radius mark regions claimed by the collected points (red). For the next iteration, if the newly sampled point falls in any of the circles (points colored in gray), this point will be rejected.

PDS is one type of repulsive point process. It demonstrates stronger local repulsion compared to DPP [2]. Typically, it is implemented with the efficient dart throwing algorithm [16], and provides point arrangements similar to those of DPP, albeit much more efficiently.

This process dictates that the smallest distance between each pair of sample points should be at least with respect to some distance measure . The second order product density for PDS is zero when the distance between two points are smaller than the disk radius , and converges to when the two points are far [26]. Thus, when the points are within distance , and when they are far.

As demonstrated in Figure 2, the basic dart throwing algorithm for PDS works as follows in each iteration: 1) randomly sample a data point; 2) if it is within a distance of any already accepted data point, reject; otherwise, accept the new point. We can also specify the maximum sampling size . This means that we terminate the algorithm when points have been accepted. The computational complexity of PDS with dart throwing111In practice, the number of accepted points can be smaller than the number of considered points in the dataset, as some of them will be eliminated by not satisfying the distance criteria. is . This is much lower than the complexity for k-DPP, where is the number of data points in the dataset. In the rest of the paper, we refer to this version of PDS as “Vanilla PDS”.

4.2 Poisson Disk Sampling with Adaptive Density

To further utilize the potential benefit of our framework, we propose several variations of PDS. In particular, we use mingling index based marked processes. We then propose three variants as explained below: Easy PDS, where only the points far from decision boundaries repulse each other, as well as Dense PDS and Anneal PDS, where we can impose preferences on point densities.

Mingling Index

The mingling index is defined as [11]:

(4)

where indicates the mark of the point . In case of a classification task, the mark is the class label. is the ratio of points with different marks than among its nearest neighbors.

Depending on the mingling index, there are three different situations. Firstly, if , the region around only includes points from the same class. This makes a relatively easy point to classify. This type of points is preferred to be sampled in the early iterations for self-paced learning. Secondly, if , this point may be close to a decision boundary. For variance reduction, we do not need to repulse this type of points. Additionally, sampling this type of points more often may help the model to refine the decision boundaries. Finally, if

is very high, the point is more likely to be a hard negative. In this case, the point is mostly surrounded by points from other classes. On a side note, points with high mingling indices can be interpreted as support vectors

[6].

Adaptive Variants of Poisson Disk Sampling 

Gradients may change drastically when points are close to decision boundaries. Points in this region, thus, violate the assumption in Remark 3. Because of this, our first simple extension, which we call “Easy PDS”, sets the disk radius to when the point has a mingling index , and to if . This means that only easy points (with ) repulse. On average, “Easy PDS” is expected to sample more of the difficult points compared with “Vanilla PDS”.

  Input: data , mini-batch size , mingling index for all data, the parameter for categorical distribution to sample mingling index , disk radius
  repeat
     sample a mingling index
     randomly sample a point with mingling index
     if  is not in disk of any samples then
        insert to
     end if
  until  points have been accepted for the mini-batch
Algorithm 1 Draw throwing for Dense PDS

For many tasks, when the data is highly structured, there are only few data points that are close to the decision boundary. To refine the decision boundary, inspired by hard example mining, we can sample points with a high mingling index more often. We thus propose the “Dense PDS” method summarized in Algorithm 1. Instead of drawing darts randomly, we draw darts based on different mingling indices. The mingling indices can assume values, where is the number of nearest neighbors. We thus can specify a parameter for a categorical distribution to sample mingling indices first. Then we randomly sample a dart which has the given mingling index. In this way, we can encode our preferred density with respect to the mingling index.

It is straightforward to introduce an annealing mechanism in “Dense PDS” by using a different at each iteration . Inspired by self-paced learning, we can give higher density to points with low mingling index in the early iterations, and slowly increase the density of points with high mingling index. We refer this method as “Anneal PDS”.

Note that all our proposed sampling methods only rely on the properties of the given data instead of any model parameters. Thus, they can be easily used as a pre-processing step, prior to the training procedure of the specific model.

5 Experiments

We evaluate the performance of the proposed method in various application scenarios. Our methods show clear improvements compared to baseline methods in each case. We first demonstrate the behavior of different varieties of our method using a synthetic dataset. Secondly, we compare Vanilla PDS with DPP as in [34] on the Oxford flower classification task. Finally, we evaluate our proposed methods with two different tasks with very different properties: image classification with MNIST, and speech command recognition.

(a) Random
(b) Vanilla PDS
(c) Easy PDS
(d) Dense PDS
Figure 3: Comparison of performance using one mini-batch sample distribution on synthetic data. Each experiment is repeated 30 times. The resulting decision boundaries are drawn as transparent black lines and the ground truth decision boundary is shown in yellow. As an example, the points from one of the sampled mini-batches are shown in blue.
(a) k=80 (b) k=150
Figure 4: Oxford Flower Classification with Softmax. PDS has similar performance as DPP for sampling mini-batches. However PDS is more efficient as Table 1 shows. Both methods converges faster than traditional SGD (Baseline).
k 80 150 k-DPP 29.687 189.0303 Fast k-DPP 0.3312 1.8745 PDS 0.0795 0.1657 Table 1: CPU time (sec) to sample one mini-batch. The disk radius is set to half of the median value of the distance measure for PDS. The table confirms that the computational complexity of PDS is , which does not depend on the dataset size , as opposed to for k-DPP. Thus, PDS is able to scale to massive datasets with large mini-batches. More results can be found in the appendix.

Synthetic Data

We evaluate our methods on two-dimensional synthetic datasets to illustrate the behavior of different sampling strategies. Figure 3 shows two classes (green and red dots) separated by a wave-shaped (sine curve) decision boundary (yellow line). Intuitively, it should be favorable to sample diverse subsets and even more beneficial to give more weight to the data points at the decision boundary, i.e., sampling them more often. We sample one mini-batch with batch size using different sampling methods. For each method, we train a neural network classifier with one hidden layer of five units, using a single mini-batch. This model, albeit simple, is sufficient to handle the non-linear decision boundary in the example.

Figure 3 shows the decision boundaries by repeating the experiment 30 times. In order to illustrate the sampling schemes, we also show one example of sampled mini-batch using blue dots. In the random sampling case (Figure 3(a)), we can see that the mini-batch is not a good representation of the original dataset as some random regions are more densely or sparsely sampled. Consequently, the learned decision boundary is very different from the ground truth. Figures 3(b) shows Vanilla PDS. Because of the repulsive effect, the sampled points cover the data space more uniformly and the decision boundary is improved compared to Figure 3(a). In Figure 3(c), we used Easy PDS, where the disk radius adapts with the mingling index of the points. We can see that points close to the decision boundary do not repel each other. This leads to a potentially more refined decision boundary as compared to Figure 3(b). Finally, Dense PDS, shown in Figure 3(d), chooses more samples close to the boundary and leads to the most precise decision boundary with a single mini-batch.

Oxford Flower Classification

We compare our proposed PDS with DPP for mini-batch sampling on the fine-grained classification task as in [34] with the same experimental setting. We use Vanilla PDS (with fixed disk radius) for fair comparison with DPP. Figure 4

shows the test accuracy at the end of each training epoch. We see that sampling with either DPP or PDS leads to accelerated convergence as compared to traditional SGD. With similar performance as using DPP, PDS demonstrates significant improvement on sampling efficiency as shown in Table

1. More experimental results with different parameter settings are presented in the appendix.

Mnist

(a) Mean Performance Comparison
(b) Detailed Comparison
Figure 5: MNIST experiment (10 repetitions). The mean performance for each method is reported in Panel (a). We compared all variations of our proposed methods with two baselines: Baseline SGD and ActiveBias [4]

. All our methods perform better than the baselines. AnnealPDS performs best. For better visualization, Panel (b) shows the mean and standard deviation of our proposed Anneal PDS comparing with two baselines closely.

We further show results for hand-written digit classification on the MNIST dataset [19]. We compare different variations of our method with two baselines: the traditional SGD and ActiveBias [4]. We use half of the training data and the full test data. As detailed in the appendix, with MNIST, data are well clustered and most data points have mingling index

(easy to classify). A standard multi-layer convolutional neural network (CNN) from Tensorflow

222 https://www.tensorflow.org/versions/r0.12/tutorials/mnist/pros/ is used in this experiment with standard experimental settings (details in appendix).

Figure 5(a) shows the test error rate evaluated after each SGD training iteration for different mini-batch sampling methods. All active sampling methods with PDS lead to improved final performance compared to traditional SGD. Vanilla PDS clearly outperforms the baseline method. Easy PDS, performs very similarly to Vanilla PDS with slightly faster convergence in the early iterations. Dense PDS leads to better final performance at the cost of a slight decrease in initial convergence speed. The decision boundary is refined because we prefer non-trivial points during training. Anneal PDS further improves Dense PDS with accelerated convergence in the early iterations. Figure 5(b) shows the performance of test accuracy for Anneal PDS compared with baseline methods in a zoomed view.

We thus conclude that all different variations of PDS obtain better final performance, or conversely, achieve the baseline performance with fewer iterations. With a proper annealing schedule to resemble self-paced learning, we can obtain even more improvement in the final performance.

Speech Command Recognition

In this section, we evaluate our method on a speech command classification task as described in [30]. The classification task includes twelve classes: ten isolated command words , silence, or unknown class.

Figure 6: Speech experiment (10 repetitions). We compare the performance over the validation set of different variations of PDS with traditional SGD for every 50 iterations. Means and standard deviations are shown.

The database consists of 64,727 one-second-long audio recordings. As in [30], for each recording, 40 MFCC features [7] are extracted at 10 msec time intervals resulting in features. We use the TensorFlow implementation333https://www.tensorflow.org/tutorials/audio_recognition with standard settings (see appendix for details). Differently from the MNIST dataset, word classes are not clearly separated and data with different mingling index values are well distributed (see appendix).

Figure 6 shows the accuracy on the validation set evaluated every 50 training iterations. Using Vanilla PDS, the model converges with fewer iterations compared to the traditional random sampling of mini-batches. Easy PDS and Dense PDS show similar improvement since few data have mingling indices in this dataset.

As compared to the MNIST experiment, the gain of Vanilla PDS and Easy PDS is larger in this case since the dataset is more challenging. On the other hand, encouraging more difficult samples has a stronger impact on the MNIST dataset than in the Speech experiment. In all different settings, our mini-batch sampling methods are beneficial for both fast convergence and final model performance.

6 Discussion

In this work, we propose the use of repulsive point processes for active mini-batch sampling. We provide both theoretical and experimental evidence that using repulsive stochastic point processes can reduce the variance of stochastic gradient estimates, which leads to faster convergence. Additionally, our general framework also allows adaptive density and adaptive pair-wise interactions. This leads to further improvements in model performance thanks to balancing the information provided by the input samples, or enhancing the information around decision boundaries.

Our work is mainly focused on similarity measures in input space, which makes the algorithms efficient without additional run-time costs for learning. In future work, we will explore the use of our framework in the gradient space directly. This can potentially lead to even greater variance reduction, but at the same time may introduce higher computational complexity. Additionally, we believe that our proposed method may show even greater advantages in the two-stage framework such as Faster-R-CNN [27], where the second stage network is trained using a subset of region proposals from the first stage.

Finally, for sampling with adaptive density, we mainly use the information from mingling index, which can only be utilized for classification problems. In future work, we would also like to explore other measures such as sequences of annotations or graphs.

References

  • [1] Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in SGD by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015.
  • [2] Christophe Ange Napoléon Biscio, Frédéric Lavancier, et al. Quantifying repulsiveness of determinantal point processes. Bernoulli, 22(4):2001–2028, 2016.
  • [3] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT, pages 177–186. Springer, 2010.
  • [4] Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training a more accurate neural network by emphasizing high variance samples. arXiv preprint arXiv:1704.07433, 2017.
  • [5] Xinlei Chen and Abhinav Gupta.

    Webly supervised learning of convolutional networks.

    In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1431–1439, 2015.
  • [6] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273–297, 1995.
  • [7] Steven B Davis and Paul Mermelstein. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(4):357–366, 1980.
  • [8] Tianfan Fu and Zhihua Zhang. CPSG-MCMC: Clustering-based preprocessing method for stochastic gradient MCMC. In AISTATS, 2017.
  • [9] Siddharth Gopal. Adaptive sampling for SGD by exploiting side information. In International Conference on Machine Learning (ICML), pages 364–372, 2016.
  • [10] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research (JMLR), 14(1):1303–1347, 2013.
  • [11] Janine Illian, Antti Penttinen, Helga Stoyan, and Dietrich Stoyan. Statistical analysis and modelling of spatial point patterns, volume 70. John Wiley & Sons, 2008.
  • [12] Tarun Kathuria, Amit Deshpande, and Pushmeet Kohli. Batched gaussian process bandit optimization via determinantal point processes. In Advances in Neural Information Processing Systems, pages 4206–4214, 2016.
  • [13] Naoki Kita and Kazunori Miyata. Multi-class anisotropic blue noise sampling for discrete element pattern generation. The Visual Computer, 32(6-8):1035–1044, 2016.
  • [14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (NIPS), pages 1097–1105, 2012.
  • [15] Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012.
  • [16] Ares Lagae and Philip Dutré. A comparison of methods for generating poisson disk distributions. In Computer Graphics Forum, volume 27, pages 114–129. Wiley Online Library, 2008.
  • [17] Frédéric Lavancier, Jesper Møller, and Ege Rubak. Determinantal point process models and statistical inference. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77(4):853–877, 2015.
  • [18] Frédéric Lavancier, Jesper Møller, and Ege Holger Rubak. Statistical aspects of determinantal point processes. Technical report, Department of Mathematical Sciences, Aalborg University, 2012.
  • [19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  • [20] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–50. Springer, 1998.
  • [21] Chengtao Li, Stefanie Jegelka, and Suvrit Sra. Efficient sampling for k-determinantal point processes. arXiv preprint arXiv:1509.01618, 2015.
  • [22] Hongwei Li, Li-Yi Wei, Pedro V Sander, and Chi-Wing Fu. Anisotropic blue noise sampling. In ACM Transactions on Graphics (TOG), volume 29, page 167. ACM, 2010.
  • [23] Odile Macchi. The coincidence approach to stochastic point processes. Advances in Applied Probability, 7(1):83–122, 1975.
  • [24] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  • [25] J. Møller and R. P. Waagepetersen. Statistical inference and simulation for spatial point processes. Chapman & Hall/CRC, 2003, 2004.
  • [26] A Cengiz Öztireli and Markus Gross. Analysis and synthesis of point distributions based on pair correlation. ACM Trans. Graph., 31(6):170:1–170:10, November 2012.
  • [27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (NIPS), pages 91–99, 2015.
  • [28] Brian D Ripley. The second-order analysis of stationary point processes. Journal of applied probability, 13(2):255–266, 1976.
  • [29] Riccardo Roveri, A. Cengiz Öztireli, and Markus Gross. General point sampling with adaptive density and correlations. Computer Graphics Forum (Proc. Eurographics), 36(2), 2017.
  • [30] Tara N. Sainath and Carolina Parada. Convolutional neural networks for small-footprint keyword spotting. In Proceedings of Interspeech, September 2015.
  • [31] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In

    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    , June 2016.
  • [32] Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient diversity: a key ingredient for scalable distributed learning. In NIPS Workshop on Optimization for Machine Learning, 2017.
  • [33] Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. arXiv preprint arXiv:1711.05597, 2017.
  • [34] Cheng Zhang, Hedvig Kjellström, and Stephan Mandt. Determinantal point processes for mini-batch diversification. In

    Conference on Uncertainty in Artificial Intelligence (UAI)

    , 2017.
  • [35] Peilin Zhao and Tong Zhang. Accelerating minibatch stochastic gradient descent using stratified sampling. arXiv:1405.3080, 2014.

Appendix A Derivation details

a.1 Derivation detail of variances of estimated Gradients

Preliminary: Campbell’s Theorem

A fundamental operation needed for analyzing point processes is computing expected values of sums of sampled values from functions , where the expectation is computed over different realizations of the point process , and are the points in a given realization. Note that the number of points in a given realization is also random, and hence is often omitted for the sums. This can also be generalized to functions of more than one variable. Campbell’s theorem is a fundamental theorem that relates these sums to integrals of arbitrary functions , and the product densities of . In particular, we will work with sums of the form and . These are given by the following expressions:

(5)

where , and

(6)

for , and under the common assumption [11] that no two points in a process can be at the same location almost surely. In practice, we typically observe sample points in a finite domain . For such cases, the integration domains can be replaced by .

Derivation of Gradient Variances

For our derivations of bias and variance in this paper, we will need the first and second order product densities. The first order product density is given by . It can be shown that the expected number of points of in a set can be written as the integral of this expression: , where is the (random) number of points of the process in set . Hence, measures the local expected density of distributions generated by a point process. It is thus usually called the intensity of and denoted by . Pairwise correlations are captured by the second order product density . The two statistics and are sufficient to exactly express bias and variance of estimators for integrals or expected values.

The expected value of the stochastic gradient , can be computed as follows

(7)

The is fundamentally related to the assumed underlying distribution for the observed data-points. If the sampler does not introduce additional adaptivity, e.g. random sampling, then is directly proportional to .

The variance of a multi-dimensional random variable can be written as

, where denotes the component of . The variance for each dimension is given by . These terms can be written in terms of and by utilizing Equations 6, and 5, respectively, as follows

(8)
(9)

Finally, summing over all dimensions, we can get the following formula for the variance of

(10)

Appendix B Additional Results

Figure 7 shows examples of samples using DPP and using Poisson disk sampling (by dart throwing).

(a) DPP exp 1
(b) DPP exp 2
(c) DPP exp 3
(d) DPP exp 4
(e) Poisson Disk exp 1
(f) Poisson Disk exp 2
(g) Poisson Disk exp 3
(h) Poisson Disk exp 4
Figure 7: Examples of sampled data points. The first row are sampled using DPP and the second row are sampled with Poisson Disk sampling. The green dots are the training data and the black circles indicated that the data point has been sampled.

For Oxford Flower classification experiments, we compare all different experimental settings with [34] here. Figure 8 presents the original results from [34] with four different mini-batch sizes. The feature space is constructed by concatenating the off-the-shelf image feature and one-hot-vector label feature. With such concatenation, the is the weight of label part and (1-) is the weight of off-the-shelf feature part. We perform the experiments under those different settings with our proposed PDS. Figure 9 shows the performance (test accuracy at the end of each training epoch) using Vanilla PDS. Our proposed method with Vanilla PDS exhibits similar performance as DPP in all different experimental settings (as expected), but much more efficiently (as discussed in the paper). However, as shown in the Table 1 in the paper, DPS is much more efficient than DPP. To be noticed, in the main paper, the results are with .

(a) k=50
(b) k=80
(c) k=102
(d) k=150
Figure 8: Result of Oxford Flower Classification with DPP from [34].
(a) k=50
(b) k=80
(c) k=102
(d) k=150
Figure 9: Oxford Flower Classification with Softmax using PDS. The performance is similar as using DPP Figure 8.
k 50 80 102 150 200
k-DPP time 7.1680 29.687 58.4086 189.0303 436.4746
Fast k-DPP 0.1032 0.3312 0.6512 1.8745 4.1964
Poisson Disk 0.0461 0.0795 0.1048 0.1657 0.2391
Table 2: Oxford Flower Classification experiment. CPU time (sec) to sample one mini-batch.
(a) Data Visualization for MNIST
(b) Mingling index distribution
Figure 10: MNIST experiment. Panel (a) visualizes the dataset using tSNE (5000 data points from training data are used for visualization). Panel (b) shows the distribution of data with different mingling index values.
(a) Mean Performance Comparison
(b) ActiveBias (baseline)
(c) VannilaPDS comparison
(d) EasyPDS comparison
(e) DensePDS comparison
(f) AnnealPDS comparison
Figure 11: MNIST experiments performance comparison. Panel (a) shows the mean performance of multiple variants of our methods and two baseline methods. Panel (b) shows the ActiveBias baseline performance in comparison with traditional SGD performance (main baseline). Panel (c) - (f) show the performance of our method compared to the traditional SGD. All the experiments are run ten times. The mean and standard deviations are visualized.

Figure 12 shows the annealing parameter that is used in the MNIST experiment in Section 5. The original distribution of data with different mingling indices is . We uses an annealing schedule where is set to normalized . In this way, data with mingling index 0 (easy examples) are sampled more often in the early iterations during training.

[capbesideposition=right,top,capbesidewidth=8cm]figure[]

Figure 12: The annealing schedule that is used in the ’Anneal PDS’ MNIST experiment in the paper. The density is deceasing for easy examples (with mingling indices ), while increasing for non-trivial examples (with bigger mingling indices).

Figure 14 shows the distributions of data with different mingling index over these twelve classes in the speech experiment. We can see that most points with mingling indices are from the “silence” class where the data are artificially constructed from background noise. Data from other classes, in general, have higher mingling indices, since they are not well separated as shown in the t-SNE plot in the paper. Additionally, we can see in Figure 14 (b) that most data points with mingling index are from the “unknown” class, that is the noisiest class because it corresponds to a number of different words.

(a) Data Visualization
(b) Mingling index distribution
Figure 13: Speech experiment. Panel (a) visualizes the dataset using t-SNE. Panel (b) shows the histogram of data with mingling index . Panel
(a) Mingling index 0
(b) Mingling index 0.2
(c) Mingling index 0.4
(d) Mingling index 0.6
(e) Mingling index 0.8
(f) Mingling index 1.0
Figure 14: Speech experiment. The distribution of data with respect to different class labels under different mingling index values. ’slc’ indicates the silence class, and ’unkn’ indicates the unknown class where the utterance does not belong to any of these ten target command words.