Acknowledgments and Disclosure of Funding
We consider the task of second order optimization in a distributed or parallel setting. Suppose that
workers are each given a small sketch of the data (e.g., a random sample or a random projection) and a parameter vector. The goal of the -th worker is to construct a local estimate of the Newton step relative to a convex loss on the full dataset. The estimates are then averaged and the parameter vector is updated using this averaged step, obtaining . This basic strategy has been extensively studied and it has proven effective for a variety of optimization tasks because of its communication-efficiency . However, a key problem that limits the scalability of this approach is that local estimates of second order steps are typically biased, which means that for a sufficiently large , adding more workers will not lead to any improvement in the convergence rate. Furthermore, for most types of sketched estimates this bias is difficult to compute, or even approximate, which makes it difficult to correct.
In this paper, we propose a new class of sketching methods, called surrogate sketches, which allow us to debias local estimates of the Newton step, thereby making distributed second order optimization more scalable. In our analysis of the surrogate sketches, we exploit recent developments in determinantal point processes (DPPs) to give exact formulas for the bias of the estimates produced with those sketches, enabling us to correct that bias. Due to algorithmic advances in DPP sampling, surrogate sketches can be implemented in time nearly linear in the size of the data, when the number of data points is much larger than their dimensionality, so our results lead to direct improvements in the time complexity of distributed second order optimization. Remarkably, our analysis of the bias of surrogate sketches leads to a simple technique for debiasing the local Newton estimates for -regularized problems, which we call scaled regularization. We show that the regularizer used on the sketched data should be scaled down compared to the global regularizer, and we give an explicit formula for that scaling. Our empirical results demonstrate that scaled regularization significantly reduces the bias of local Newton estimates not only for surrogate sketches, but also for a range of other sketching techniques.
1.1 Debiasing via Surrogate Sketches and Scaled Regularization
Our scaled regularization technique applies to sketching the Newton step over a convex loss, as described in Section 3, however, for concreteness, we describe it here in the context of regularized least squares. Suppose that the data is given in the form of an matrix and an -dimensional vector , where . For a given regularization parameter , our goal is to approximately solve the following problem:
Following the classical sketch-and-solve paradigm, we use a random sketching matrix , where , to replace this large regularized least squares problem with a smaller problem of the same form. We do this by sketching both the matrix and the vector , obtaining the problem given by:
where we deliberately allow to be different than . The question we pose is: What is the right choice of so as to minimize , i.e., the bias of , which will dominate the estimation error in the case of massively parallel averaging? We show that the choice of is controlled by a classical notion of effective dimension for regularized least squares .
Given a matrix and regularization parameter , the -effective dimension of is defined as .
For surrogate sketches, which we define in Section 2, it is in fact possible to bring the bias down to zero and we give an exact formula for the correct that achieves this (see Theorem 6 in Section 3 for a statement which applies more generally to the Newton’s method).
If is constructed using a size surrogate sketch from Definition 3, then:
Thus, the regularization parameter used to compute the local estimates should be smaller than the global regularizer . While somewhat surprising, this observation does align with some prior empirical  and theoretical  results which suggest that random sketching or sampling introduces some amount of implicit regularization. From this point of view, it makes sense that we should compensate for this implicit effect by reducing the amount of explicit regularization being used.
One might assume that the above formula for is a unique property of surrogate sketches. However, we empirically show that our scaled regularization applies much more broadly, by testing it with the standard Gaussian sketch ( has i.i.d. entries ), a Rademacher sketch ( has i.i.d. entries equal or
with probability), and uniform row sampling. In Figure 1, we plot normalized estimates of the bias, , by averaging i.i.d. copies of , as grows to infinity, showing the results with both scaled (dotted curves) and un-scaled (solid curves) regularization. Remarkably, the scaled regularization seems to correct the bias of very effectively for Gaussian and Rademacher sketches as well as for the surrogate sketch, resulting in the estimation error decaying to zero as grows. For uniform sampling, scaled regularization also noticeably reduces the bias. In Section 5 we present experiments on more datasets which further verify these claims.
|||i.i.d. row sample||uniform|
|||i.i.d. row sample||determinantal|
|Thm. 2||surrogate sketch||uniform|
1.2 Convergence Guarantees for Distributed Newton Method
We use the debiasing technique introduced in Section 1.1 to obtain the main technical result of this paper, which gives a convergence and time complexity guarantee for distributed Newton’s method with surrogate sketching. Once again, for concreteness, we present the result here for the regularized least squares problem (1), but a general version for convex losses is given in Section 4 (see Theorem 10). Our goal is to perform a distributed and sketched version of the classical Newton step: , where is the Hessian of the quadratic loss, and is the gradient. To efficiently approximate this step, while avoiding the cost of computing the exact Hessian, we use a distributed version of the so-called Iterative Hessian Sketch (IHS), which replaces the Hessian with a sketched version , but keeps the exact gradient, resulting in the update direction [31, 28, 25, 23]. Our goal is that
should be cheap to construct and it should lead to an unbiased estimate of the exact Newton step. When the matrix is sparse, it is desirable for the algorithm to run in time that depends on the input sparsity, i.e., the number of non-zeros denoted .
Let denote the condition number of the Hessian , let be the initial parameter vector and take any . There is an algorithm which returns a Hessian sketch in time , such that if are i.i.d. copies of then,
with probability enjoys a linear convergence rate given as follows:
Crucially, the linear convergence rate decays to zero as goes to infinity, which is possible because the local estimates of the Newton step produced by the surrogate sketch are unbiased. Just like commonly used sketching techniques, our surrogate sketch can be interpreted as replacing the matrix with a smaller matrix , where is a sketching matrix, with denoting the sketch size. Unlike the Gaussian and Rademacher sketches, the sketch we use is very sparse, since it is designed to only sample and rescale a subset of rows from , which makes the multiplication very fast. Our surrogate sketch has two components: (1) standard i.i.d. row sampling according to the so-called -ridge leverage scores [18, 1]; and (2) non-i.i.d. row sampling according to a determinantal point process (DPP) . While leverage score sampling has been used extensively as a sketching technique for second order methods, it typically leads to biased estimates, so combining it with a DPP is crucial to obtain strong convergence guarantees in the distributed setting. The primary computational costs in constructing the sketch come from estimating the leverage scores and sampling from the DPP.
1.3 Related Work
While there is extensive literature on distributed second order methods, it is useful to first compare to the most directly related approaches. In Table 1, we contrast Theorem 2 with two other results which also analyze variants of the Distributed IHS, with all sketch sizes fixed to . The algorithm of  simply uses an i.i.d. row sampling sketch to approximate the Hessian, and then uniformly averages the estimates. This leads to a bias term in the convergence rate, which can only be reduced by increasing the sketch size. In , this is avoided by performing weighted averaging, instead of uniform, so that the rate decays to zero with increasing
. Similarly as in our work, determinants play a crucial role in correcting the bias, however with significantly different trade-offs. While they avoid having to alter the sketching method, the weighted average introduces a significant amount of variance, which manifests itself through the additional factorin the term . Our surrogate sketch avoids the additional variance factor while maintaining the scalability in . The only trade-off is that the time complexity of the surrogate sketch has a slightly worse polynomial dependence on , and as a result we require the sketch size to be at least , i.e., that . Finally, unlike the other approaches, our method uses a scaled regularization parameter to debias the Newton estimates.
. Distributed averaging has been discussed in the context of linear regression problems in works such as
and studied for ridge regression in. However, unlike our approach, all of these methods suffer from biased local estimates for regularized problems. Our work deals with distributed versions of iterative Hessian sketch and Newton sketch and convergence guarantees for non-distributed version are given in  and . Sketching for constrained and regularized convex programs and minimax optimality has been studied in [30, 39, 35]. Optimal iterative sketching algorithms for least squares problems were investigated in [22, 25, 23, 24, 26]. Bias in distributed averaging has been recently considered in , which provides expressions for regularization parameters for Gaussian sketches. The theoretical analysis of 
assumes identical singular values for the data matrix whereas our results make no such assumption. Finally, our analysis of surrogate sketches builds upon a recent line of works which derive expectation formulas for determinantal point processes in the context of least squares regression[15, 16, 12, 13].
2 Surrogate Sketches
In this section, to motivate our surrogate sketches, we consider several standard sketching techniques and discuss their shortcomings. Our purpose in introducing surrogate sketches is to enable exact analysis of the sketching bias in second order optimization, thereby permitting us to find the optimal hyper-parameters for distributed averaging.
Given an data matrix , we define a standard sketch of as the matrix , where is a random matrix with i.i.d. rows distributed according to measure with identity covariance, rescaled so that . This includes such standard sketches as:
Gaussian sketch: each row of is distributed as .
Rademacher sketch: each entry of is with probability and otherwise.
Row sampling: each row of is , where and .
Here, the row sampling sketch can be uniform (which is common in practice), and it also includes row norm squared sampling and leverage score sampling (which leads to better results), where the distribution depends on the data matrix .
Standard sketches are generally chosen so that the sketched covariance matrix is an unbiased estimator of the full data covariance matrix, . This is ensured by the fact that . However, in certain applications, it is not the data covariance matrix itself that is of primary interest, but rather its inverse. In this case, standard sketching techniques no longer yield unbiased estimators. Our surrogate sketches aim to correct this bias, so that, for example, we can construct an unbiased estimator for the regularized inverse covariance matrix, (given some ). This is important for regularized least squares and second order optimization.
We now give the definition of a surrogate sketch. Consider some -variate measure , and let be the i.i.d. random design of size for , i.e., an random matrix with i.i.d. rows drawn from . Without loss of generality, assume that has identity covariance, so that . In particular, this implies that is a random sketching matrix.
Before we introduce the surrogate sketch, we define a so-called determinantal design (an extension of the definitions proposed by [13, 16]), which uses determinantal rescaling to transform the distribution of into a non-i.i.d. random matrix . The transformation is parameterized by the matrix , the regularization parameter and a parameter which controls the size of the matrix .
Given scalars and a matrix , we define the determinantal design as a random matrix with randomized row-size, so that
We next give the key properties of determinantal designs that make them useful for sketching and second-order optimization. The following lemma is an extension of the results shown for determinantal point processes by .
Let . Then, we have:
The row-size of , denoted by
, is a random variable, and this variable isnot distributed according to , even though can be used to control its expectation. As a result of the determinantal rescaling, the distribution of is shifted towards larger values relative to , so that its expectation becomes:
We can now define the surrogate sketching matrix by rescaling the matrix , similarly to how we defined the standard sketching matrix for .
Let . Moreover, let be the unique positive scalar for which , where . Then, is a surrogate sketching matrix for .
Note that many different surrogate sketches can be defined for a single sketching distribution , depending on the choice of and
. In particular, this means that a surrogate sketching distribution (even when the pre-surrogate i.i.d. distribution is Gaussian or the uniform distribution) always depends on the data matrix, whereas many standard sketches (such as Gaussian and uniform) are oblivious to the data matrix.
Of particular interest to us is the class of surrogate row sampling sketches, i.e. where the probability measure is defined by for . In this case, we can straightforwardly leverage the algorithmic results on sampling from determinantal point processes [10, 11] to obtain efficient algorithms for constructing surrogate sketches.
Given any matrix , and , we can construct the surrogate row sampling sketch with respect to (of any size ) in time .
3 Unbiased Estimates for the Newton Step
Consider a convex minimization problem defined by the following loss function:
where each is a twice differentiable convex function and are the input feature vectors in . For example, if , then we recover the regularized least squares task; and if
, then we recover logistic regression. The Newton’s update for this minimization task can be written as follows:
Newton’s method can be interpreted as solving a regularized least squares problem which is the local approximation of at the current iterate . Thus, with the appropriate choice of matrix (consisting of scaled row vectors ) and vector , the Hessian and gradient can be written as: and . We now consider two general strategies for sketching the Newton step, both of which we discussed in Section 1 for regularized least squares.
We first analyze the classic sketch-and-solve paradigm which has been popularized in the context of least squares, but also applies directly to the Newton’s method. This approach involves constructing sketched versions of both the Hessian and the gradient, by sketching with a random matrix . Crucially, we modify this classic technique by allowing the regularization parameter to be different than in the global problem, obtaining the following sketched version of the Newton step:
Our goal is to obtain an unbiased estimate of the full Newton step, i.e., such that , by combining a surrogate sketch with an appropriately scaled regularization .
We now establish the correct choice of surrogate sketch and scaled regularization to achieve unbiasedness. The following result is a more formal and generalized version of Theorem 1. We let be any distribution that satisfies the assumptions of Definition 3, so that corresponds to any one of the standard sketches discussed in Section 2.
If is constructed using a surrogate sketch of size , then:
3.2 Newton Sketch
We now consider the method referred to as the Newton Sketch [32, 29], which differs from the sketch-and-solve paradigm in that it only sketches the Hessian, whereas the gradient is computed exactly. Note that in the case of least squares, this algorithm exactly reduces to the Iterative Hessian Sketch, which we discussed in Section 1.2. This approach generally leads to more accurate estimates than sketch-and-solve, however it requires exact gradient computation, which in distributed settings often involves an additional communication round. Our Newton Sketch estimate uses the same as for the sketch-and-solve, however it enters the Hessian somewhat differently:
The additional factor comes as a result of using the exact gradient. One way to interpret it is that we are scaling the data matrix instead of the regularization. The following result shows that, with chosen as before, the surrogate Newton Sketch is unbiased.
If is constructed using a surrogate sketch of size , then:
4 Convergence Analysis
Here, we study the convergence guarantees of the surrogate Newton Sketch with distributed averaging. Consider i.i.d. copies of the Hessian sketch defined in Section 3.2. We start by finding an upper bound for the distance between the optimal Newton update and averaged Newton sketch update at the ’th iteration, defined as . We will use Mahalanobis norm as the distance metric. Let denote the Mahalanobis norm, i.e., . The distance between the updates is equal to the distance between the next iterates:
We can bound this quantity in terms of the spectral norm approximation error of as follows:
Note that the second term, , is the exact Newton step. To upper bound the first term, we now focus our discussion on a particular variant of surrogate sketch that we call surrogate leverage score sampling. Leverage score sampling is an i.i.d. row sampling method, i.e., the probability measure is defined by for . Specifically, we consider the so-called -ridge leverage scores which have been used in the context of regularized least squares , where the probabilities must satisfy ( denotes a row of ). Such ’s can be found efficiently using standard random projection techniques [17, 9].
If and we use the surrogate leverage score sampling sketch of size , then the i.i.d. copies of the sketch with probability satisfy:
Note that, crucially, we can invoke the unbiasedness of the Hessian sketch, , so we obtain that with probability at least ,
We now move on to measuring how close the next Newton sketch iterate is to the global optimizer of the loss function . For this part of the analysis, we assume that the Hessian matrix is -Lipschitz.
The Hessian matrix is -Lipschitz continuous, that is, for all and .
Let and be the condition number and smallest eigenvalue of the Hessian
be the condition number and smallest eigenvalue of the Hessian, respectively. The distributed Newton Sketch update constructed using a surrogate leverage score sampling sketch of size and averaged over workers, satisfies:
5 Numerical Results
In this section we present numerical results, with further details provided in Appendix D. Figures 2 and 4 show the estimation error as a function of the number of averaged outputs for the regularized least squares problem discussed in Section 1.1, on Cifar-10 and Boston housing prices datasets, respectively.
Figure 2 illustrates that when the number of averaged outputs is large, rescaling the regularization parameter using the expression , as in Theorem 1, improves on the estimation error for a range of different values. We observe that this is true not only for the surrogate sketch but also for the Gaussian sketch (we also tested the Rademacher sketch, which performed exactly as the Gaussian did). For uniform sampling, rescaling the regularization parameter does not lead to an unbiased estimator, but it significantly reduces the bias in most instances. Figure 4 compares the surrogate row sampling sketch to the standard i.i.d. row sampling used in conjunction with averaging methods suggested by  (unweighted averaging) and  (determinantal averaging), on the Boston housing dataset. We used: , , and sketch size
. We show an average over 100 trials, along with the standard error. We observe that the better theoretical guarantees achieved by the surrogate sketch, as shown in Table1, translate to improved empirical performance.
Figure 3 shows the estimation error against iterations for the distributed Newton sketch algorithm running on a logistic regression problem with regularization on three different binary classification UCI datasets. We observe that the rescaled regularization technique leads to significant speedups in convergence, particularly for Gaussian and surrogate sketches.
We introduced two techniques for debiasing distributed second order methods. First, we defined a family of sketching methods called surrogate sketches, which admit exact bias expressions for local Newton estimates. Second, we proposed scaled regularization, a method for correcting that bias.
This work was partially supported by the National Science Foundation under grant IIS-1838179. Also, MD and MWM acknowledge DARPA, NSF, and ONR for providing partial support of this work.
-  Ahmed El Alaoui and Michael W. Mahoney. Fast randomized kernel ridge regression with statistical guarantees. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 775–783, Montreal, Canada, December 2015.
-  D. Bajović, D. Jakovetić, N. Krejić, and N.K. Jerinkić. Newton-like method with diagonal correction for distributed optimization. SIAM Journal on Optimization, 27(2):1171–1203, 2017.
-  Burak Bartan and Mert Pilanci. Distributed averaging methods for randomized second order optimization. arXiv e-prints, arXiv:2002.06540, 2020.
-  Burak Bartan and Mert Pilanci. Distributed sketching methods for privacy preserving regression. arXiv e-prints, arXiv:2002.06538, 2020.
-  Dennis S. Bernstein. Matrix Mathematics: Theory, Facts, and Formulas. Princeton University Press, second edition, 2011.
-  Julius Borcea, Petter Brändén, and Thomas Liggett. Negative dependence and the geometry of polynomials. Journal of the American Mathematical Society, 22(2):521–567, 2009.
-  Daniele Calandriello, Michał Dereziński, and Michal Valko. Sampling from a -dpp without looking at all items. arXiv e-prints, arXiv:2006.16947, 2020.
-  Clément Canonne. A short note on poisson tail bounds. Technical report, Columbia University, 2017.
-  Kenneth L. Clarkson and David P. Woodruff. Low-rank approximation and regression in input sparsity time. J. ACM, 63(6):54:1–54:45, January 2017.
-  Michał Dereziński. Fast determinantal point processes via distortion-free intermediate sampling. In Proceedings of the 32nd Conference on Learning Theory, 2019.
-  Michał Dereziński, Daniele Calandriello, and Michal Valko. Exact sampling of determinantal point processes with sublinear time preprocessing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 11542–11554. Curran Associates, Inc., 2019.
Michał Dereziński, Kenneth L. Clarkson, Michael W. Mahoney, and
Manfred K. Warmuth.
Minimax experimental design: Bridging the gap between statistical and
worst-case approaches to least squares regression.
In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the
Thirty-Second Conference on Learning Theory, volume 99 of
Proceedings of Machine Learning Research, pages 1050–1069, Phoenix, USA, 25–28 Jun 2019.
-  Michał Dereziński, Feynman Liang, and Michael W. Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. arXiv e-prints, page arXiv:1912.04533, Dec 2019.
-  Michał Dereziński and Michael W Mahoney. Distributed estimation of the inverse hessian by determinantal averaging. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 11401–11411. Curran Associates, Inc., 2019.
-  Michał Dereziński and Manfred K. Warmuth. Reverse iterative volume sampling for linear regression. Journal of Machine Learning Research, 19(23):1–39, 2018.
Michał Dereziński, Manfred K. Warmuth, and Daniel Hsu.
Correcting the bias in least squares regression with volume-rescaled
Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, 2019.
-  Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. J. Mach. Learn. Res., 13(1):3475–3506, December 2012.
-  Petros Drineas, Michael W Mahoney, and S Muthukrishnan. Sampling algorithms for regression and applications. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm, pages 1127–1136. Society for Industrial and Applied Mathematics, 2006.
-  Alex Gittens, Richard Y. Chen, and Joel A. Tropp. The masked sample covariance estimator: an analysis using matrix concentration inequalities. Information and Inference: A Journal of the IMA, 1(1):2–20, 05 2012.
-  Alex Kulesza and Ben Taskar. Determinantal Point Processes for Machine Learning. Now Publishers Inc., Hanover, MA, USA, 2012.
-  Rasmus Kyng and Zhao Song. A matrix chernoff bound for strongly rayleigh distributions and spectral sparsifiers from a few random spanning trees. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 373–384. IEEE, 2018.
-  Jonathan Lacotte, Sifan Liu, Edgar Dobriban, and Mert Pilanci. Limiting spectrum of randomized hadamard transform and optimal iterative sketching methods. arXiv preprint arXiv:2002.00864, 2020.
-  Jonathan Lacotte and Mert Pilanci. Faster least squares optimization. arXiv preprint arXiv:1911.02675, 2019.
-  Jonathan Lacotte and Mert Pilanci. Effective dimension adaptive sketching methods for faster regularized least-squares optimization. arXiv preprint arXiv:2006.05874, 2020.
-  Jonathan Lacotte and Mert Pilanci. Optimal randomized first-order methods for least-squares problems. arXiv preprint arXiv:2002.09488, 2020.
-  Jonathan Lacotte, Mert Pilanci, and Marco Pavone. High-dimensional optimization in adaptive random subspaces. In Advances in Neural Information Processing Systems, pages 10847–10857, 2019.
-  Aryan Mokhtari, Qing Ling, and Alejandro Ribeiro. Network newton distributed optimization methods. Trans. Sig. Proc., 65(1):146–161, January 2017.
-  Ibrahim Kurban Ozaslan, Mert Pilanci, and Orhan Arikan. Regularized momentum iterative hessian sketch for large scale linear system of equations. arXiv preprint arXiv:1912.03514, 2019.
-  Mert Pilanci. Fast Randomized Algorithms for Convex Optimization and Statistical Estimation. PhD thesis, UC Berkeley, 2016.
-  Mert Pilanci and Martin J. Wainwright. Randomized sketches of convex programs with sharp guarantees. IEEE Trans. Info. Theory, 9(61):5096–5115, September 2015.
-  Mert Pilanci and Martin J Wainwright. Iterative hessian sketch: Fast and accurate solution approximation for constrained least-squares. The Journal of Machine Learning Research, 17(1):1842–1879, 2016.
-  Mert Pilanci and Martin J Wainwright. Newton sketch: A near linear-time optimization algorithm with linear-quadratic convergence. SIAM Journal on Optimization, 27(1):205–245, 2017.
-  Sashank J. Reddi, Jakub Konecný, Peter Richtárik, Barnabás Póczós, and Alex Smola. AIDE: Fast and Communication Efficient Distributed Optimization. arXiv e-prints, page arXiv:1608.06879, Aug 2016.
-  Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate Newton-type method. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1000–1008, Bejing, China, 22–24 Jun 2014. PMLR.
-  Srivatsan Sridhar, Mert Pilanci, and Ayfer Özgür. Lower bounds and a near-optimal shrinkage estimator for least squares using random projections. arXiv preprint arXiv:2006.08160, 2020.
-  Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4):389–434, August 2012.
-  Shusen Wang, Alex Gittens, and Michael W. Mahoney. Sketched ridge regression: Optimization perspective, statistical perspective, and model averaging. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3608–3616, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.
-  Shusen Wang, Fred Roosta, Peng Xu, and Michael W Mahoney. Giant: Globally improved approximate newton method for distributed optimization. In Advances in Neural Information Processing Systems 31, pages 2332–2342. Curran Associates, Inc., 2018.
-  Yun Yang, Mert Pilanci, Martin J Wainwright, et al. Randomized sketches for kernels: Fast and optimal nonparametric regression. The Annals of Statistics, 45(3):991–1023, 2017.
-  Yuchen Zhang and Xiao Lin. Disco: Distributed optimization for self-concordant empirical loss. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 362–370, Lille, France, 07–09 Jul 2015. PMLR.
Appendix A Expectation Formulas for Surrogate Sketches
In this section show the expectation formulas given in Lemma 4. First, we derive the normalization constant of the determinantal design introduced in Definition 2. For this, we rely on the framework of determinant preserving random matrices recently introduced by . The proofs here roughly follow the techniques from , the main difference being that we consider regularized matrices, whereas they focus on the unregularized case.
A square random matrix is determinant preserving (d.p.) if taking expectation commutes with computing a determinant for that matrix and all its submatrices. Consider the matrix for an isotropic -variate measure and , as in Definition 2. In Lemma 5,  show that the matrix is determinant preserving. Thus, using closure under addition (Lemma 4 in ), the matrix is also d.p., so the normalization constant for the probability defined in Definition 2 is:
Proof of Lemma 4 By definition, any d.p. matrix satisfies , where denotes the adjugate of a square matrix, which for any positive definite matrix is given by . This allows us to show the second expectation formula from Lemma 4 for . Note that the proof is analogous to the proof of Lemma 11 in .
We next prove the first expectation formula from Lemma 4, by following the steps outlined by  in the proof of their Lemma 13. Let denote any vector in . The th entry of the vector can be obtained by left multiplying it by the th standard basis vector . We will use the following observation (Fact 2.14.2 from ):
Combining this with the fact that both the matrices and are determinant preserving for defined as before, we obtain that:
Since the above holds for all indices and all vectors , this completes the proof.
where in we used both formulas from Lemma 4. This concludes the proof.
The proof of Theorem 7 follows analogously.
Appendix B Efficient Algorithms for Surrogate Sketches
In this section, we provide a framework for implementing surrogate sketches by relying on the algorithmic techniques from the DPP sampling literature. We then use these results to give the input-sparsity time implementation of the surrogate leverage score sampling sketch.
Given a probability measure over domain and a kernel function , we define a determinantal point process as a distribution over finite subsets of , such that for any and event :
If is the set of row vectors of an matrix and , then , with being the uniform measure over , reduces to a standard L-ensemble DPP . In particular, let denote a random subset of sampled so that . Then the set of rows of is distributed identically to .
A key property of L-ensembles, which relates them to the -effective dimension is that if for an isotropic measure and , then .
We next show that our determinantal design (Definition 2) can be decomposed into a DPP portion and an i.i.d. portion, which enables efficient sampling for surrogate sketches. A similar result was previously shown by  for their determinantal design (which is different from ours in that it is not regularized). The below result also immediately leads to the formula for the expected size of a surrogate sketch given in Section 2, which states that .
Let be a probability measure over . Given scalars and a matrix , let , where and for . Then the matrix formed by adding the elements of as rows into the matrix and then randomly permuting the rows of the obtained matrix, is distributed as .
Proof Let be an event measurable with respect to . We have:
which concludes the proof.
We now give an algorithm for sampling a surrogate of the i.i.d. row-sampling sketch where the importance sampling distribution is given by . Here, the probability measure is defined so that for each . The surrogate sketch for this can be constructed as follows:
Sample set so that , i.e., according to .
Draw for and sample i.i.d. from .
Let be a sequence consisting of and randomly permuted together.
Then, the th row of is for .
We next present an implementation of the surrogate row sampling sketch which runs in input-sparsity time for tall matrices (i.e., when ). The algorithm samples exactly from the surrogate sketching distribution, which is crucial for the analysis. Our algorithm is based on two recent papers on DPP sampling [10, 11], however we use a slight modification due to , which ensures exact sampling in input sparsity time.