Detecting dependencies between two random vectors and
is a fundamental problem in statistics and machine learning. Dependence measures such as Pearson’s correlation, Spearman’s, and Kendall’s are used in almost all quantitative areas; example areas are bioinformatics (Guo et al., 2014; Sferra et al., 2017) and time-series (Zhou, 2012). However, those classical dependence measures are usually designed to detect one specific dependence structure such as a monotonic or linear structure. It is easy to construct highly dependent and whose dependence cannot be detected by classical dependence measures. To overcome these limitations, Székely et al. (2007); Székely and Rizzo (2009) proposed distance covariance as a weighted distance between the joint characteristic function and the product of marginal characteristic distributions. The distance covariance is if and only if two random vectors and are independent. A closely related measure is the Hilbert-Schmidt independence measure (HSIC). HSIC has been extensively studied in machine learning literature (Gretton et al., 2005, 2008; Pfister et al., 2018). Sejdinovic et al. (2013) established equivalence of distance covariance with HSIC.
Despite the power of sample distance covariance to detect a dependence structure, its use for large sample sizes, is inhibited by the high computational cost required. The sample distance covariance and HSIC computation typically requires pairwise distance (kernel) calculations and memory for storing them. This is undesirable and greatly limits the application of distance correlation to large datasets. In the era of big data, it is not rare to see data that consists of millions of observations. For such data, an algorithm is almost impossible to run on a personal computer. To approximate the distance covariance or HSIC for large data, Nyström approach or the random Fourier feature method is often adopted. However, the use of these approximations leads to a reduction in power (Zhang et al., 2018). In this article, we describe an exact method to compute the sample distance covariance between two univariate random variables with computational cost and memory cost . Our proposed method essentially consists of just two sorting steps, which makes it easy to implement.
A closely related algorithm for sample distance covariance was proposed by Huo and Székely (2016). Our algorithm differs from Huo and Székely (2016) in the following ways: First, they implicitly assume that there are no ties in the data (see Algorithm 1 and proof in Huo and Székely (2016)), whereas our proposed method is valid for any pair of real-valued univariate variables. In practice, it is common to see datasets with ties, especially for discrete variables or bootstrap sample. Second, we use a merge sort instead of an AVL tree-type implementation to compute the Frobenius inner product of the distance matrices of and . Empirical results show that our proposed method is significantly faster; for example, for one million observations our MATLAB implementation runs 10 times faster (finishing in 4 seconds) on our desktop, whereas the implementation in Huo and Székely (2016) requires 40 seconds. Because our implementation consists only of MATLAB code while the key step in the Huo and Székely (2016) routine is implemented in C, even greater speed increases are possible by rewriting the critical parts of our implementation in C.
The rest of paper is organized as follows. In Section 2
, we briefly introduce the definition of distance covariance and its sample estimate. In Section3, we describe the proposed algorithm for sample distance covariance. In Section 4, experiment results are presented. Finally, conclusions and remarks are made in Section 5.
2 Some Preliminaries
Denote the joint characteristic function of and as , and denote the marginal characteristic functions of and as and , respectively. Denote as the Euclidean norm in . The squared distance covariance is defined as the weighted distance between and ,
where , and are constants. It is obvious that if and only if and are independent. Under some mild conditions, the squared distance covariance can be defined equivalently as the expectation of Euclidean distances
are identical independent copies from the joint distribution of.
The squared distance correlation is defined by
Let and be the sample collected. Define
The squared sample distance covariance between and is
which is similar in form to (2). The squared sample distance correlation is given by
From (3), it is easy to see a brute force algorithm exists for distance covariance. However, the brute force implementation is difficult to handle large datasets. Moreover, the p-value of distance covariance or correlation is typically calculated by using permutation test, which makes it more computationally intensive.
If we can compute and all for and in steps, then we can also compute in steps.
In this paper, we consider the case where and are univariate random variables; that is, . For the rest of this document we assume that (because after an sort step, we can ensure that ).
3 Fast Algorithm for Distance Covariance
Define the function as
For any two real and we have
We use (6) extensively in the rest of paper.
3.1 Fast computation of the and
Define for and note that can be computed in time.
Since we have
So can be computed in time.
We can use an sorting algorithm to determine a permutation of such that . Therefore as in (7), can be computed in time after is sorted.
3.2 Fast computation of D
In this subsection, we describe an algorithm for computing . First, we have
In (3.2) note that if , thus showing that
Let and . Now the second term in (3.2) is
Therefore it can be computed in steps.
Define for The first term in (3.2) can be easily expressed in terms of . Note that
Thus the first term in (3.2) is expanded into a sum of four terms, each of which is of the form
For any , define If it can be shown that all of can be computed in time, then the sample distance covariance can be computed in time. We will show this in the next subsection. The preceding arguments lead to the following theorem.
For any real-valued univariate variables with sample and , the sample distance covariance can be computed in time.
3.3 Fast computation of where
We are given a series along with weights . Define . Our objective is to compute in steps where
It is well known that the number of inversions in a permutation can be obtained by a merge sort (Ginat, 2004). We use a similar strategy to compute the while performing a merge sort on to sort the in decreasing order.
Merge sort works by successively merging sorted subarrays until the final array is sorted. Assume that we also keep an auxiliary array for storing the original indices of each element of the array, another auxiliary array for storing partially computed (say ), and third one for storing the partial sums of the intermediate results.
A merge sort makes passes over the data. At the beginning of the pass, the array consists of contiguous subarrays, each of which is sorted in decreasing order. We merge two consecutive subarrays times.
Let and be two such consecutive subarrays with and . We can also assume for all meaningful (here and refer to the original indices of these elements). During the merge step, if we notice that for some , then are all the terms in that are less than and we increment by . Note that if we also store running sums as we output the results, can be computed using just one difference.
The extra computation of maintaining the additional auxiliary arrays does not increase the order of computation. The detailed algorithm is presented in Algorithm 1. For a better understanding of the proposed algorithm, we also include MATLAB code in the Appendix.
We illustrate the proposed algorithm by using a simple series,
and weight . The iteration history is presented in Table 1. The index row in Table 1 denotes the original order of the series, and the cusumT row denotes the cumulative sum of . The algorithm finishes in iterations and outputs the results . Consider the computation of and for example. At iteration 1, we merge , because we exchange and and set and . At iteration 2, we merge and ; and because and we increment and because we increment . At the final merge, we note so we increment , where is calculated from the difference of cumulative sums of . Similarly for we have
The proposed algorithm has the same order of computational cost as a merge sort. Denote the computational cost as . We know . It is trivial to show that the complexity is by using the master theorem (Cormen et al., 2001). As a byproduct of calculating all , are sorted. Therefore, calculating all as in Section 3.1 takes only extra time.
As discussed previously, the proposed algorithm essentially consists of two sorting steps. First we sort X and calculate for . Then we sort Y and calculate and all for . The correctness of the described algorithm can be easily concluded from the previous discussion. To verify the correctness of our implementation we matched our numbers with a simple brute force implementation and we confirmed that the numbers match exactly.
We also note another factor that impacts performance. A non recursive merge sort algorithm makes passes over the data, and in the pass makes merges of two subarrays of size . If is large then for small we end up merging a large number of small subarrays, which has a large overhead. We have observed that the speed increases by a factor of if we replace these initial merges by a single insert sort step — that is, we divide the input array into groups of length and sort each one of them using insert sort, store the intermediate values, and then continue with the merge steps with . A MATLAB implementation is provided as supplemental material to this paper.
In this section, we compare the speed of the proposed fast algorithm with the dyadic updating method (Huo and Székely, 2016) and also with the brute force implementation. We implemented the proposed algorithm and the brute force method in MATLAB. For the dyadic updating method, we use the authors’ MATLAB implementation, in which the key step is implemented in C. Therefore, this comparison strongly favors the dyadic updating method because more than of its calculations are done in C instead of MATLAB according to the MATLAB code profiler. All simulations are run on a PC with Intel® Xeon® Gold CPU @ 2.40GHZ processor and 16GB memory running MATLAB version 22.214.171.1241360 (R2016a).
The data are generated from a simple quadratic model, for , where and are i.i.d. standard normal. For each sample size , average running time is calculated based on 10 generated samples. The speed comparison results are presented in Table 2. The column Merge Sort 2 in the table corresponds to proposed algorithm with initial merge steps replaced by insertion sort. For a moderately sized data (for example ), the dyadic update method is about 25 times faster than the brute force implementation, whereas our proposed algorithm is about 200 times faster. Since brute force implementation requires space, no results are available when . For large data (), proposed method is 7 to 10 times faster than dyadic updating. Greater speed increases are expected if our proposed algorithm is implemented in C. Note that the p-value of the distance covariance is typically obtained by a permutation test, which is computationally intensive. For permutations and , the brute force implementation takes around seconds ( minutes) versus seconds (0.8 minutes) or seconds (0.4 minutes) for proposed implementation.
|n||Brute Force||Dyadic Update||Merge Sort||Merge Sort 2|
Average time in seconds for 10 replications. The numbers in parentheses are the estimated standard deviation of running time. No results are available for brute force implementation whenbecause the required memory exceeds the maximum memory size. Column Merge Sort 2 corresponds to proposed algorithm with initial merge steps replaced by insertion sort.
In this paper, we presented an algorithm for calculating the sample distance covariance for univariate variables. For multivariate random variables, random projection approach can be adopted (Huang and Huo, 2017), which depends on calculation of distance covariance for univariate variables. The proposed algorithm is intuitive and simple to implement. Empirical results show that it outperforms existing methods in the literature. Our algorithm will speed up any computational technique that depends on the calculation of univariate distance covariance for example, feature screening (Li et al., 2012).
The proposed faster algorithm provides a tool for scientists to explore complicated dependence structures using distance covariance in larger data than what was previously possible.
- Cormen et al. (2001) Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. (2001). Introduction to algorithms second edition.
- Ginat (2004) Ginat, D. (2004). Do Senior CS Students Capitalize on Recursion? SIGCSE Bull., 36(3):82–86.
- Gretton et al. (2005) Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. (2005). Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pages 63–77. Springer.
- Gretton et al. (2008) Gretton, A., Fukumizu, K., Teo, C. H., Song, L., Schölkopf, B., and Smola, A. J. (2008). A kernel statistical test of independence. In Advances in Neural Information Processing Systems, pages 585–592.
- Guo et al. (2014) Guo, X., Zhang, Y., Hu, W., Tan, H., and Wang, X. (2014). Inferring nonlinear gene regulatory networks from gene expression data based on distance correlation. PloS one, 9(2):e87446.
- Huang and Huo (2017) Huang, C. and Huo, X. (2017). A statistically and numerically efficient independence test based on random projections and distance covariance. arXiv preprint arXiv:1701.06054.
- Huo and Székely (2016) Huo, X. and Székely, G. J. (2016). Fast computing for distance covariance. Technometrics, 58(4):435–447.
- Knuth (1997) Knuth, D. E. (1997). The Art of Computer Programming: Sorting and Searching, volume 3. Pearson Education.
- Li et al. (2012) Li, R., Zhong, W., and Zhu, L. (2012). Feature screening via distance correlation learning. Journal of the American Statistical Association, 107(499):1129–1139.
- Pfister et al. (2018) Pfister, N., Bühlmann, P., Schölkopf, B., and Peters, J. (2018). Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(1):5–31.
- Sejdinovic et al. (2013) Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. (2013). Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics, pages 2263–2291.
- Sferra et al. (2017) Sferra, G., Fratini, F., Ponzi, M., and Pizzi, E. (2017). Phylo_dcor: distance correlation as a novel metric for phylogenetic profiling. BMC bioinformatics, 18(1):396.
- Székely and Rizzo (2009) Székely, G. J. and Rizzo, M. L. (2009). Brownian distance covariance. The annals of applied statistics, pages 1236–1265.
- Székely et al. (2007) Székely, G. J., Rizzo, M. L., Bakirov, N. K., et al. (2007). Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6):2769–2794.
- Zhang et al. (2018) Zhang, Q., Filippi, S., Gretton, A., and Sejdinovic, D. (2018). Large-scale kernel methods for independence testing. Statistics and Computing, 28(1):113–130.
- Zhou (2012) Zhou, Z. (2012). Measuring nonlinear dependence in time-series, a distance correlation approach. Journal of Time Series Analysis, 33(3):438–457.
MATLAB Code for Fast Distance Covariance