randomized_dependence_coefficient
Paper and code for "The Randomized Dependence Coefficient", NIPS 2013
view repo
We introduce the Randomized Dependence Coefficient (RDC), a measure of non-linear dependence between random variables of arbitrary dimension based on the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient. RDC is defined in terms of correlation of random non-linear copula projections; it is invariant with respect to marginal distribution transformations, has low computational cost and is easy to implement: just five lines of R code, included at the end of the paper.
READ FULL TEXT VIEW PDF
Measuring strength or degree of statistical dependence between two rando...
read it
The maximal correlation coefficient is a well-established generalization...
read it
In this paper, a robust non-parametric measure of statistical dependence...
read it
This paper introduces a nonparametric copula-based approach for detectin...
read it
This article provides an extension to recent work on the development of ...
read it
We propose a methodology to explore and measure the pairwise correlation...
read it
We develop a robust data fusion algorithm for field reconstruction of
mu...
read it
Paper and code for "The Randomized Dependence Coefficient", NIPS 2013
Measuring statistical dependence between random variables is a fundamental problem in statistics. Commonly used measures of dependence, Pearson’s rho, Spearman’s rank or Kendall’s tau are computationally efficient and theoretically well understood, but consider only a limited class of association patterns, like linear or monotonically increasing functions. The development of non-linear dependence measures is challenging because of the radically larger amount of possible association patterns.
Despite these difficulties, many non-linear statistical dependence measures have been developed recently. Examples include the Alternating Conditional Expectations or backfitting algorithm (ACE) [2, 9], Kernel Canonical Correlation Analysis (KCCA) [1], (Copula) Maximum Mean Discrepancy (MMD, CMMD in their HSIC formulations) [6, 5, 15], Distance or Brownian Correlation (dCor) [24, 23] and the Maximal Information Coefficient (MIC) [18]. However, these methods exhibit high computational demands (at least quadratic costs in the number of samples for KCCA, MMD, CMMD, dCor or MIC), are limited to measuring dependencies between scalar random variables (ACE, MIC), show poor performance under the existence of additive noise (MIC) or can be difficult to implement (ACE, MIC).
This paper develops the Randomized Dependence Coefficient
(RDC), an estimator of the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient (HGR) addressing the issues listed above. RDC defines dependence between two random variables as the largest canonical correlation between random non-linear projections of their respective empirical copula-transformations. RDC is invariant to monotonically increasing transformations, operates on random variables of arbitrary dimension, and has computational cost of
with respect to the sample size. Moreover, it is easy to implement: just five lines of R code, included in Appendix A.The following Section reviews the classic work of Alfréd Rényi [17], who proposed seven desirable fundamental properties of dependence measures, proved to be satisfied by the Hirschfeld-Gebelein-Rényi’s Maximum Correlation Coefficient (HGR). Section 3 introduces the Randomized Dependence Coefficient as an estimator designed in the spirit of HGR, since HGR itself is computationally intractable. Properties of RDC and its relationship to other non-linear dependence measures are analysed in Section 4. Section 5 validates the empirical performance of RDC on a series of numerical experiments on both synthetic and real-world data.
In 1959 [17], Alfréd Rényi argued that a measure of dependence between random variables and should satisfy seven fundamental properties:
is defined for any pair of non-constant random variables and .
iff and are statistically independent.
For bijective Borel-measurable functions , .
if for Borel-measurable functions or , or .
If , then , where is the correlation coefficient.
Rényi also showed the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient (HGR) [3, 17] to satisfy all these properties. HGR was defined by Gebelein in 1941 [3] as the supremum of Pearson’s correlation coefficient over all Borel-measurable functions
of finite variance:
(1) |
Since the supremum in (1) is over an infinite-dimensional space, HGR is not computable. It is an abstract concept, not a practical dependence measure. In the following we propose a scalable estimator with the same structure as HGR: the Randomized Dependence Coefficient.
The Randomized Dependence Coefficient (RDC) measures the dependence between random samples and as the largest canonical correlation between randomly chosen non-linear projections of their copula transformations. Before Section 3.4 defines this concept formally, we describe the three necessary steps to construct the RDC statistic: copula-transformation of each of the two random samples (Section 3.1), projection of the copulas through randomly chosen non-linear maps (Section 3.2) and computation of the largest canonical correlation between the two sets of non-linear random projections (Section 3.3). Figure 1 offers a sketch of this process.
To achieve invariance with respect to transformations on marginal distributions (such as shifts or rescalings), we operate on the empirical copula transformation of the data [14, 15]
. Consider a random vector
with continuous marginal cumulative distribution functions (cdfs)
, . Then the vector , known as the copula transformation, has uniform marginals:(Probability Integral Transform
[14]) For a random variable with cdf , the random variableis uniformly distributed on
.The random variables are known as the observation ranks of . Crucially, preserves the dependence structure of the original random vector , but ignores each of its marginal forms [14]
. The joint distribution of
is known as the copula of :(Sklar [20]) Let the random vector have continuous marginal cdfs , . Then, the joint cumulative distribution of is uniquely expressed as:
(2) |
where the distribution is known as the copula of .
A practical estimator of the univariate cdfs is the empirical cdf:
(3) |
which gives rise to the empirical copula transformations of a multivariate sample:
(4) |
The Massart-Dvoretzky-Kiefer-Wolfowitz inequality [13] can be used to show that empirical copula transformations converge fast to the true transformation as the sample size increases:
(Convergence of the empirical copula, [15, Lemma 7]) Let
be an i.i.d. sample from a probability distribution over
with marginal cdf’s . Let be the copula transformation and the empirical copula transformation. Then, for any :(5) |
Computing involves sorting the marginals of , thus operations.
The second step of the RDC computation is to augment the empirical copula transformations with non-linear projections, so that linear methods can subsequently be used to capture non-linear dependencies on the original data. This is a classic idea also used in other areas, particularly in regression. In an elegant result, Rahimi and Brecht [16]
proved that linear regression on random, non-linear projections of the original feature space can generate high-performance regressors:
(Rahimi-Brecht) Let be a distribution on and . Let . Draw iid from . Further let , and be some
-Lipschitz loss function, and consider data
drawn iid from some arbitrary . The for which minimizes the empirical risk has a distance from the -optimal estimator in bounded by(6) |
with probability at least .
Intuitively, Theorem 4 states that randomly selecting in instead of optimising them causes only bounded error.
The choice of the non-linearities is the main, unavoidable assumption in RDC. This choice is a well-known problem common to all non-linear regression methods and has been studied extensively in the theory of regression as the selection of reproducing kernel Hilbert space [19, §3.13]. The choice of the family (space) of features, and of probability distributions over it, is unlimited. The only way to favour one such family and distribution over another is to use prior assumptions about which kind of distributions the method will typically have to analyse.
Features popular in parts of the literature are sigmoids, parabolas, radial basis functions, complex sinusoids, sines or cosines. In our experiments, we will use sine and cosine projections,
. Arguments favouring this choice are that shift-invariant kernels are approximated with these features when using the appropriate random parameter sampling distribution [16],[4, p. 208] [22, p. 24], and that functions with absolutely integrable Fourier transforms are approximated with
error below by of these features [10].Let the random parameters , . Choosing to be Normal is analogous to the use of the Gaussian kernel for MMD, CMMD or KCCA [16]. Tuning is analogous to selecting the kernel width, that is, to regularize the non-linearity of the random projections.
Given a data collection , we will denote by
(7) |
the th order random non-linear projection from to . The computational complexity of computing using naive matrix multiplications is . However, recent techniques [11] allow computing these feature expansions within a computational cost of using storage.
The final step of RDC is to compute the linear combinations of the augmented empirical copula transformations that have maximal correlation. Canonical Correlation Analysis (CCA, [7]) is the calculation of pairs of basis vectors such that the projections and of two random samples and are maximally correlated. The correlations between the projected (or canonical) random samples are referred to as canonical correlations. There exist up to of them. Canonical correlations are the solutions to the eigenproblem:
(8) |
where and the matrices and are assumed to be invertible. Therefore, the largest canonical correlation between and is the supremum of the correlation coefficients over their linear projections, that is:
When , the cost of CCA is dominated by the estimation of the matrices , and , hence being for two random variables of dimensions and , respectively.
Given the random samples and and the parameters and , the Randomized Dependence Coefficient between and is defined as:
(9) |
In the typical setup (very large , large and , small ) the computational complexity of RDC is dominated by the calculation of the copula-transformations. Hence, we achieve a cost in terms of the sample size of .
An implementation of RDC in R is included in the Appendix A.
It is tempting to wonder whether RDC is a consistent, or even an efficient estimator of the HGR coefficient. However, a simple experiment shows that it is not desirable to approximate HGR exactly on finite datasets: Consider which is independent, thus, by both Rényi’s 4th and 7th properties, has . However, for finitely many samples from , almost surely, values in both and are pairwise different and separated by a finite difference. So there exist continuous (thus Borel measurable) functions and mapping both and to the sorting ranks of , i.e. . Therefore, the finite-sample version of Equation (1
) is constant and equal to “1” for continuous random variables. Meaningful measures of dependence from finite samples thus must rely on some form of regularization. RDC achieves this by approximating the space of Borel measurable functions with the restricted function class
from Theorem 4:Assume the optimal transformations and (Equation 1) to belong to the Reproducing Kernel Hilbert Space (Theorem 4), with associated shift-invariant, positive semi-definite kernel function . Then, with probability greater than :
(10) |
where and , denote the sample size and number of random features. The bound (10) is the sum of two errors. The error
is due to the convergence of CCA’s largest eigenvalue in the finite sample size regime. This result
[8, Theorem 6] is originally obtained by posing CCA as a least squares regression on the product space induced by the feature map . Because of approximating with random features, an additional error is introduced in the least squares regression [16, Lemma 3]. Therefore, an equivalence between RDC and KCCA is established if RDC uses an infinite number of sine/cosine features, the random sampling distribution is set to the inverse Fourier transform of the shift-invariant kernel used by KCCA and the copula-transformations are discarded. However, when regularization is needed to avoid spurious perfect correlations, as discussed above.Table 1 summarizes several state-of-the-art dependence measures showing, for each measure, whether it allows for general non-linear dependence estimation, handles multidimensional random variables, is invariant with respect to changes in marginal distributions, returns a statistic in , satisfy Rényi’s properties (Section 2), and how many parameters it requires. As parameters, we here count the kernel function for kernel methods, the basis function and number of random features for RDC, the stopping tolerance for ACE and the search-grid size for MIC, respectively. Finally, the table lists computational complexities with respect to sample size.
Consider the hypothesis “the two sets of non-linear projections are mutually uncorrelated”. Under normality assumptions (or large sample sizes), Bartlett’s approximation [12] can be used to show:
(11) |
where are the canonical correlations between the two sets of non-linear projections and . Alternatively, non-parametric asymptotic distributions can be obtained from the spectrum of the inner products of the non-linear random projection matrices [25, Theorem 3].
We performed numerical experiments on both synthetic and real-world data to validate the empirical performance of RDC versus the non-linear dependence measures listed in Table 1. In some experiments, we don’t compare against to KCCA due its prohibitive running times (see Table 2).
The number of random features for RDC was set to symmetrically for both random samples, since no significant improvements were observed for larger values. However, this parameter can be set to the largest value that fits within the available computational budget. The random sampling parameters were set independently for each of the two random samples, equal to their squared euclidean distance empirical median [5]. Competing kernel methods make use of Gaussian RBF kernels of the form for the random variable and analogously for the random variable . For the MIC statistic, the search-grid size is set to , as recommended by the authors [18]. The stopping tolerance for ACE is set to , the default value in the R package acepack^{1}^{1}1http://cran.r-project.org/web/packages/acepack/index.html.
We define the power of a dependence measure as its ability to discern between dependent and independent samples that share equal marginal forms. In the spirit of Simon and Tibshirani^{2}^{2}2http://www-stat.stanford.edu/~tibs/reshef/comment.pdf, we conducted experiments to estimate the power of RDC as a measure of non-linear dependence. We chose 8 bivariate association patterns, depicted inside little boxes in Figure 3. For each of the 8 association patterns, 500 repetitions of 500 samples were generated, in which the input variable was uniformly distributed on the unit interval. Next, we regenerated the input variable randomly, to generate independent versions of each sample with equal marginals. Figure 3 shows the power for the discussed non-linear dependence measures as the variance of some zero-mean Gaussian additive noise increases from to . RDC shows worse performance in the linear association pattern due to noise overfitting and in the step-function due to the smoothness prior induced by the use of sine/cosine basis functions, but has good performance in non-functional association patterns (such as the circle and the mixture of sinusoidal waves).
Table 2 summarizes running times (in seconds) for the considered non-linear dependence measures on scalar, uniformly distributed, independent samples of sizes when averaging over 100 runs. Single runs above ten minutes were cancelled (empty cells in table). In this comparison, Pearson’s , ACE, dCor and MIC are using compiled C code, while RDC, along with MMD, CMMD and KCCA are implemented as interpreted R code.
sample size | Pearson’s | RDC | ACE | dCor | MMD | CMMD | MIC | KCCA |
---|---|---|---|---|---|---|---|---|
1,000 | 0.0001 | 0.0047 | 0.0080 | 0.3417 | 0.3103 | 0.3501 | 1.0983 | 166.29 |
10,000 | 0.0002 | 0.0557 | 0.0782 | 59.587 | 27.630 | 29.522 | — | — |
100,000 | 0.0071 | 0.3991 | 0.5101 | — | — | — | — | — |
1,000,000 | 0.0914 | 4.6253 | 5.3830 | — | — | — | — | — |
Figure 4 shows RDC, ACE, dCor, MIC, Pearson’s , Spearman’s rank and Kendall’s dependence estimates for 14 different associations of two scalar random variables. RDC scores values close to one on all the proposed dependent associations, whilst scoring values close to zero for the independent association, depicted last. When the associations are Gaussian (first row), RDC scores values close to the Pearson’s correlation coefficient, as suggested in the seventh property of Rényi (Section 2).
We performed greedy feature selection via dependence maximization
[21] on eight real-world datasets. More specifically, we attempted to construct the subset of features that minimizes the normalized mean squared regression error (NMSE) of a Gaussian process regressor. We do so by selecting the feature maximizing dependence between the feature set and the target variable at each iteration , such that and .We considered 12 heterogeneous datasets, obtained from the UCI dataset repository^{3}^{3}3http://www.ics.uci.edu/~mlearn, the Gaussian process web site Data^{4}^{4}4http://www.gaussianprocess.org/gpml/data/
and the Machine Learning data set repostitory
^{5}^{5}5http://www.mldata.org. Random training/test partitions are computed to be disjoint and equal sized.Since can be multi-dimensional, we compare RDC to the non-linear methods dCor, MMD and CMMD. Given their quadratic computational demands, dCor, MMD and CMMD use up to points when measuring dependence; this constraint only applied on the sarcos and calcensus datasets. Results are averages of random training/test partitions.
Figure 2 summarizes the results for all datasets and algorithms as the number of selected features increases. RDC performs best in most datasets, with much lower running time than its contenders.
We have presented the randomized dependence coefficient, a lightweight non-linear measure of dependence between multivariate random samples. Constructed as a finite-dimensional estimator in the spirit of the Hirschfeld-Gebelein-Rényi maximum correlation coefficient, RDC performs well empirically, is scalable to very large datasets, and is easy to adapt to concrete problems.
rdc <- function(x,y,k,s) { x <- cbind(apply(as.matrix(x),2,function(u) ecdf(u)(u)),1) y <- cbind(apply(as.matrix(y),2,function(u) ecdf(u)(u)),1) wx <- matrix(rnorm(ncol(x)*k,0,s),ncol(x),k) wy <- matrix(rnorm(ncol(y)*k,0,s),ncol(y),k) cancor(cbind(cos(x%*%wx),sin(x%*%wx)), cbind(cos(y%*%wy),sin(y%*%wy)))$cor[1] }
Kernel independent component analysis.
JMLR, 3:1–48, 2002.A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training.
Annals of Statistics, 20(1):608–613, 1992.
Comments
There are no comments yet.