Discovering Potential Correlations via Hypercontractivity

Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms that we expect a measure of potential correlation to satisfy; (ii) we show that the rate of information bottleneck, i.e., the hypercontractivity coefficient, satisfies all the proposed axioms; (iii) we provide a novel estimator to estimate the hypercontractivity coefficient from samples; and (iv) we provide numerical experiments demonstrating that this proposed estimator discovers potential correlations among various indicators of WHO datasets, is robust in discovering gene interactions from gene expression time series data, and is statistically more powerful than the estimators for other correlation measures in binary hypothesis testing of canonical examples of potential correlations.

Authors

• 13 publications
• 9 publications
• 27 publications
• 41 publications
• 36 publications
• Testing for equal correlation matrices with application to paired gene expression data

We present a novel method for testing the hypothesis of equality of two ...
03/18/2018 ∙ by Adria Caballe, et al. ∙ 0

• Hypothesis Testing in High-Dimensional Instrumental Variables Regression with an Application to Genomics Data

Gene expression and phenotype association can be affected by potential u...
10/21/2019 ∙ by Jiarui Lu, et al. ∙ 0

• Distance Correlation Methods for Discovering Associations in Large Astrophysical Databases

High-dimensional, large-sample astrophysical databases of galaxy cluster...
08/19/2013 ∙ by Elizabeth Martinez-Gomez, et al. ∙ 0

• Measuring and Discovering Correlations in Large Data Sets

In this paper, a class of statistics named ART (the alternant recursive ...
01/07/2016 ∙ by Lijue Liu, et al. ∙ 0

• Discovering Reliable Correlations in Categorical Data

In many scientific tasks we are interested in discovering whether there ...
08/30/2019 ∙ by Panagiotis Mandros, et al. ∙ 20

• Large Scale Correlation Screening

This paper treats the problem of screening for variables with high corre...
02/06/2011 ∙ by Alfred O. Hero, et al. ∙ 0

• Detecting correlations and triangular arbitrage opportunities in the Forex by means of multifractal detrended cross-correlations analysis

Multifractal detrended cross-correlation methodology is described and ap...
06/18/2019 ∙ by Robert Gębarowski, et al. ∙ 0

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Measuring the strength of an association between two random variables is a fundamental topic of broad scientific interest. Pearson’s correlation coefficient

[1] dates from over a century ago and has been generalized seven decades ago as maximal correlation (mCor) to handle nonlinear dependencies  [2, 3, 4]. Novel correlation measures to identify different kinds of associations continue to be proposed in the literature; these include maximal information coefficient (MIC) [5] and distance correlation (dCor) [6]. Despite the differences, a common theme of measurement of the empirical average dependence unites the different dependence measures. Alternatively, these are factual measures of dependence and their relevance is restricted when we seek a potential

dependence of one random variable on another. For instance, consider a hypothetical city with very few smokers. A standard measure of correlation on the historical data in this town on smoking and lung cancer will fail to discover the fact that smoking causes cancer, since the average correlation is very small. On the other hand, clearly, there is a potential correlation between smoking and lung cancer; indeed applications of this nature abound in several scenarios in modern data science, including a recent one on genetic pathway discovery

[7].

Discovery of a potential correlation naturally leads one to ask for a measure of potential correlation that is statistically well-founded and addresses practical needs. Such is the focus of this work, where our proposed measure of potential correlation is based on a novel interpretation of the Information Bottleneck (IB) principle [8]

. The IB principle has been used to address one of the fundamental tasks in supervised learning: given samples

, how do we find a compact summary of a variable that is most informative in explaining another variable . The output of the IB principle is a compact summary of that is most relevant to and has a wide range of applications [9, 10].

We use this IB principle to create a measure of correlation based on the following intuition: if is (potentially) correlated with , then a relatively compact summary of can still be very informative about . In other words, the maximal ratio of how informative a summary can be in explaining to how compact a summary is with respect to is, conceptually speaking, an indicator of potential correlation from to . Quantifying the compactness by and the information by we consider the rate of information bottleneck as a measure of potential correlation:

 s(X;Y)≡supU--X--YI(U;Y)I(U;X), (1)

where

forms a Markov chain and the supremum is over all summaries

of . This intuition is made precise in Section 2, where we formally define a natural notion of potential correlation (Axiom 6), and show that the rate of information bottleneck captures this potential correlation (Theorem 1) while other standard measures of correlation fail (Theorem 2).

This ratio has only recently been identified as the hypercontractivity coefficient [11]. Hypercontractivity has a distinguished and central role in a large number of technical arenas including quantum physics [12, 13], theoretical computer science [14, 15], mathematics [16, 17, 18]

[19, 20]. In this paper, we provide a novel interpretation to the hypercontractivity coefficient as a measure of potential correlation by demonstrating that it satisfies a natural set of axioms such a measure is expected to obey.

For practical use in discovering correlations, the standard correlation coefficients are equipped with corresponding natural sample-based estimators. However, for hypercontractivity coefficient, estimating it from samples is widely acknowledged to be challenging, especially for continuous random variables

[21, 22, 23]. There is no existing algorithm to estimate the hypercontractivity coefficient in general [21], and there is no existing algorithm for solving IB from samples either [22, 23]. We provide a novel estimator of the hypercontractivity coefficient – the first of its kind – by bringing together the recent theoretical discoveries in [11, 24]

of an alternate definition of hypercontractivity coefficient as ratio of Kullback-Leibler divergences defined in (

10), and recent advances in joint optimization (the max step in Equation 1) and estimating information measures from samples using importance sampling [25].

Our main contributions are the following:

• We postulate a set of natural axioms that a measure of potential correlation from to should satisfy (Section 2.1).

• We show that , our proposed measure of potential correlation, satisfies all the axioms we postulate (Section 2.2). In comparison, we prove that existing standard measures of correlation not only fail to satisfy the proposed axioms, but also fail to capture canonical potential correlations captured by (Section 2.3). Another natural candidate is mutual information, but it is not clear how to interpret the value of mutual information as it is unnormalized, unlike all other measures of correlation which are between zero and one.

• Computation of the hypercontractivity coefficient from samples is known to be a challenging open problem. We in troduce a novel estimator to compute hypercontractivity coefficient from i.i.d. samples in a statistically consistent manner for continuous random variables, using ideas from importance sampling and kernel density estimation (Section

3).

• In a series of synthetic experiments, we show empirically that our estimator for the hypercontractivity coefficient is statistically more powerful in discovering a potential correlation than existing correlation estimators; a larger power means a larger successful detection rate for a fixed false alarm rate (Section 4.1).

• We show applications of our estimator of hypercontractivity coefficient in two important datasets: In Section 4.2, we demonstrate that it discovers hidden potential correlations among various national indicators in WHO datasets, including how aid is potentially correlated with the income growth. In Section 4.3, we consider the following gene pathway recovery problem: we are given samples of four gene expressions time series. Assuming we know that gene A causes B, that B causes C, and that C causes D, the problem is to discover that these causations occur in the sequential order: A to B, and then B to C, and then C to D. We show empirically that the estimator of the hypercontractivity coefficient recovers this order accurately from a vastly smaller number of samples compared to other state-of-the art causal influence estimators.

2 Axiomatic approach to measure potential correlations

We propose a set of axioms that we expect a measure of potential correlation to satisfy. We then show that hypercontractivity coefficient, first introduced in [19], satisfies all the proposed axioms, hence propose hypercontractivity coefficient as a measure of potential correlation. We also show that other standard correlation coefficients and mutual information, on the other hand, violate the proposed axioms.

2.1 Axioms for potential correlation

We postulate that a measure of potential correlation between two random variables and should satisfy:

1. is defined for any pair of non-constant random variables and .

2. .

3. iff and are statistically independent.

4. For bijective Borel-measurable functions , .

5. If , then , where is the Pearson correlation coefficient.

6. if there exists a subset such that for a pair of continuous random variables , for a Borel-measurable and non-constant continuous function .

Axioms 1-5 are identical to a subset of the celebrated axioms of Rényi in [4], which ensure that the measure is properly normalized and invariant under bijective transformations, and recovers the Pearson correlation for jointly Gaussian random variables. Rényi’s original axioms for a measure of correlation in [4] included Axioms 1-5 and also that the measure of correlation should satisfy

1. if for Borel-measurable functions or , or .

2. .

The Pearson correlation violates a subset (3, 4, and 6’) of Rényi’s axioms. Together with recent empirical successes in multimodal deep learning (e.g.

[26, 27, 28]), Rényi’s axiomatic approach has been a major justification of Hirschfeld-Gebelein-Rényi (HGR) maximal correlation coefficient defined as , which satisfies all Rényi’s axioms [2]. Here, the supremum is over all measurable functions with and . However, maximal correlation is not the only measure satisfying all of Rényi’s axioms, as we show in the following.

Proposition 1.

For any function satisfying , , and only if , the symmetrized satisfies all Rényi’s axioms.

This follows from the fact that the hypercontractivity coefficient satisfies all but the symmetry in Axiom 7’ (Theorem 1), and it follows that a symmetrized version satisfies all axioms, e.g.  and . A formal proof is provided in Section 5.1.

From the original Rényi’s axioms, for a potential correlation measure, we remove Axiom 7’ that ensures symmetry, as directionality is fundamental in measuring the potential correlation from to . We further replace Axiom 6’ by Axiom 6, as a variable has a full potential to be correlated with if there exists a domain such that and are deterministically dependent and non-degenerate (i.e. not a constant function), as illustrated in Figure 1 for a linear function and a quadratic function.

2.2 The hypercontractivity coefficient satisfies all axioms

We show that the hypercontractivity coefficient defined in Equation (1) satisfies all Axioms 1-6. Intuitively, measures how much potential correlation has with . For example, if and are independent, then as has no correlation with (Axiom 3). By data processing inequality, it follows that it is a measure between zero and one (Axiom 2) and also invariant under bijective transformations (Axiom 4). For jointly Gaussian variables and with the Pearson correlation , we can show that . Hence, the squared-root of satisfies Axiom 5. In fact, satisfies all desired axioms for potential correlation, and we make this precise in the following theorem whose proof is provided in Section 5.2.

Theorem 1.

Hypercontractivity coefficient satisfies Axioms 1-6.

In particular, the hypercontractivity coefficient satisfies Axiom 6 for potential correlation, unlike other measures of correlation (see Theorem 2 for examples). If there is a potential for in a possibly rare regime in to be fully correlated with such that , then the hypercontractivity coefficient is maximum: . In the following section, we show that existing correlation measures, on the other hand, violate the proposed axioms.

2.3 Standard correlation coefficients violate the Axioms.

We analyze existing measures of correlations under the scenario with potential correlation (Axiom 6), where we find that none of the existing correlation measures satisfy Axiom 6. Suppose and are independent (i.e. no correlation) in a subset of the domain , and allow and to be arbitrarily correlated in the rest of the domain, such that . We further assume that the independent part is dominant and the correlated part is rare; let and we consider the scenario when is small. A good measure of potential correlation is expected to capture the correlation in even if it is rare (i.e., is small). To make this task more challenging, we assume that the conditional distribution of is the same as . Figure 1 (of this section) illustrates sampled points for two examples from such a scenario and more examples are in Figure 3. Our main result is the analysis of HGR maximal correlation (mCor) [2], distance correlation (dCor) [6], maximal information coefficients (MIC) [5], which shows that these measures are vanishing with even if the dependence in the rare regime is very high. Suppose , then all three correlation coefficients are vanishing as gets small. This in particular violates Axiom 6. The reason is that standard correlation coefficients measure the average correlation whereas the hypercontractivity coefficient measures the potential correlation. The experimental comparisons on the power of these measures confirm our analytical predictions in Figure 4 in Section 4. The formal statement is below and the proof is provided in Section 5.3.

Theorem 2.

Consider a pair of continuous random variables . Suppose is partitioned as such that for all , and is independent of for . Let . The HGR maximal correlation coefficient is

 mCor(X,Y)=√αmCor(Xr,Y), (2)

the distance correlation coefficient is

 dCor(X,Y)=αdCor(Xr,Y), (3)

the maximal information coefficient is upper bounded by

 MIC(X,Y)≤αMIC(Xr,Y), (4)

where is the random variable conditioned on the rare domain .

Under the rare/dominant scenario considered in Theorem 2, . It is well known that this inequality holds for any and  [19]. In particular, Theorem 3 in [29] shows that hypercontractivity coefficient is a natural extension of the popular HGR maximal correlation coefficient as follows.

Remark 1 (Connection between s(X;Y) and mCor(X,y)[29]).

The squared HGR maximal correlation is a special case of the hypercontractivity optimization in Equation (10) restricted to searching over a distribution in a close neighborhood of .

As searches over a larger space, it is always larger than or equal to mCor. This gives an intuitive justification for using as a measure of potential influence; we allow search over larger space, but properly normalized by the KL divergence, in a hope to find a potential distribution that can influence significantly. While hypercontractivity coefficient is a natural extension of HGR maximal correlation coefficient, there is an important difference between hypercontractivity coefficient and HGR maximal correlation coefficient (and other correlation measures); hypercontractivity is directional.

Remark 2 (Asymmetry of s(X;Y)).

Hypercontractivity coefficient is asymmetric in and while HGR maximal correlation, distance correlation, and MIC are symmetric.

Under the rare/dominant scenario considered in Theorem 2, the hypercontracitivy coefficient is large because it measures the potential correlation from to . On the other hand, inverse hypercontractivity coefficient , which measures the potential correlation from to , is small as there is no apparent potential correlation from to . This is made precise in the following proposition.

Proposition 2.

Under the hypotheses of Theorem 2, the hypercontractivity coefficient from to is

 s(Y;X)=αs(Y;Xr),

where is the random variable conditioned on the rare domain .

Proof is provided in Section 5.4.

2.4 Mutual Information violates the Axioms

Beside standard correlation measures, another measure widely used to quantify the strength of dependence is mutual information. We can show that mutual information satisfies Axiom 6 if we replace 1 by . However there are two key problems:

• Practically, mutual information is unnormalized, i.e., . Hence, it provides no absolute indication of the strength of the dependence.

• Mathematically, we are looking for a quantity that tensorizes, i.e., doesn’t change when there are many i.i.d. copies of the same pair of random variables.

Remark 3 (Tensorization property of s(X;Y)[30]).

Hypercontractivity coefficient tensorizes, i.e,

 s(X1,...,Xn;Y1,..,Yn)=s(X1,Y1),  for i.i.d.\ (Xi,Yi),  i=1,⋯,n.

On the other hand, mutual information is additive, i.e.,

 I(X1,⋯,Xn;Y1,⋯,Yn)=nI(X1;Y1),  for i.i.d.\ % (Xi,Yi),  i=1,⋯,n.

Tensorizing quantities capture the strongest relationship among independent copies while additive quantities capture the sum. For instance, mutual information could be large because a small amount of information accumulates over many of the independent components of and (when and are high dimensional) while tensorizing quantities would rule out this scenario, where there is no strong dependence. When the components are not independent, hypercontractivity indeed pools information from different components to find the strongest direction of dependence, which is a desirable property.

One natural way to normalize mutual information is by the log of the cardinality of the input/output alphabets [31]. One can interpret a popular correlation measure MIC as a similar effort for normalizing mutual information and is one of our baselines.

Given that other correlation measures and mutual information do not satisfy our axioms, a natural question to ask is whether hypercontractivity is a unique solution that satisfies all the proposed axioms. In the following, we show that the hypercontractivity coefficient is not the only one satisfying all the proposed axioms – just as HGR correlation is not the only measure satisfying Rényi’s original axioms.

2.5 Hypercontractivity ribbon

We show that a family of measures known as hypercontractivity ribbon, which includes hypercontractivity coefficient as a special case, satisfies all the axioms. The hypercontractivity ribbon [19, 32] is a class of measures parametrized by as

 rα(X;Y)=supr(x,y)≠p(x,y)D(r(y)∥p(y))D(r(x)∥p(x))+αD(r(y|x)∥p(y|x)), (5)

where denotes the KL divergence of and . An alternative characterization of hypercontractivity ribbon in terms of mutual information is provided in [32, 24];

 (6)

from which we can see that hypercontractivity coefficient is a special case of hypercontractivity ribbon [11]:

 s(X;Y) =limα→∞rα(X;Y)=limα→∞sα(X;Y).
Proposition 3.

The (re-parameterized) hypercontractivity ribbon , for , satisfies Axioms 1-6.

Proof.

By definition, is defined for any pair of non-constant random variables (Axiom 1) and is between 0 and 1 by data processing inequality (Axiom2). We can show that satisfies Axioms 3 and 4, in a similar way to show satisfies Axioms 3 and 4. Also, for a jointly Gaussian , with Pearson correlation  [24] (Axiom 5). Finally, satisfies Axiom 6 because is non-increasing in , which implies that if . ∎

Although hypercontracitivy ribbon satisfies all axioms, a few properties of the hypercontractivity coefficient make it more attractive than hypercontractivity ribbon for practical use; hypercontractivity coefficient can be efficiently estimated from samples (see Section 3). Hypercontractivity coefficient is a natural extension of the popular HGR maximal correlation coefficient (Remark 1).

2.6 Multidimensional X and Y

In this section, we discuss potential correlation of multidimensional and . While most of the correlation coefficients, including the hypercontractivity coefficient, are well-defined for multi-dimensional and , the axioms are specific to univariate and . To bridge this gap, we propose replacing Axiom 5, as this is the only axiom specific to univariate random variables.

1. Axiom 5’. If , then , where is the spectral norm of a matrix.

This recovers the original Axiom 5 when restricted to univariate and . This naturally generalizes both Rényi’s axioms and the proposed potential correlation axioms to multidimensional and .

Proposition 4.

Axiom 5’, together with original Rényi’s Axioms 1-4, 6’, and 7’, recovers maximal correlation (mCor) as a measure satisfying all Axioms even in this multi-dimensional case. Axiom 5’, together with our proposed Axioms 1-4, and 6, recovers the hypercontractivity coefficient as a measure satisfying all axioms.

The second statement in the proposition follows from the analyses of the hypercontractivity coefficient of Gaussian distributions in

[33]. A formal proof is provided in Section 5.7.

2.7 Noisy, discrete, noisy and discrete potential correlations

In this section, we consider more general scenarios of potential correlation than the one in Axiom 6. We consider (i) noisy potential correlation where for a Gaussian noise for , (ii) discrete potential correlation, where , and (iii) noisy discrete potential correlation – a random corruption model. For these three examples, we obtain a lower bound on .

Example 1.

Suppose that for a pair of random variables , there exists a subset for which , and for , , where

 Σ=[1ρρ1].

Then

 s(X;Y)≥log11−ρ2+log11+ρ2log11−ρ2+H(α)α (7)

Proof is in Section 5.5.

We now consider for discrete . We start with the case for which and are perfectly correlated for .

Example 2.

Suppose that for a pair of discrete random variables

, there exists a subset for which , and and for . Then,

 s(X;Y)≥logklogk+log(1/α).

The inequality holds by considering in (10).

We conjecture this lower bound is indeed tight for based on numerical simulations. From this lower-bound, we can see the trade-off between and . As , the lower bounds approaches to 1. As , the lower bound approaches to 1. As , the lower bound approaches to 0. In the following, we consider the case where and are not perfectly correlated in for discrete . In particular, we consider a random corruption model for and obtain a lower bound on .

Example 3.

Suppose that for a pair of random variables , there exists a subset for which , and for ,

 Y={Xw.p.\ 1−kk−1ϵ,Unif[1:k]w.p.\ 1k−1ϵ.

Then

 s(X;Y)≥(1−ϵ)logk(1−ϵ)+ϵlogkϵ/(k−1)log(k/α)=logk−H2(ϵ)−ϵlog(k−1)log(k/α). (8)

On the other hand,

 mCor2(X;Y)=α(1−kk−1ϵ)2,  0≤ϵ≤k−1k. (9)

Proof is in Section 5.6.

In Figure 2, we show plots of lower bounds on and mCor in Examples 1-3; from these figures, we can see that increases as and . In comparison, mCor remains small.

3 Estimator of the hypercontractivity coefficient from samples

In this section, we present an algorithm111Code available at https://github.com/wgao9/hypercontractivity to compute the hypercontractivity coefficient from i.i.d. samples . The computation of the hypercontractivity coefficient from samples is known to be challenging for continuous random variables [22, 23], and to the best of our knowledge, there is no known efficient algorithm to compute the hypercontractivity coefficient from samples. Our estimator is the first efficient algorithm to compute the hypercontractivity coefficient, based on the following equivalent definition of the hypercontractivity coefficient, shown recently in [11]:

 s(X;Y) ≡ suprx≠pxD(ry||py)D(rx||px). (10)

There are two main challenges for computing . The first challenge is – given a marginal distribution and samples from , how do we estimate the KL divergences and . The second challenge is the optimization over the infinite dimensional simplex. We need to combine estimation and optimization together in order to compute . Our approach is to combine ideas from traditional kernel density estimates and from importance sampling. Let be the likelihood ratio evaluated at sample . We propose the estimation and optimization be solved jointly as follows:

Estimation: To estimate KL divergence , notice that

 D(rx||px) = EX∼px[rx(X)px(X)logrx(X)px(X)].

Using empirical average to replace the expectation over , we propose

 ˆD(rx||px)=1nn∑i=1rx(Xi)px(Xi)logrx(Xi)px(Xi)=1nn∑i=1wilogwi.

For , we follow the similar idea, but the challenge is in computing . To do this, notice that , so

 ry(Yj) =

Replacing the expectation by empirical average again, we get the following estimator of :

 ˆvj = 1nn∑i=1py|x(Yj|Xi)py(Yj)rx(Xi)px(Xi)=1nn∑i=1pxy(Xi,Yj)px(Xi)py(Yj)Ajiwi.

We can write this expression in matrix form as . We use a kernel density estimator from [34] to estimate the matrix , but our approach is compatible with any density estimator of choice.

Optimization: Given the estimators of the KL divergences, we are able to convert the problem of computing

into an optimization problem over the vector

. Here a constraint of is needed to satisfy . To improve numerical stability, we use as the objective function. Then the optimization problem has the following form:

 maxw log((wTAlog(ATw))−log(wTlogw) subjectto 1nn∑i=1wi=1 wi≥0,∀i

where for short. Although this problem is not convex, we apply gradient descent to maximize the objective. In practice, we initialize for . Hence, the initial is perturbed mildly from . Although we are not guaranteed to achieve the global maximum, we consistently observe in extensive numerical experiments that we have 50%-60% probability of achieving the same maximum value, which we believed to be the global maximum. A theoretical analysis of the landscape of local and global optima and their regions of attraction with respect to gradient descent is an interesting and challenging open question, outside the scope of this paper.

3.1 Consistency of Estimation

While a theoretical understanding of the performance of gradient descent on the optimization step (where the number of samples is fixed) above is technically very challenging, we can study the performance of the solution as the number of samples increases. In particular we show below (under suitable simplifying assumptions to get to the essence of the proof) that the optimal solution to the finite sample optimization problem is consistent. Suppose that is discrete. Further we restrict the optimization over a quantized and bounded set , where is quantized by a gap and satisfies: (1) for all ; (2) . We further assume that we have access of . Define

, then with two further simplifying conditions on the joint distribution (formally stated in Section

5.8), we can prove consistency of our estimation procedure:

Theorem 3.

As goes to infinity, converges to up to a resolution of quantization in probability, i.e., for any , and , we have

 limn→∞P(|ˆsΔ(X;Y)−s(X;Y)|>ε+s(Δ))=0. (11)

4 Experimental results

We present experimental results on synthetic and real datasets showing that the hypercontractivity coefficient is more powerful in detecting potential correlation compared to existing measures; discovers hidden potential correlations among various national indicators in WHO datasets; and is more robust in discovering pathways of gene interactions from gene expression time series data.

4.1 Synthetic data: power test on potential correlation

As our estimator (and the measure itself) involves a maximization, it is possible that we are sensitive to outliers and may capture spurious noise. Via a series of experiments we show that the hypercontractivity coefficient and our estimator are capturing the true potential correlation. As shown in Figure

3, we generate pairs of datasets – one where and are independent and one where there is a potential correlation as per our scenario. We run experiment with eight types of functional associations, following the examples from [5, 35, 36]. For the correlated datasets, out of samples , rare but correlated samples are in and dominant but independent samples are in . The rare but correlated samples are generated as for . The dominant samples are generated as for .

Table 1 shows the hypercontractivity coefficient and the other correlation coefficients for correlated and independent datasets shown in Figure 3, along with the chosen value of and . Correlation estimates with the largest separation for each row is shown in bold. The hypercontractivity coefficient gives the largest separation between the correlated and the independent dataset for most functional types.

A formal statistical approach to test the robustness as well as accuracy is to run power tests: testing for the power of the estimator in binary hypothesis tests. To compute the power of each estimator, we compare the false negative rate at a fixed false positive rate of, say, 5%. We generate 500 independent datasets and 500 correlated datasets. We compute the correlation estimates on 500 independent samples, and take the top 5% as a threshold. We compute the correlation estimates on 500 correlated samples. Power is defined as the fraction of correlated datasets for which the correlation estimate is larger than the threshold.

We show empirically that for linear, quadratic, sine with period 1/2, and the step function, the hypercontractivity coefficient is more powerful as compared to other measures. For a given setting, a larger power means a larger successful detection rate for a fixed false alarm rate. Figure 4 shows the power of correlation estimators as a function of the additive noise level, , for and . The hypercontractivity coefficient is more powerful than other correlation estimators for most functions. The power of all the estimators are very small for sine (period 1/8) and circle functions. This is not surprising given that it is very hard to discern the correlated and independent cases even visually, as shown in Figure 3.

Figure 5 plots the power of correlation estimators as a function of noise level for and . As we can see from these figures, hypercontractivity estimator is more powerful than other correlation estimators for most functions. For circle function, the gap between the power of hypercontractivity estimator and the powers of other estimators is significantly large.

On the other hand, hypercontractivity estimator is power deficient for the cubic function. This is because in estimating hypercontractivity coefficient, we estimate using the kernel density estimator (KDE), which gives a smooth estimate of , i.e., for and close to each other, estimated and are close to each other. Hence, for a correlated dataset for a cubic function, shown in Figure 6 (A)-(right), the estimated does not vary much for . (Estimated for and for are close to each other). This results in a small hypercontracitivy, which in turn results in a low power in the hypothesis testing. To further analyze this effect, we considered the same dataset but with dominant independent samples appear on the left, as shown in Figure 6 (B)-(middle) and (right), and computed the power of hypercontractivity estimator, shown in Figure 6-(B) (left). Hypercontractivity estimator is much more powerful than the one for the original dataset. This is because the estimated for is very different from the estimated for , which results in a large hypercontractivity coefficient for the correlated dataset.

To investigate the dependency of power on more closely, in Figure 7, we plot the power vs. or and . Hypercontractivity estimator is more powerful than other estimators for most , for all functions except for cubic function. For a sine with period 1/8, due to its high frequency, the powers of all the correlation estimators do not increase as increases. Figure 8 plots the power vs. sample size for and . For sine with period 1/2, hypercontractivity estimator is much more powerful than the other estimators for all sample sizes. We can also see that for sine with period 1/8, powers of all correlation estimators do not increase as sample size increases.

4.2 Real data: correlation between indicators of WHO datasets

We computed the hypercontractivity coefficient, MIC, and Pearson correlation of 1600 pairs of indicators for 202 countries in the World Health Organization (WHO) dataset [5]. Figure 9 illustrates that the hypercontractivity coefficient discovers hidden potential correlation (e.g. in (E) - (H)), whereas other measures fail. Scatter plots of Pearson correlation vs. the hypercontractivity coefficient and MIC vs. the hypercontractivity coefficient for all pairs are presented in (A) and (D). The samples for pairs of indicators corresponding to B,C,E – J are shown in (B),(C),(E) - (J), respectively. In Figure 9 (B), it is reasonable to assume that the number of bad teeth per child is uncorrelated with the democracy score. The hypercontractivity coefficient, MIC, and Pearson correlation are all small, as expected. In Figure 9 (C), the correlation between CO emissions and energy use is clearly visible, and all three correlation estimates are close to one.

However, only the hypercontractivity coefficient discovers the hidden potential correlation in Figure 9 (E) – (H). In Figure 9 (E), the data is a mixture of two types of countries – one with small amount of aid received (less than ), and the other with large amount of aid received (larger than ). Dominantly many countries (104 out of 146) belong to the first type (small aid), and for those countries, the amount of aid received and the income growth are independent. For the remaining countries with larger aid received, although those are rare, there is a clear correlation between the amount of aid received and the income growth.

Similarly in Figure 9 (F), there are two types of countries – one with small arms exports (less than ) and the other with large arms exports (larger than ). Dominantly many countries (71 out of 82) belong to the first type, for which the amount of arms exports and the health expenditure are independent. For the remaining countries that belong to the second type, on the other hand, there is a visible correlation between the arms exports and the health expenditure. This is expected as for those countries that export arms the GDP is positively correlated with both arms exports and health expenditure, whereas for those do not have arms industry, these two will be independent.

In Figure 9 (G), for dominant number of countries, the number of male deaths from the colon and rectum cancer is small (145 out of 169 countries have it less than 2000), and it is independent of the amount of health expenditure. On the other hand, for the remaining countries with larger number of male deaths from colon and rectum cancer, the two indicators are positively associated. This is expected as both indicators are positively correlated with the population. Only hypercontractivity discovers this hidden potential correlation. MIC and Pearson correlation are small.

In Figure 9 (H), for dominant number of countries, the number of broadband subscribers is very small and is independent of the private health expenditure; 155 out of 180 countries have broadband subscribers less than . On the other hand, for the remaining countries, the number of broadband subscribers is positively correlated with the private health expenditure. This is as expected because both indicators are positively correlated with the population. Hypercontractivity is large for this dataset, discovering the hidden correlation, whereas all other correlations all small.

In Figure 9 (I), most countries do not have large hydroelectricity facilities, and for those countries, energy use and hydroelectricity consumption are independent (41 out of 53 countries have hydroelectricity 0.25). On the other hand, for the countries which have hydroelectrocity facilities, the amount of total energy use and the amount of hydroelectricity consumption are positively correlated. Hypercontractivity discovers this hidden potential correlation. Unlike in (G) and (H) for which the fraction of correlated samples was only about , in (I), the fraction of correlated samples is about . Hence, Pearson correlation is larger compared to Pearson correlation values for (G) and (H).

In Figure 9 (J), there is one country (Luxembourg) with very large amounts of foreign direct investment net inflow and outflow. Due to this outlier, Pearson correlation is close to 1. Hypercontractivity is also close to 1, whereas MIC is small. To analyze the effect of the outlier in correlation measures, in the following, we compute the correlation measures for samples without an outlier.

4.2.1 How hypercontractivity changes as we remove outliers

Figure 1015, on the left, are shown samples from Figure 9 (E),(F),(G),(H),(I),(J) respectively. On the middle and on the right are shown all samples but one outlier and all samples but two outliers, respectively. By comparing the hypercontractivity coefficients for the three datasets for each pair of indicators, we can analyze the effect of outliers on hypercontractivity. For a comparison, on the top of each figure, we show the estimated hypercontractivity (HC), MIC, Pearson correlation (Cor), distance correlation (dCor), maximal correlation (mCor), and the hypercontractivity for reversed direction (HCR). In Figures 10 and 11, we can see that hypercontractivity is more sensitive to an outlier than other correlation measures.

In Figure 12 (left), the two countries with the largest number of male deaths from the colon and rectum cancer are China and United States. As China is removed from the dataset, in (middle), hypercontractivity remains unchanged. As we also remove United States, in (right), hypercontractivity becomes small, 0.17. This value is still larger than the typical coefficient for two independent indicators (), we can see that hypercontractivity is more sensitive to the outlier than other correlation measures.

In Figure 13, the two countries with the largest number of broadband subscribers are United States and China. When we remove United States from the samples, hypercontractivity becomes close to zero, which also shows hypercontractivity is sensitive to the outliers.

In Figure 14, hypercontractivity remains large even after we remove outliers. The two countries with the largest amount of hydroelectricity consumption are Norway and Iceland. Even after we remove Norway from the samples, as shown in (middle), hypercontractivity remains large. As we further remove one outlier (Iceland) from the samples, as shown in (right), hypercontractivity becomes 0.49.

In Figure 15, (middle), all samples but Luxembourg is shown. We can see that most countries have a very small absolute amount of foreign direct investment net outflows (For 126 out of 157 countries, it is between ), and for those countries, the foreign direct investment net outflow is independent of foreign direct investment net inflows. For the remaining countries, there is a positive association between the outflow and the inflow. Hypercontractivity captures this hidden correlation better than other correlations; hypercontractivity is 0.47, whereas MIC and Pearson correlation are small. If we further remove the rightmost sample, as shown in (right), hypercontractivity becomes small.

Whether we should consider a sample in a rare type as a meaningful sample or as an outlier depends on the application. If we use hypercontractivity to discover a pair of measures for which one variable can be potentially correlated with the other, then we would expect to discover that an aid for a country has potential correlation in the income growth. Other measures will fail. It is possible that hypercontractivity might have larger false positive rate, and depending on the application, one might prefer to error on the side of having more positive cases to be screened by further experiments, surveys, or human judgements.

4.2.2 Hypercontractivity detecting an outlier

In Figure 16 (A) and (B), we show examples of pairs of indicators for which there is one outlier and the remaining samples are independent, but hypercontractivity is large. As shown in Figure 16 (A) and (B) (left), hypercontractivity is close to 1, when there is an outlier. As shown in (right), hypercontractivity is close to 0, when the outlier is removed. This implies that one single outlier can make the hypercontractivity large. We can see similar patterns for other correlation measures, such as for Pearson correlation, distance correlation, and maximal correlation for both (A) and (B), and MIC for (B), but are less sensitive than hypercontractivity.

To further study how hypercontractivity estimator is affected by outliers, we ran simulations on synthetic data. We generated three sets of synthetic data shown in Figure 17 and computed hypercontractivity coefficients. In Figure 17 (left), an outlier is located far from the rest of samples, and the estimated hypercontractivity coefficient is 0.99. In Figure 17 (middle), an outlier is located close to the rest of samples, and the estimated hypercontractivity coefficient is 0.04. In Figure 17 (right),  and are potentially correlated, and the hypercontractivity estimate is 0.17. As can bee seen from this simulation and experimental results on WHO dataset, our hypercontractivity estimator is sensitive to outliers. If one wants to filter out the effect of outliers, one can combine methods for robust estimation, such as trimming and winsorizing [37, 38, 39], along with the hypercontractivity estimator. This is an interesting future research direction.

4.3 Gene pathway recovery from single cell data

We replicate the genetic pathway detection experiment from [7], and show that hypercontractivity correctly discovers the genetic pathways from smaller number of samples. A genetic pathway is a series of genes interacting with each other as a chain. Consider the following setup where four genes whose expression values in a single cell are modeled by random processes , , and respectively. These 4 genes interact with each other following a pathway ; it is biologically known that causes with a negligible delay, and later at time , causes , and so on. Our goal is to recover this known gene pathway from sampled datapoints. For a sequence of time points , we observe i.i.d. samples generated from the random process . We use the real data obtained by the single-cell mass flow cytometry technique [7].

Given these samples from time series, the goal of [7] is to recover the direction of the interaction along the known pathway using correlation measures as follows, where they proposed a new measure called DREMI. The DREMI correlation measure is evaluated on each pairs on the pathway, , and , at each time points . It is declared that a genetic pathway is correctly recovered if the peak of correlation follows the expected trend: . In [25], the same experiment has been done with evaluated by UMI and CMI estimators. In this paper, we evaluate using our proposed estimator of hypercontractivity.

The Figure 18 shows the scatter plots pCD3-pSLP76-pERK-pS6 chain at different time points after TCR activation. The data comes from CD4+ naïve T lymphocytes from B6 mice with CD3, CD28, and CD4 cross-linking. Each row represents a pair of data in the chain, and each column stands for a time point after TCR activation. Estimate of hypercontractivity is shown below the scatter plot for each pair of data and each time point and we highlight the time point where each pair of data is maximally correlated. We can see that the peak of the correlation of pCD3-pSLP76, pSLP76-pERK and pERK-pS6 appears at 0.5 min, 1 min and 2 min respectively, hence the pathway is correctly identified. In Figure 19, the similar plots was shown for T-cells exposed with an antigen. Similarly, hypercontractivity is able to capture the trend.

We subsample the raw data from [7] to evaluate the ability to find the trend from smaller samples. Precisely, given a resampling rate , we randomly select a subset of indices with , compute , and from subsamples , and determine whether we can recover the trend successfully, i.e., whether . We repeat the experiment several times with independent subsamples and compute the probability of successfully recovering the trend. Figure 20 illustrates that when the entire dataset is available, all methods are able to recover the trend correctly. When only fewer samples are available, hypercontractivity improves upon other competing measures in recovering the hidden chronological order of interactions of the pathway. For completeness, we run datasets for both regular T-cells (shown in left figure) and T-cells exposed with an antigen (shown right figure), for which we expect distinct biological trends. Hypercontractivity method can capture the trend for both datasets correctly and sample-efficiently.

5 Proofs

In this section, we provide proofs for our main results and technical lemmas.

5.1 Proof of Proposition 1

Let for satisfying conditions in Proposition 1. We show that satisfies all Rényi’s axioms, i.e., Axioms 1-5 and 6’ and 7’.

1. is defined for any pair of non-constant random variables because and are defined for any random variables , by Theorem 1.

2. because the output of a function is in by the condition on .

3. If and are statistically independent, . By the condition on , it follows that . If , by the condition on , , which implies that and are statistically independent.

4. for any bijective Borel-measurable functions because and by Theorem 1.

5. For with Pearson correlation , . Hence, .

6. If for a non-constant function , it follows that because if is discrete, and otherwise, . Hence

 s(X;f(X))=supU−X−f(X)I(U;f(X))/I(U;X)=I(f(X);f(X))/I(f(X);X)=1.

Similarly, . Hence, . Likewise, we can show that if .

7. because .

5.2 Proof of Theorem 1

We show that satisfies Axioms 1-6 in Section 2.

1. For any non-constant random variable , s.t. . Hence, is defined for any pair of non-constant random variables and .

2. Since mutual information is non-negative, . By data processing inequality, for any , . Hence, .

3. If and are independent, for any , . Hence, . If and are dependent, , which implies that .

4. For any bijective functions ,

Similarly, . Hence,

 s(f(X);g(Y)) =supU:U−f(X)−g(Y),I(U;f(X))>0I(U;g(Y))I(U;f(X)) =supU:U−X−f(X)−g(Y)−Y,I(U;X)>0I(U;Y)I(U;X) =s(X;Y).
5. By Theorem 3.1 in [33], for jointly Gaussian with correlation coefficient ,

 minU:U−X−Y(I(U;X)−βI(U;Y))=0

for . Equivalently,

 maxU:U−X−Y(I(U;Y)−ρ2I(U;X))=0,

which implies that