On Conditional Correlations

1 Introduction

In the literature, there are various measures available to quantify the strength of dependence between two random variables, including the Pearson correlation coefficient, the correlation ratio, the maximal correlation coefficient, etc. The Pearson correlation coefficient is such a well-known measure that quantifies the linear dependence between two real-valued random variables. For real-valued random variables

X

and

Y

, it is defined as

ρ (X; Y) = ⎧ ⎨ ⎩ \begin{matrix} \frac{c o v (X, Y)}{\sqrt{v a r (X)} \sqrt{v a r (Y)}}, & v a r (X) v a r (Y) > 0, 0, & v a r (X) v a r (Y) = 0. \end{matrix}

The correlation ratio was introduced by Pearson (see e.g. [6]), and studied by Rényi [15, 14]. For a real-valued random variable $X$ and an arbitrary random variable $Y$ , the correlation ratio of $X$ on $Y$ is defined by

θ (X; Y) = sup g ρ (X; g (Y)),

where the supremum is taken over all Borel-measurable real-valued functions $g (y)$ such that $v a r (g (Y)) < \infty$ . It was shown that

θ (X; Y)

= \sqrt{\frac{v a r (E [X | Y])}{v a r (X)}} = \sqrt{1 - \frac{E [v a r (X | Y)]}{v a r (X)}} .

Another related dependence measure is Hirscbfeld-Gebelein-Rényi maximal correlation (or simply maximal correlation), which measures the maximum possible (Pearson) correlation between square integrable real-valued random variables generated by either of two random variables. For two arbitrary random variables $X$ and $Y$ , the maximal correlation of $X$ and $Y$ is defined by

ρ_{m} (X; Y) = sup f, g ρ (f (X); g (Y)),

where the supremum is taken over all Borel-measurable real-valued functions $f (x), g (y)$ such that $v a r (f (X)),$ $v a r (g (Y)) < \infty$ . This measure was first introduced by Hirschfeld [10] and Gebelein [8], then studied by Rényi [15], and recently it has been exploited to study some interesting problems in information theory, such as measuring non-local correlations [3], maximal correlation secrecy [12], deriving a converse result for distributed communication [18], etc. Furthermore, the maximal correlation also indicates the existence of Gács-Körner’s or Wyner’s common information [7, 17].

2 Definition

Let $(Ω, Σ, P)$

be a probability space. Let

(X, Y, Z, U) : (Ω, Σ) \to (R^{4}, B_{R}^{\otimes 4})

be a real-valued random vector, where

B_{R}

denotes the Borel

σ

-algebra on

R

. For a random variable (or random vector)

W

, we denote the probability measure (a.k.a. distribution) as

P_{W}

. If

W

is discrete, then we use

P_{W}

to denote the probability mass function (pmf). If

W

is absolutely continuous (the distribution is absolutely continuous respect to the Lebesgue measure), then we use

p_{W}

to denote the probability density function (pdf).

In the following, we define several conditional correlations, including the conditional (Pearson) correlation, conditional correlation ratio, and conditional maximal correlation.

Definition 1.

The conditional (Pearson) correlation¹¹1Here $U$ does not need to be real-valued. But for brevity, we assume it is. Similarly, in the following, $(Y, U)$ does not need to be real-valued in the definition of conditional correlation ratio, and $(X, Y, U)$ does not need to be real-valued in the definition of conditional maximal correlation. of $X$ and $Y$ given $U$ is defined by

ρ (X; Y | U) = ⎧ ⎨ ⎩ \begin{matrix} \frac{E [c o v (X, Y | U)]}{\sqrt{E [v a r (X | U)]} \sqrt{E [v a r (Y | U)]}}, & E [v a r (X | U)] E [v a r (Y | U)] > 0, 0, & E [v a r (X | U)] E [v a r (Y | U)] = 0. \end{matrix}

Definition 2.

The conditional correlation ratio of $X$ on $Y$ given $U$ is defined by

θ (X; Y | U) = sup g ρ (X; g (Y, U) | U),

where the supremum is taken over all Borel-measurable real-valued functions $g (y, u)$ such that $E [v a r (g (Y, U) | U)] < \infty$ .

Definition 3.

The conditional maximal correlation of $X$ and $Y$ given $U$ is defined by

ρ_{m} (X; Y | U) = sup f, g ρ (f (X, U); g (Y, U) | U),

where the supremum is taken over all Borel-measurable real-valued functions $f (x, u), g (y, u)$ such that $E [v a r (f (X, U) | U)]$ , $E [v a r (g (Y, U) | U)] < \infty$ .

Remark 1.

If $U$ is degenerate, then these three conditional correlations reduce to the unconditional versions.

Remark 2.

Note that $ρ (X; Y | U) = ρ (Y; X | U)$ and $ρ_{m} (X; Y | U) = ρ_{m} (Y; X | U)$ , but in general $θ (X; Y | U) \neq θ (Y; X | U)$ . That is, the conditional correlation and conditional maximal correlation are symmetric, but the conditional correlation ratio is not.

By the definitions, it is easy to verify that

ρ_{m} (X; Y | U) = sup f θ (f (X, U); Y | U),

(1)

where the supremum is taken over all Borel-measurable real-valued functions $f (x, u)$ such that $E [v a r (f (X, U) | U)] < \infty$ .

Note that the unconditional versions of correlation coefficient, correlation ratio, and maximal correlation were well studied in the literature; see [15, 14]. The conditional version of maximal correlation was first introduced by Ardestanizadeh et al. [1]. Later Beigi and Gohari [3] used it to study the problem of non-local correlations. In this paper, we study these conditional correlations, especially the conditional maximal correlation, and derive some useful properties. Furthermore, to state our results clearly, we also need to define event conditional correlations as follows.

Definition 4.

Given an event $A$ , denote the conditional distribution of $(X, Y)$ given $A$ as $P_{X, Y | A}$ . Assume $(X^{'}, Y^{'})$ is a pair of random variables satisfying $(X^{'}, Y^{'}) \sim P_{X, Y | A}$ . Then we define $κ (X; Y | A) := κ (X^{'}; Y^{'})$ as the event conditional correlations of $X$ and $Y$ given $A$ , where $κ \in {ρ, θ, ρ_{m}}$ and $κ (X^{'}; Y^{'})$ denotes the corresponding unconditional correlation of $X^{'}$ and $Y^{'}$ .

Obviously, event conditional correlations are special cases of corresponding conditional correlations. Moreover, if the distribution of $(X^{'}, Y^{'})$ is the same as the conditional distribution of $(X, Y)$ given $U = u$ , then the unconditional correlations of $(X^{'}, Y^{'})$ respectively equal the corresponding event conditional correlations of $(X, Y)$ given $U = u$ , i.e., $κ (X^{'}; Y^{'}) = κ (X; Y | U = u)$ where $κ \in {ρ, θ, ρ_{m}}$ . If the distribution of $U$ satisfies $P (U = u) = 1$ for some $u$ , then the conditional correlations of $(X, Y)$ given $U$ respectively equal the corresponding event conditional correlations of $(X, Y)$ given $U = u$ , i.e., $κ (X; Y | U) = κ (X; Y | U = u)$ where $κ \in {ρ, θ, ρ_{m}}$ .

3 Properties

3.1 Basic Properties: Other Characterizations, Continuity, and Concavity

In this subsection, we provide other characterizations for the conditional correlation ratio and conditional maximal correlation, and then study continuity (or discontinuity) and concavity properties of the conditional maximal correlation. First by the definitions, we have the following basic properties.

Theorem 1.

For any random variables $X, Y, Z, U$ , the following inequalities hold.

	$θ (X; Y, Z \| U)$	$\geq θ (X; Y \| U);$
	$ρ_{m} (X; Y, Z \| U)$	$\geq ρ_{m} (X; Y \| U) .$

Next we characterize the conditional correlation ratio and conditional maximal correlation by ratios of variances.

Theorem 2.

(Characterization by the ratio of variances). For any random variables $X, Y, Z, U$ , the following properties hold.

$θ (X; Y \| U)$	$= \sqrt{\frac{E [v a r (E [X \| Y, U] \| U)]}{E [v a r (X \| U)]}}$
	$= \sqrt{1 - \frac{E [v a r (X \| Y, U)]}{E [v a r (X \| U)]}};$	(2)
$ρ_{m} (X; Y \| U)$	$= sup f \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U)]}{E [v a r (f (X, U) \| U)]}}$
	$= sup f \sqrt{1 - \frac{E [v a r (f (X, U) \| Y, U)]}{E [v a r (f (X, U) \| U)]}} .$	(3)

Remark 3.

The correlation ratio is also closely related to the Minimum Mean Square Error (MMSE). The optimal MMSE estimator is

E [X | Y, U]

, hence the variance of the MMSE for estimating

X

given

(Y, U)

m m s e (X | Y, U) = E (X - E [X | Y, U])^{2} = E [v a r (X | Y, U)] = E [v a r (X | U)] (1 - θ^{2} (X; Y | U)) .

The unconditional version of Theorem 2 was proven by Rényi [15]. Theorem 2 can be proven by a proof similar to that in [15], and hence omitted here. Next we characterize conditional correlations by event conditional versions.

Theorem 3.

(Characterization by event conditional correlations). For any random variables $X, Y, U,$

$ρ (X; Y \| U)$	$\leq e s s s u p u ρ (X; Y \| U = u),$	(4)
$e s s i n f u θ (X; Y \| U = u) \leq θ (X; Y \| U)$	$\leq e s s s u p u θ (X; Y \| U = u),$	(5)
$ρ_{m} (X; Y \| U)$	$= e s s s u p u ρ_{m} (X; Y \| U = u),$	(6)

where ${e s s s u p}_{u} f (u) := inf {λ : P (f (U) > λ) = 0}$ and ${e s s i n f}_{u} f (u) := sup {λ : P (f (U) < λ) = 0}$ respectively denote the essential supremum and the essential infimum of $f$ .

Remark 4.

It is worth noting that $ρ (X; Y | U) \geq {e s s i n f}_{u} ρ (X; Y | U = u)$ does not hold in general. This can be seen from the following example. Assume $(a, b, η)$ are three numbers such that $0 < η \leq 1, 0 < a < b$ . Suppose that $(W, Z)$ is a pair of random variables such that $v a r (W) = a, v a r (Z) = b$ and $ρ (W; Z) = η$ . (It is obvious that there are many random variable pairs satisfying the conditions.) Denote the distribution of $(W, Z)$ as $P_{W, Z}$ . Now we consider a triple of random variables $(X, Y, U)$ such that $P_{U} (0) = P_{U} (1) = \frac{1}{2}$ and $(X, Y) | U = 0 \sim P_{W, Z}$ and $(X, Y) | U = 1 \sim P_{Z, W}$ . Then we have $ρ (X; Y | U = 0) = ρ (X; Y | U = 1) = η$ . Hence ${e s s i n f}_{u} ρ (X; Y | U = u) = η$ . However, $ρ (X; Y | U) = \frac{2 η \sqrt{a b}}{a + b} < η$ . Hence $ρ (X; Y | U) < {e s s i n f}_{u} ρ (X; Y | U = u)$ for this example.

Remark 5.

If $U$

is a discrete random variable, then

ρ_{m} (X; Y | U) = sup u : P_{U} (u) > 0 ρ_{m} (X; Y | U = u),

(7)

where $P_{U}$ denotes the pmf of $U$ . Note that Beigi and Gohari [3] defined the conditional maximal correlation via (7). This theorem implies the equivalence between the conditional maximal correlation defined by us and that defined by Beigi and Gohari.

Remark 6.

If $U$

is an absolutely continuous random variable, then

ρ_{m} (X; Y | U) = inf q_{U} : q_{U} = p_{U} a . e . sup u : q_{U} (u) > 0 ρ_{m} (X; Y | U = u),

where $p_{U}$ denotes the pdf of $U$ .

Proof.

We first prove (4). Denote $A_{λ} := {u : ρ (X; Y | U = u) > λ}$ and $λ^{*} := inf {λ : P_{U} (A_{λ}) = 0}$ . Hence $P_{U} (A_{λ}) = 0$ for any $λ > λ^{*}$ ; and $P_{U} (A_{λ}) > 0$ for any $λ < λ^{*}$ . It means that $λ^{*} = {e s s s u p}_{u} ρ (X; Y | U = u)$ . Therefore, to show (4), we only need to show $ρ (X; Y | U) \leq λ^{*}$ . We can upper bound $ρ (X; Y | U)$ as follows.

$ρ (X; Y \| U)$	$= \frac{E [c o v (X, Y \| U)]}{\sqrt{E [v a r (X \| U)]} \sqrt{E [v a r (Y \| U)]}}$	(8)
	$\leq \frac{E [c o v (X, Y \| U)]}{E \sqrt{v a r (X \| U) v a r (Y \| U)}}$	(9)
	$= inf λ > λ^{*} \frac{E [c o v (X, Y \| U) \cdot 1 {U \in R ∖ A_{λ}}]}{E [\sqrt{v a r (X \| U) v a r (Y \| U)} \cdot 1 {U \in R ∖ A_{λ}}]}$	(10)
	$\leq inf λ > λ^{*} sup u \in R ∖ A_{λ} ρ (X; Y \| U = u)$
	$\leq inf λ > λ^{*} λ$
	$= λ^{*},$	(11)

where (9) follows by the Cauchy-Schwarz inequality, and (10) follows from Theorem 15.2 of [4] and the fact $P_{U} (A_{λ}) = 0$ for any $λ > λ^{*}$ .

From (2), (5) follows immediately.

Finally, we prove (6). Similarly as in the proof above, we denote $A_{λ} := {u : ρ_{m} (X; Y | U = u) > λ}$ and $λ^{*} := inf {λ : P_{U} (A_{λ}) = 0}$ . Hence $P_{U} (A_{λ}) = 0$ for any $λ > λ^{*}$ ; $P_{U} (A_{λ}) > 0$ for any $λ < λ^{*}$ ; and $λ^{*} = {e s s s u p}_{u} ρ_{m} (X; Y | U = u)$ . Therefore, to prove (6), we only need to show $ρ_{m} (X; Y | U) = λ^{*}$ . On one hand, by derivations similar as (8)-(11), we can upper bound $ρ_{m} (X; Y | U)$ as follows.

	$ρ_{m} (X; Y \| U)$	$= sup f \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U)]}{E [v a r (f (X, U) \| U)]}}$
		$= sup f inf λ > λ^{*} \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U) \cdot 1 {U \in R ∖ A_{λ}}]}{E [v a r (f (X, U) \| U) \cdot 1 {U \in R ∖ A_{λ}}]}}$
		$\leq sup f inf λ > λ^{*} sup u \in R ∖ A_{λ} \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U = u)]}{E [v a r (f (X, U) \| U = u)]}}$
		$\leq inf λ > λ^{*} sup u \in R ∖ A_{λ} sup f \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U = u)]}{E [v a r (f (X, U) \| U = u)]}}$

		$\leq inf λ > λ^{*} λ$
		$= λ^{*} .$

On the other hand, we assume $˜ f (x, u)$ is a function such that $\sqrt{\frac{v a r (E [˜ f (X, U) | Y, U = u] | U = u)}{v a r (˜ f (X, U) | U = u)}} \geq α ρ_{m} (X; Y | U = u)$ for each $u \in A_{λ}$ , where $λ < λ^{*}$ and $0 < α < 1$ . The existence of $˜ f (x, u)$ follows from the definition of $ρ_{m} (X; Y | U = u)$ . According to the definition of $A_{λ}$ , we have that $P_{U} (A_{λ}) > 0$ , and for each $u \in A_{λ}$ ,

\frac{v a r (E [˜ f (X, U) | Y, U = u] | U = u)}{v a r (˜ f (X, U) | U = u)} \geq {(α λ)}^{2} .

(12)

Set $f (x, u) = ˜ f (x, u) \cdot 1 {u \in A_{λ}}$ . Then

$ρ_{m} (X; Y \| U)$	$\geq \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U)]}{E [v a r (f (X, U) \| U)]}}$
	$= \sqrt{\frac{E [v a r (E [˜ f (X, U) \| Y, U] \| U) \cdot 1 {U \in A_{λ}}]}{E [v a r (˜ f (X, U) \| U) \cdot 1 {U \in A_{λ}}]}}$
	$\geq \sqrt{inf u \in A_{λ} \frac{v a r (E [˜ f (X, U) \| Y, U = u] \| U = u)}{v a r (˜ f (X, U) \| U = u)}}$
	$\geq α λ,$	(13)

where (13) follows from (12). Since $λ < λ^{*}$ and $0 < α < 1$ are arbitrary, we have $ρ_{m} (X; Y | U) \geq λ^{*}$ .

Combining the two points above, we have $ρ_{m} (X; Y | U) = λ^{*}$ . ∎

For discrete $(X, Y)$ with finite supports, without loss of generality, the supports of $X$ and $Y$ are assumed to be ${1, 2, . . ., m}$ and ${1, 2, . . ., n}$ , respectively. For this case, denote $λ_{2} (u)$

as the second largest singular value of the matrix

Q_{u}

with entries

Q_{u} (x, y) := \frac{P (x, y | u)}{\sqrt{P (x | u) P (y | u)}} = \frac{P (x, y, u)}{\sqrt{P (x, u) P (y, u)}} .

For absolutely-continuous $X, Y$ , denote $λ_{2} (u)$ as the second largest singular value of the bivariate function $\frac{p (x, y | u)}{\sqrt{p (x | u) p (y | u)}}$ , where $p (x, y | u)$ denotes a conditional pdf of $(X, Y)$ respect to $U$ . Then we have the following singular value characterization of conditional maximal correlation.

Theorem 4.

(Singular value characterization). Assume $X, Y$ are discrete random variables with finite supports, or absolutely-continuous random variables such that $\int_{x, y} {(\frac{p (x, y | u)}{\sqrt{p (x | u) p (y | u)}})}^{2} d x d y < \infty$ a.s. Then

ρ_{m} (X; Y | U) = e s s s u p u λ_{2} (u) .

(14)

Remark 7.

This property is consistent with the one of the unconditional version by setting $U$ to a constant, i.e., $ρ_{m} (X; Y) = λ_{2} .$

Proof.

The unconditional version of this theorem was proven in [17]. That is, for discrete $X, Y$ with finite supports, $ρ_{m} (X; Y)$ equals the second largest singular value $λ_{2}$ of the matrix $Q$ with entries $Q (x, y) := \frac{P (x, y)}{\sqrt{P (x) P (y)}}$ ; for absolutely-continuous $X, Y$ such that $\int_{x, y} {(\frac{p (x, y)}{\sqrt{p (x) p (y)}})}^{2} d x d y < \infty$ with $p (x, y)$ denoting a pdf of $(X, Y)$ , $ρ_{m} (X; Y)$ equals the second largest singular value $λ_{2}$ of the bivariate function $\frac{p (x, y)}{\sqrt{p (x) p (y)}}$ . Combining this with Theorem 3, we have (14). ∎

Note that, $ρ_{m} (X; Y | U)$ is a mapping that maps a distribution $P_{X, Y, U}$ to a real number in $[0, 1]$ . Now we study the concavity of such a mapping.

Corollary 1.

(Concavity). Given $P_{X, Y | U}$ , $ρ_{m} (X; Y | U)$ is concave in $P_{U}$ . That is, for any distributions $P_{U}$ and $Q_{U}$ , and any $λ \in [0, 1]$ , $ρ_{m}^{(((1 - λ) P_{U} + λ Q_{U}) P_{X, Y | U})} (X; Y | U) \geq (1 - λ) ρ_{m}^{(P_{U} P_{X, Y | U})} (X; Y | U) + λ ρ_{m}^{(Q_{U} P_{X, Y | U})} (X; Y | U)$ , where $ρ_{m}^{(Q_{X, Y, U})} (X; Y | U)$ denotes the conditional maximal correlation under distribution $Q_{X, Y, U}$ .

Proof.

This theorem directly follows from the characterization in (6). ∎

For a discrete random variable, the distribution is uniquely determined by its pmf. Therefore, for discrete random variables $(X, Y, U)$ , $ρ_{m} (X; Y | U)$ can be also seen as a mapping that maps a pmf $P_{X, Y, U}$ to a real number in $[0, 1]$ . Assume $X, Y, U \subset R$ are three finite sets. Denote $P (X \times Y \times U)$ as the set of pmfs defined on $X \times Y \times U$ (i.e., the $| X | | Y | | U | - 1$ dimensional probability simplex). Consider $ρ_{m} (X; Y | U)$ as a mapping $ρ_{m} (X; Y | U) : P (X \times Y \times U) \to [0, 1]$ . Now we study the continuity (or discontinuity) of such a mapping.

Corollary 2.

(Continuity and discontinuity). For finite sets $X, Y, U \subset R$ , $ρ_{m} (X; Y | U)$ is continuous (under the total variation distance) on ${P_{X, Y, U} \in P (X \times Y \times U) : P_{U} (u) > 0, \forall u \in U}$ . But in general, $ρ_{m} (X; Y | U)$ is discontinuous at $P_{U}$ such that $P_{U} (u) = 0, \exists u \in U$ .

Proof.

For a pmf $P_{X, Y, U} \in P (X \times Y \times U)$ , $ρ_{m} (X; Y | U) = max u : P (u) > 0 λ_{2} (u)$ . On the other hand, singular values are continuous in the matrix (see [9, Corollary 8.6.2]), hence $λ_{2} (u)$ is continuous in $P_{X, Y | U = u}$ . Furthermore, since $P_{U} (u) > 0, \forall u \in U$ , $Q_{X, Y, U} \to P_{X, Y, U}$ in the total variation distance sense implies $Q_{U} \to P_{U}$ and $Q_{X, Y | U = u} \to P_{X, Y | U = u}, \forall u \in U$ . Therefore, $ρ_{m}^{(Q)} (X; Y | U) \to ρ_{m} (X; Y | U)$ , where $ρ_{m}^{(Q)} (X; Y | U)$ denotes the conditional maximal correlation under distribution $Q_{X, Y, U}$ . However, if there exists $u_{0} \in U$ such that $P_{U} (u_{0}) = 0$ and $λ_{2} (u_{0}) > {max}_{u : P (u) > 0} λ_{2} (u)$ . Then letting $Q_{U} \to P_{U}$ in a direction such that $Q_{U} (u_{0}) > 0$ always holds, we have that $ρ_{m}^{(Q)} (X; Y | U) \geq λ_{2} (u_{0}) > ρ_{m} (X; Y | U)$ always holds. This implies $ρ_{m} (X; Y | U)$ is discontinuous at $P_{X, Y, U}$ . ∎

Theorem 5.

1) For any random variables $X, Y, U$ ,

0 \leq | ρ (X; Y | U) | \leq θ (X; Y | U) \leq ρ_{m} (X; Y | U) \leq 1.

2) Moreover, $ρ_{m} (X; Y | U) = 0$ if and only if $X$ and $Y$ are conditionally independent given $U$ . Furthermore, for discrete random variables $X, Y, U$ with finite supports, $ρ_{m} (X; Y | U) = 1$ if and only if $f (X, U) = g (Y, U)$ for some functions $f$ and $g$ such that $H (f (X, U) | U) > 0$ (i.e., $X$ and $Y$ have conditional Gács-Körner common information given $U$ [7]), where $H (Z | W) := - E log P_{Z | W} (Z | W)$ denotes the conditional entropy of $Z$ given $X$ .

Proof.

The statement 1) follows from the definitions of these three conditional correlations. The statement 2) with degenerate $U$ (unconditional version) was proven by Rényi [15]. The statement 2) (conditional version) follows by combining Rényi’s results and the characterization in (6). ∎

Next we show that the three conditional correlations are equal for the Gaussian case.

Theorem 6.

(Gaussian case). For jointly Gaussian random variables $X, Y, U$ , we have

| ρ (X; Y | U) | = θ (X; Y | U) = θ (Y; X | U) = ρ_{m} (X; Y | U) .

(15)

Proof.

The unconditional version of (15) was proven in [16, Sec. IV, Lem. 10.2]. On the other hand, given $U = u,$ $(X, Y)$

also follows jointly Gaussian distribution, and

ρ (X; Y | U = u) = ρ (X; Y | U)

for any

u

. Hence

$ρ_{m} (X; Y \| U)$	$= e s s s u p u ρ_{m} (X; Y \| U = u)$	(16)
	$= e s s s u p u \| ρ (X; Y \| U = u) \|$	(17)
	$= \| ρ (X; Y \| U) \|,$

where (16) follows from Theorem 3, and (17) follows from [16, Sec. IV, Lem. 10.2].

Furthermore, both $θ (X; Y | U)$ and $θ (Y; X | U)$ are between $ρ_{m} (X; Y | U)$ and $| ρ (X; Y | U) |$ . Hence (15) holds. ∎

3.2 Other Properties: Tensorization, DPI, Correlation ratio equality, and Conditioning reducing covariance gap

The tensorization property and the data processing inequality for the unconditional maximal correlation were proven in

[17, Thm. 1] and [20, Lem. 2.1] respectively. Here we extend them to the conditional case.

Theorem 7.

(Tensorization). Assume given $U,$ $(X^{n}, Y^{n})$ with $X^{n} := (X_{1}, X_{2}, . . ., X_{n})$ and $Y^{n} := (Y_{1}, Y_{2}, . . ., Y_{n})$ , is a sequence of pairs of conditionally independent random variables, then we have

ρ_{m} (X^{n}; Y^{n} | U) = max 1 \leq i \leq n ρ_{m} (X_{i}; Y_{i} | U) .

Proof.

The unconditional version

for a sequence of pairs of independent random variables $(X^{n}, Y^{n})$ is proven in [17, Thm. 1]. Hence the result for the event conditional maximal correlation also holds. Using this result and Theorem 4, we have

$ρ_{m} (X^{n}; Y^{n} \| U)$	$= e s s s u p u ρ_{m} (X^{n}; Y^{n} \| U = u)$
	$= e s s s u p u max 1 \leq i \leq n ρ_{m} (X_{i}; Y_{i} \| U = u)$
	$= max 1 \leq i \leq n e s s s u p u ρ_{m} (X_{i}; Y_{i} \| U = u)$	(18)
	$= max 1 \leq i \leq n ρ_{m} (X_{i}; Y_{i} \| U),$

where (18) follows by the following lemma.

Lemma 1.

Assume $I$ is a countable set, and $P_{U}$ is an arbitrary distribution on $(R, B_{R})$ . Then for any function $f : I \times R \to R$ , we have

e s s s u p u sup i \in I f (i, u) = sup i \in I e s s s u p u f (i, u) .

This lemma follows from the following two points. For a number $ϵ > 0$ , assume $i^{*} \in I$ satisfies that ${e s s s u p}_{u} f (i^{*}, u) \geq {sup}_{i \in I} {e s s s u p}_{u} f (i, u) - ϵ$ . Then for any function $f$ ,

e s s s u p u sup i \in I f (i, u) \geq e s s s u p u f (i^{*}, u) = sup i \in I e s s s u p u f (i, u) - ϵ .

Since $ϵ > 0$ is arbitrary, we have ${e s s s u p}_{u} {sup}_{i \in I} f (i, u) \geq {sup}_{i \in I} {e s s s u p}_{u} f (i, u)$ .

On the other hand, denote $λ_{i}^{*} := {e s s s u p}_{u} f (i, u)$ . Then by the definition of $e s s s u p$ , we have $P_{U} (u : f (i, u) > λ_{i}^{*}) = 0$ for all $i \in I$ . Hence by the union bound, we have $P_{U} (u : \exists i \in I s.t. f (i, u) > λ_{i}^{*}) = 0$ . Furthermore, for a number $ϵ > 0$ , ${sup}_{i \in I} f (i, u) > {sup}_{i \in I} λ_{i}^{*} + ϵ$ implies that there exists an $i^{'} \in I$ such that $f (i^{'}, u) \geq {sup}_{i \in I} f (i, u) - ϵ > {sup}_{i \in I} λ_{i}^{*} \geq λ_{i^{'}}^{*}$ . Hence

	$P_{U} (u : sup i \in I f (i, u) > sup i \in I λ_{i}^{*} + ϵ)$	$\leq P_{U} (u : \exists i \in I s.t. f (i, u) > λ_{i}^{*})$
		$= 0.$

By the definition of $e s s s u p$ , we have ${e s s s u p}_{u} {sup}_{i \in I} f (i, u) \leq {sup}_{i \in I} λ_{i}^{*} + ϵ$ . Since $ϵ > 0$ is arbitrary, we have ${e s s s u p}_{u} {sup}_{i \in I} f (i, u) \leq {sup}_{i \in I} {e s s s u p}_{u} f (i, u)$ . ∎

Theorem 8.

(Data processing inequality). If random variables $X, Y, Z, U$

form a Markov chain

X \to (Z, U) \to Y

(i.e.,

X

and

Y

are conditionally independent given

(Z, U)

), then

$\| ρ (X; Y \| U) \|$	$\leq θ (X; Z \| U) θ (Y; Z \| U),$	(19)
$θ (X; Y \| U)$	$\leq θ (X; Z \| U) ρ_{m} (Y; Z \| U),$	(20)
$ρ_{m} (X; Y \| U)$	$\leq ρ_{m} (X; Z \| U) ρ_{m} (Y; Z \| U) .$	(21)

Moreover, equalities hold in (19)-(21) if $(X, Z, U)$ and $(Y, Z, U)$

have the same joint distribution.

Proof.

Consider

$E [c o v (X, Y \| U)]$	$= E [(X - E [X \| U]) (Y - E [Y \| U])]$
	$= E [E [(X - E [X \| U]) (Y - E [Y \| U]) \| Z, U]]$
		(22)
	$= E [(E [X \| Z, U] - E [X \| U]) (E [Y \| Z, U] - E [Y \| U])]$
	$\leq \sqrt{E [(E [X \| Z, U] - E [X \| U])^{2}] E [(E [Y \| Z, U] - E [Y \| U])^{2}]}$	(23)
	$= \sqrt{E [v a r (E [X \| Z, U] \| U)] E [v a r (E [Y \| Z, U]}$

	$θ (X; Y, Z \| U)$	$\geq θ (X; Y \| U);$
	$ρ_{m} (X; Y, Z \| U)$	$\geq ρ_{m} (X; Y \| U) .$

	$ρ_{m} (X; Y \| U)$	$= sup f \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U)]}{E [v a r (f (X, U) \| U)]}}$
		$= sup f inf λ > λ^{*} \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U) \cdot 1 {U \in R ∖ A_{λ}}]}{E [v a r (f (X, U) \| U) \cdot 1 {U \in R ∖ A_{λ}}]}}$
		$\leq sup f inf λ > λ^{*} sup u \in R ∖ A_{λ} \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U = u)]}{E [v a r (f (X, U) \| U = u)]}}$
		$\leq inf λ > λ^{*} sup u \in R ∖ A_{λ} sup f \sqrt{\frac{E [v a r (E [f (X, U) \| Y, U] \| U = u)]}{E [v a r (f (X, U) \| U = u)]}}$

		$\leq inf λ > λ^{*} λ$
		$= λ^{*} .$

$ρ_{m} (X^{n}; Y^{n} \| U)$	$= e s s s u p u ρ_{m} (X^{n}; Y^{n} \| U = u)$
	$= e s s s u p u max 1 \leq i \leq n ρ_{m} (X_{i}; Y_{i} \| U = u)$
	$= max 1 \leq i \leq n e s s s u p u ρ_{m} (X_{i}; Y_{i} \| U = u)$	(18)
	$= max 1 \leq i \leq n ρ_{m} (X_{i}; Y_{i} \| U),$

$E [c o v (X, Y \| U)]$	$= E [(X - E [X \| U]) (Y - E [Y \| U])]$
	$= E [E [(X - E [X \| U]) (Y - E [Y \| U]) \| Z, U]]$
		(22)
	$= E [(E [X \| Z, U] - E [X \| U]) (E [Y \| Z, U] - E [Y \| U])]$
	$\leq \sqrt{E [(E [X \| Z, U] - E [X \| U])^{2}] E [(E [Y \| Z, U] - E [Y \| U])^{2}]}$	(23)
	$= \sqrt{E [v a r (E [X \| Z, U] \| U)] E [v a r (E [Y \| Z, U]}$

On Conditional Correlations

Authors

The Locally Gaussian Partial Correlation

On the Importance of Asymmetry and Monotonicity Constraints in Maximal Correlation Analysis

Correlations with tailored extremal properties

Noncommutative versions of inequalities in quantum information theory

On the correlation between a level of structure order and properties of composites. In Memory of Yu.L. Klimontovich

A Leisurely Look at Versions and Variants of the Cross Validation Estimator

The Derivation of Failure Event Correlation Based on Shadowing Cross-Correlation

This week in AI

1 Introduction

2 Definition

Definition 1.

Definition 2.

Definition 3.

Remark 1.

Remark 2.

Definition 4.

3 Properties

3.1 Basic Properties: Other Characterizations, Continuity, and Concavity

Theorem 1.

Theorem 2.

Remark 3.

Theorem 3.

Remark 4.

Remark 5.

Remark 6.

Proof.

Theorem 4.

Remark 7.

Proof.

Corollary 1.

Proof.

Corollary 2.

Proof.

Theorem 5.

Proof.

Theorem 6.

Proof.

3.2 Other Properties: Tensorization, DPI, Correlation ratio equality, and Conditioning reducing covariance gap

Theorem 7.

Proof.

Lemma 1.

Theorem 8.

Proof.