On the Rényi Cross-Entropy

I Introduction

The Rényi entropy [11] of order $α$

of a discrete distribution (probability mass function)

p

with finite support

S

, defined as

H_{α} (p) = \frac{1}{1 - α} ln \sum x \in S p (x)^{α}

for $α > 0, α \neq 1$ , is a generalization of the Shannon entropy,¹¹1For ease of reference, a table summarising the Shannon entropy and cross-entropy measures as well as the Kullback-Liebler (KL) divergence is provided in Appendix A. $H (p)$ , in that ${lim}_{α \to 1} H_{α} (p) = H (p)$ . Similarly, the Rényi divergence (of order $α$ ) between two discrete distributions $p$ and $q$ with common finite support $S$ , given by

D_{α} (p | | q) = \frac{1}{α - 1} ln \sum x \in S p (x)^{α} q (x)^{1 - α},

reduces to the KL divergence, $D (p ∥ q)$ , as $α \to 1$ .

Since the introduction of these measures, several other Rényi-type information measures have been put forward, each obeying the condition that their limit as $α$ goes to one reduces to a Shannon-type information measure (e.g., see [16] and the references therein for three different order $α$ extensions of Shannon’s mutual information due to Sibson, Arimoto and Csiszár.)

Many of these definitions admit natural counterparts in the (absolutely) continuous case (i.e., when the involved distributions have a probability density function (pdf)), giving rise to information measures such as the Rényi differential entropy for pdf

p

with support

S

h_{α} (p) = \frac{1}{1 - α} ln \int_{S} p (x)^{α} d x,

and the Rényi (differential) divergence between pdfs $p$ and $q$ with common support $S$ ,

D_{α} (p | | q) = \frac{1}{α - 1} ln \int_{S} p (x)^{α} q (x)^{1 - α} d x .

The Rényi cross-entropy between distributions $p$ and $q$ is an analogous generalization of the Shannon cross-entropy $H (p; q)$ . Two definitions for this measure have been suggested. In [12], mirroring the fact that Shannon’s cross-entropy satisfies $H (p; q) = D (p ∥ q) + H (p)$ , the authors define Rényi cross-entropy as

{~ H}_{α} (p; q) := D_{α} (p | | q) + H_{α} (p) .

(1)

In contrast, prior to [12], the authors of [15] introduced the Rényi cross-entropy in their study of the so-called shifted Rényi measures (expressed as the logarithm of weighted generalized power means). Specifically, upon simplifying Definition 6 in [15], their expression for the Rényi cross-entropy between distributions $p$ and $q$ is given by

H_{α} (p; q) := \frac{1}{1 - α} ln \sum x \in S p (x) q {(x)}^{α - 1} .

(2)

For the continuous case, the definition in (2) can be readily converted to yield the Rényi differential cross-entropy between pdfs $p$ and $q$ :

h_{α} (p; q) := \frac{1}{1 - α} ln \int_{S} p (x) q (x)^{α - 1} d x .

(3)

As the Rényi differential divergence and entropy were already calculated for numerous distributions in [5] and [14], respectively, determining the Rényi differential cross-entropy using the definition in (1) is straightforward. As such, this paper’s focus is to establish closed-form expressions of the Rényi differential cross-entropy as defined in (3) for various distributions, as well as to derive the Rényi cross-entropy rate for two important classes of sources with memory, Gaussian and Markov sources.

Motivation for determining formulae for the Rényi cross-entropy extends beyond idle curiosity. The Shannon differential cross-entropy was used as a loss function for the design of deep learning generative adversarial networks (GANs) [6]. Recently, the Rényi differential cross-entropy measures in (3) and (1), were used in [1, 2] and [12], respectively, to generalize the original GAN loss function. It is shown that in [1] and [2] that the resulting Rényi-centric generalized loss function preserves the equilibrium point satisfied by the original GAN based on the Jensen-Rényi divergence [8], a natural extension of the Jensen-Shannon divergence [9]. In [12], a different Rényi-type generalized loss function is obtained and is shown to benefit from stability properties. Improved stability and system performance are shown in [1, 2] and [12] by virtue of the $α$ parameter that can be judiciously used to fine-tune the adopted generalized loss functions which recover the original GAN loss function as $α \to 1$ .

The rest of this paper is organised as follows. In Section II, basic properties of the Rényi cross-entropy are examined. In Section III, the Rényi differential cross-entropy for members of the exponential family is calculated. In Section IV, the Rényi differential cross-entropy between two different distributions is obtained. In Section V, the Rényi differential cross-entropy rate is derived for stationary Gaussian sources. Finally in Section VI, the Rényi cross-entropy rate is established for finite-alphabet time-invariant Markov sources.

Ii Basic Properties of the Rényi cross-entropy and differential cross-entropy

For the Rényi cross-entropy $H_{α} (p; q)$ to deserve its name it would be preferable that it satisfies at least two key properties: it reduces to the Rényi entropy when $p = q$ and its limit as $α$ goes to one is the Shannon cross-entropy. Similarly, it is desirable that the Rényi differential cross-entropy $h_{α} (p; q)$ reduces to the Rényi differential entropy when $p = q$ and its limit as $α$ tends to one yields the Shannon differential cross-entropy. In both cases, the former property is trivial, and the latter property was proven in [2] for the continuous case under some finiteness conditions (in the discrete case, the result holds directly via L’Hôpital’s rule).

It is also proven in [2] that the Rényi differential cross-entropy $h_{α} (p; q)$ is non-increasing in $α$ by showing that its derivative with respect to $α$ is non-positive. The same monotonicity property holds in the disrcrete case.

Like its Shannon counterpart, the Rényi cross-entropy is non-negative ( $H_{α} (p; q) \geq 0$ ); while the Rényi differential cross-entropy can be negative. This is easily verified when, for example, $α = 2$ and $p$ and $q$

are both Gaussian (normal) distributions with zero mean and variance

1 / (8 \sqrt{π})

, and parallels the same lack of non-negativity of the Shannon differential cross-entropy.

We close this section by deriving the cross-entropy limit, ${lim}_{α \to \infty} H_{α} (p; q)$ . To begin with, for any non-zero constant $~ c$ , we have

	$lim α \to \infty \frac{1}{1 - α} ln \sum x \in S ~ c q {(x)}^{α - 1}$
	$= lim α \to \infty \frac{1}{1 - α} ln ~ c + lim α \to \infty \frac{1}{1 - α} ln \sum x \in S q {(x)}^{α - 1}$
	$= lim β \to \infty \frac{1 - β}{- β} \frac{1}{1 - β} ln \sum S q {(x)}^{β} (β = α - 1)$
	$= lim β \to \infty H_{β} (q) = - ln q_{M},$		(4)

where $q_{M} := {max}_{x \in S} q (x)$ and where we have used the fact that for the Rényi entropy, ${lim}_{α \to \infty} H_{α} (q) = - ln q_{M}$ . Now, denoting the minimum and maximum values of $p (x)$ over $S$ by $p_{m}$ and $p_{M}$ , respectively, we have that for $α > 1$ ,

	$\frac{1}{1 - α} ln \sum x \in S p_{m} q {(x)}^{α - 1}$	$\leq \frac{1}{1 - α} ln \sum x \in S p (x) q {(x)}^{α - 1}$
	and
	$\frac{1}{1 - α} ln \sum x \in S p (x) q {(x)}^{α - 1}$

and hence by (4) we obtain

lim α \to \infty H_{α} (p; q) = - ln q_{M} .

(5)

Iii Rényi Differential Cross-Entropy for Exponential Family Distributions

A probability distribution on

R

R^{n}

with parameter

θ

is said to belong to the exponential family (e.g., see [3]) if on its support

S

it admits a pdf of the form

f (x) = c (θ) b (x) exp (η (θ) \cdot T (x)), x \in S,

(6)

for some real-valued (measurable) functions $c$ , $b$ , $η$ and $T$ .²²2Note that $θ$ and consequently $T (x)$

can be vectors in cases where the distribution admits multiple parameters.

Here

η

is known as the natural parameter of the distribution,

T (x)

is the sufficient statistic and

c (θ)

is the normalization constant in the sense that for all

θ

within the parameter space

\int_{S} b (x) exp (η (θ) \cdot T (x)) d x = c (θ)^{- 1} .

The pdf in (6) can also be written as

f (x) = b (x) exp (η \cdot T (x) + A (η)),

(7)

where $A (η (θ)) = ln c (θ)$

. Examples of distributions in the exponential family include the Gaussian, Beta, and exponential distributions.

Lemma 1.

Let $f_{1} (x)$ and $f_{2} (x)$ be pdfs of the same type in the exponential family with natural parameters $η_{1}$ and $η_{2}$ , respectively. Define $f_{h} (x)$ as being of the same type as $f_{1}$ and $f_{2}$ but with natural parameter $η_{h} = η_{1} + (α - 1) η_{2}$ . Then

h_{α} (f_{1}; f_{2}) = \frac{A (η_{1}) - A (η_{h}) + ln E_{h}}{1 - α} - A (η_{2}),

(8)

where $E_{h} = E_{f_{h}} [b (X)^{α - 1}] = \int b (x)^{α - 1} f_{h} (x) d x$

Proof.

Using (7), we have

	$f_{1}$	$(x) f_{2} {(x)}^{α - 1}$
		$= b (x) exp (η_{1} \cdot T (x) + A (η_{1}))$
		$\cdot (b (x) exp (η_{2} \cdot T (x) + A (η_{2})))^{α - 1}$
		$= b {(x)}^{α} exp ((η_{1} + (α - 1) η_{2}) \cdot T (x))$
		$\cdot exp (A (η_{1}) + (α - 1) A (η_{2}))$
		$= b {(x)}^{α} exp (η_{h} \cdot T (x) + A (η_{h}))$
		$\cdot exp (A (η_{1}) + (α - 1) A (η_{2}) - A (η_{h}))$
		$= b {(x)}^{α - 1} f_{h} (x) exp (A (η_{1}) + (α - 1) A (η_{2}) - A (η_{h})) .$

Thus,

	$\int_{S}$	$f_{1} (x) f_{2} {(x)}^{α - 1} d x$
		$= \int_{S} b {(x)}^{α - 1} f_{h} (x) d x$
		$\cdot exp (A (η_{1}) + (α - 1) A (η_{2}) - A (η_{h}))$
		$= exp (A (η_{1}) + (α - 1) A (η_{2}) - A (η_{h})) E_{h},$

and therefore,

h_{α} (f_{1}; f_{2}) = \frac{A (η_{1}) - A (η_{h}) + ln E_{h}}{1 - α} - A (η_{2}) .

∎

Remark.

If $b (x) = b$ is a constant for all $x \in S$ , then

\frac{ln E_{h}}{1 - α} = - ln b .

In many cases, we have that $b (x) = 1$ on $S$ , and thus the $\frac{ln E_{h}}{1 - α}$ term disappears in (8).

Table I lists Rényi differential cross-entropy expressions we derived using Lemma 1 for some common distributions in the exponential family (which we describe in Appendix B for convenience). In the table, the subscript of $i$ is used to denote that a parameter belongs to pdf $f_{i}$ , $i = 1, 2$ .

Name	$h_{α} (f_{1}; f_{2})$
Beta	$ln B (a_{2}, b_{2}) + \frac{1}{α - 1} ln \frac{B (a_{h}, b_{h})}{B (a_{1}, b_{1})}$
	$a_{h} := a_{1} + (α - 1) (a_{2} - 1)$ ,
	$b_{h} := b_{1} + (α - 1) (b_{2} - 1)$
$χ^{2}$
	$+ \frac{2 - ν_{2}}{2} ln (α) + ln 2 Γ (\frac{ν_{2}}{2})$
	$ν_{h} := ν_{1} + (α - 1) (k - 2)$
Exponential	$\frac{1}{1 - α} ln \frac{λ_{i}}{λ_{h}} - ln λ_{2}$
	$λ_{h} := λ_{1} + (α - 1) λ_{2}$
Gamma	$ln Γ (k_{2}) + k_{2} ln θ_{2}$
	$+ \frac{1}{1 - α} (ln \frac{Γ (k_{h})}{Γ (k_{1})} - k_{h} ln θ_{h} - k_{1} ln θ_{1})$
	$θ_{α}^{*} := \frac{θ_{1} + (a - 1) θ_{2}}{(α - 1) θ_{1} θ_{1}}$ , $k_{h} := k_{i} + (α - 1) k_{2}$
Gaussian	$\frac{1}{2} (ln (2 π σ_{2}^{2}) + \frac{1}{1 - α} ln (\frac{σ_{2}^{2}}{(σ^{2})_{h}^{}}) + \frac{(μ_{1} - μ_{2}^{2}}{(σ^{2})_{h}^{}})$
	$(σ^{2})_{h}^{*} := σ_{2}^{2} + (α - 1) σ_{1}^{2}$
$\begin{matrix} Laplace \end{matrix}$	$ln (2 b_{2}) + \frac{1}{1 - α} ln (\frac{b_{2}}{2 b_{h}})$
( $μ_{1} = μ_{2}$ )	$b_{h} := b_{2} + (1 - α) b_{1}$

TABLE I: Rényi Differential Cross-Entropies for Common Continuous Distributions

Iv Rényi differential Cross-Entropy between different distributions

Let $p$ and $q$ be pdfs with common support $S \subseteq R$ . Below are some general formulae for the differential Rényi cross-entropy between one specific (common) distribution and any general distribution. If $S$ is an interval below, then $| S |$ denotes its length.

Iv-a Distribution $q$ is uniform

Let $q$

be uniformly distributed on

S

. Then

h_{α} (p; q)

= \frac{1}{1 - α} ln \int_{S} p (x) q (x)^{α - 1} d x = ln | S | .

Iv-B Distribution $p$ is uniform

Now suppose $p$ is uniformly distributed on $S$ . Then

	$h_{α} (p; q)$	$= \frac{1}{1 - α} ln \int_{S} p (x) q (x)^{α - 1} d x$
		$= \frac{1}{1 - α} ln \frac{1}{\| S \|} - h_{α - 1} (q) .$

Iv-C Distribution $q$ is exponentially distributed

Suppose the $S = R^{+}$ and $q$ is exponential with parameter $λ$

. Suppose also that the moment generating function (MGF) of

p

M_{p} (t)

exists. We have

	$h_{α} (p; q)$	$= \frac{1}{1 - α} ln \int_{S} p (x) q (x)^{α - 1} d x$
		$= \frac{1}{1 - α} ln E_{p} [q (x)^{α - 1}]$
		$= \frac{1}{1 - α} ln E_{p} [{(λ exp (- λ x))}^{α - 1}]$
		$= - ln λ + \frac{1}{1 - α} ln M_{p} (λ (1 - α)) .$

Iv-D Distribution $q$ is Gaussian

Now assume that $q$ is a (normal) Gaussian $N (μ, σ^{2})$ distribution and that the MGF of $Y := (X - μ)^{2}$ , $M_{Y}$ , exists, where $X$

is a random variable with distribution

p

. Then

	$h_{α} (p; q)$	$= \frac{1}{1 - α} ln E_{p} [q (X)^{α - 1}]$
		$= \frac{1}{1 - α} ln σ (\sqrt{2 π})^{1 - α} E (exp ((1 - α) \frac{Y}{2 σ^{2}}))$
		$= ln σ \sqrt{2 π} + \frac{1}{1 - α} ln M_{Y} (\frac{1 - α}{2 σ^{2}}) .$

The case where $q$ is a half-normal distribution can be directly derived from the above. Given $q$ is a half-normal distribution, on its support its pdf is the same as that of a normal $N (0, σ^{2})$ distribution times 2. Hence if $p$ ’s support is $R^{+}$ , then .

V Rényi Differential Cross-Entropy Rate for Stationary Gaussian Processes

Lemma 2.

The Rényi differential cross-entropy between two zero-mean multivariate dimension- $n$ Gaussian distributions with invertible covariance matrices $Σ_{1}$ and $Σ_{2}$ , respectively, is given by

h_{α} (p; q) = \frac{ln | Σ_{1} | | S |}{2 α - 2} + \frac{1}{2} ln | Σ_{2} | + \frac{n}{2} ln 2 π,

(9)

where $S := Σ_{1}^{- 1} + (α - 1) Σ_{2}^{- 1}$ .

Proof.

Recall that the pdf of a multivariate Gaussian with mean $0 = (0, 0..., 0)^{T}$ and invertible covariance matrix $Σ$ is given by:

f (x) = \frac{exp (\frac{- 1}{2} x^{T} Σ^{- 1} x)}{(2 π)^{k / 2} | Σ |^{1 / 2}}

for $x \in R^{n}$ . Note that this distribution is a member of the exponential family, where $T (x) = x$ , $η = \frac{1}{2} Σ^{- 1}$ , $A (η) = \frac{1}{2} ln | - 2 η |$ and $b (x) = (2 π)^{\frac{- n}{2}}$ . Hence the Rényi differential cross-entropy between two zero-mean multivariate Gaussian distributions with covariance matrices $Σ_{1}$ and $Σ_{2}$ , respectively, is

	$h_{α} (p; q)$	$= \frac{1}{1 - α} (\frac{1}{2} ln ∣ ∣ ∣ ∣ 2 \frac{Σ_{1}^{- 1}}{2} ∣ ∣ ∣ ∣$
		$- \frac{1}{2} ln ∣ ∣ ∣ ∣ 2 \frac{Σ_{1}^{- 1} + (α - 1) Σ_{2}^{- 1}}{2} ∣ ∣ ∣ ∣)$
		$- \frac{1}{2} ln ∣ ∣ ∣ ∣ 2 \frac{Σ_{2}^{- 1}}{2} ∣ ∣ ∣ ∣ - ln (2 π)^{\frac{- n}{2}}$
		$= \frac{ln \| Σ_{1} \| \| S \|}{2 α - 2} + \frac{1}{2} ln \| Σ_{2} \| + \frac{n}{2} ln 2 π .$

∎

Let ${X_{j}}_{j = 1}^{\infty}$ and ${Y_{j}}_{j = 1}^{\infty}$ be stationary zero-mean Gaussian processes. For a given $n$ , $X^{n} := (X_{1}, X_{2}, . . ., X_{n})$ and $Y^{n} := (Y_{1}, Y_{2}, . . ., Y_{n})$ are multivariate Gaussian random variables with mean 0 and covariance matrices $Σ_{X^{n}}$ and $Σ_{Y^{n}}$ , respectively. Since ${X_{j}}$ and ${Y_{j}}$ are stationary, their covariance matrices are Toeplitz. Furthermore, $B^{n} := Σ_{Y^{n}} + (α - 1) Σ_{X^{n}}$ is Toeplitz.

Lemma 3.

Let $~ f (λ)$ , $~ g (λ)$ and $~ h (λ)$ be the power spectral densities of ${X_{j}}$ , ${Y_{j}}$ and the zero-mean Gaussian process with covariance matrix $B^{n}$ , respectively.

Then the Rényi differential cross-entropy rate between ${X_{j}}$ and ${Y_{j}}$ , ${lim}_{n \to \infty} \frac{1}{n} h_{α} (X^{n}; Y^{n})$ , is given by

\frac{ln 2 π}{2} + \frac{1}{4 π (1 - α)} \int_{0}^{2 π} [(2 - α) ln ~ g (λ) - ln ~ h (λ)] d λ .

Proof.

From Lemma 2, we first note that $S = Σ_{X^{n}}^{- 1} B_{n} Σ_{Y^{n}}^{- 1}$ . With this in mind the Rényi differential cross-entropy can be rewritten using (9) as

	$\frac{1}{n} (\frac{ln \| Σ_{X^{n}} \| \| Σ_{X^{n}}^{- 1} B^{n} Σ_{Y^{n}}^{- 1} \|}{2 (α - 1)} + \frac{1}{2} ln \| Σ_{Y^{n}} \| + \frac{n}{} 2 ln 2 π)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n} (\frac{ln \| Σ_{X^{n}} \| \| Σ_{X^{n}}^{- 1} \| \| B^{n} \| \| Σ_{Y^{n}}^{- 1} \|}{(α - 1)} + ln \| Σ_{Y^{n}} \|)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n} (\frac{ln \| B^{n} \| - ln \| Σ_{Y^{n}} \|}{(α - 1)} + ln \| Σ_{Y_{n}} \|)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n (1 - α)} ((2 - α) ln \| Σ_{Y^{n}} \| - ln \| B^{n} \|) .$

It was proven in [7] that for a sequence of Toeplitz matrices $T_{n}$ with spectral density $t (λ)$ such that $ln t (λ)$ is Reimann integrable, one has

lim n \to \infty ln | T^{n} | = \frac{1}{2 π} \int_{0}^{2 π} ln t (λ) d λ .

We therefore obtain that the Rényi differential cross-entropy rate is given by

\frac{ln 2 π}{2} + \frac{1}{4 π (1 - α)} \int_{0}^{2 π} [(2 - α) ln ~ g (λ) - ln ~ h (λ)] d λ .

Note that $~ h (λ) = ~ g (λ) + (α - 1) ~ f (λ)$ . ∎

Vi Rényi cross-entropy rate for Markov sources

Consider two time-invariant Markov sources ${X_{j}}_{j = 1}^{\infty}$ and ${Y_{j}}_{j = 1}^{\infty}$ with common finite alphabet $S$ and with transition distribution $P (\cdot | \cdot)$ and $Q (\cdot | \cdot)$ , respectively. Then for any $i^{n} = (i_{1}, \dots, i_{n}) \in S^{n}$ , their $n$

-dimensional joint distributions are given by

p^{(n)} (i^{n}) = P (i_{n} | i_{n - 1}) P (i_{n - 1} | i_{n - 2}) . . . P (i_{2} | i_{1}) q (i_{1})

and

q^{(n)} (i^{n}) = Q (i_{n} | i_{n - 1}) Q (i_{n - 1} | i_{n - 2}) . . . Q (i_{2} | i_{1}) p (i_{1}),

respectively, with arbitrary initial distributions, $p (i_{1})$ and $q (i_{1})$ , $i_{1} \in S$ . Define the Rényi cross-entropy rate between ${X_{j}}$ and ${Y_{j}}$ as

		$lim n \to \infty \frac{1}{n} H_{α} (X^{n}; Y^{n})$
	$=$	$lim n \to \infty \frac{1}{n} \frac{1}{1 - α} ln ⎛ ⎝ \sum i_{n} \in S^{n} p^{(n)} (i^{n}) q^{(n)} (i_{n})^{α - 1} ⎞ ⎠ .$

Note that by defining the matrix $R$ using the formula

R_{i j} = P (i | j) Q (i | j)^{α - 1}

and the row vector s as having components $s_{i} = P (i) P (i)^{α - 1}$ , the Rényi cross-entropy rate can be written as

lim n \to \infty \frac{1}{n} \frac{1}{1 - α} ln s R^{n - 1} 1,

(10)

where 1 is a column vector whose dimension is the cardinatliy of the alphabet $S$ and with all its entries equal to 1.

A result derived by [10] for the Rényi divergence between Markov sources can thus be used to find the Rényi cross-entropy rate for Markov sources.

Lemma 4.

Let $P$ , $Q$ , s and R be defined as above. If $R$ is irreducible, then

lim n \to \infty \frac{1}{n} H_{α} (X^{n}; Y^{n}) = \frac{ln λ}{1 - α},

(11)

where $λ$

is the largest positive eigenvalue of

R

Proof.

Since the non-negative matrix $R$ is irreducible, by the Frobenius theorem (e.g., cf. [13, 4]), it has a largest positive eigenvalue $λ$

with associated positive eigenvector

b. Let

b_{m}

and

b_{M}

be the minimum and maximum elements, respectively, of b. Then due to the non-negativity of s,

λ^{n - 1} s \cdot b = s R^{n - 1} b \leq % s R^{n - 1} 1 b_{M},

where $\cdot$ denotes the Euclidean inner product. Similarly, $λ^{n - 1} s \cdot b \geq s R^{n - 1} 1 b_{m} .$ As a result,

\frac{1}{n} ln \frac{λ^{n - 1} s \cdot b}{b_{M}} \leq \frac{1}{} n ln s R^{n - 1} 1 \leq \frac{1}{n} ln \frac{λ^{n - 1} % s \cdot b}{b_{m}} .

Note that for all $n$ , $\frac{s \cdot b}{b_{M}}$ is a constant. Thus

	$lim n \to \infty \frac{1}{n} ln \frac{λ^{n - 1} s \cdot b}{b_{M}}$	$= lim n \to \infty \frac{n - 1}{n} ln λ + lim n \to \infty \frac{1}{n} ln \frac{s \cdot b}{b_{M}}$
		$= ln λ .$

Similarly, we have

lim n \to \infty \frac{1}{n} ln \frac{λ^{n - 1} s \cdot b}{b_{m}} = ln λ .

Hence,

lim n \to \infty \frac{1}{n} H_{α} (X^{n}; Y^{n}) = lim n \to \infty \frac{1}{n} ln \frac{λ^{n - 1} s \cdot b}{(1 - α) b_{m}} = \frac{ln λ}{1 - α} .

∎

Another technique can be borrowed from [10] to generalize Lemma 4 to the case where $R$ is reducible. First $R$ is rewritten in the canonical form detailed in Proposition 1 of [10]. Let $λ_{k}$ be the largest positive eigenvalue of each self-communicating sub-matrix of $R$ . For each inessential class $C_{i}$ let $λ_{j}$ be the largest positive eigenvalue of each class that is reachable from $C_{i}$ . Define $λ = max {λ_{k}, λ_{j}}$ . Then (11) holds.

Appendix A: Shannon-type information measures

Name	Definition
$\begin{matrix} Shannon Entropy \end{matrix}$	$H (p) = - \sum x \in S p (x) ln p (x)$
$\begin{matrix} Shannon Differential Entropy \end{matrix}$	$h (p) = - \int_{S} p (x) ln p (x) d x$
$\begin{matrix} Shannon Cross-Entropy \end{matrix}$	$H (p; q) = - \sum x \in S p (x) ln q (x)$
$\begin{matrix} Shannon Differential Cross-Entropy \end{matrix}$	$h (p; q) = - \int_{S} p (x) ln q (x) d x$
$\begin{matrix} KL Divergence, (Discrete) \end{matrix}$	$D (p ∥ q) = - \sum x \in S p (x) ln \frac{p (x)}{q (x)}$
$\begin{matrix} KL Divergence, (Continuous) \end{matrix}$	$D (p ∥ q) = - \int_{S} p (x) ln \frac{p (x)}{q (x)} d x$

Appendix B: Distributions listed in Table I

Name	PDF $f (x)$
(Parameters)	(Support)
Beta	$B (a, b) x^{a - 1} (1 - x)^{b - 1}$
( $a > 0$ , $b > 0$ )	$S = (0, 1)$
$χ^{2}$	$\frac{1}{2^{\frac{ν}{2}} Γ (\frac{ν}{2})} x^{\frac{ν}{2} - 1} e^{- \frac{x}{2}}$
( $ν \in Z^{+}$ )	$S = R^{+}$
Exponential	$λ e^{- λ x}$
( $λ > 0$ )	$S = R^{+}$
Gamma	$\frac{1}{θ^{k} Γ (k)} x^{k - 1} e^{- \frac{k}{θ}}$
( $k > 0$ , $θ > 0$ )	$S = R^{+}$
Gaussian	$\frac{1}{\sqrt{2 π σ^{2}}} e^{- \frac{1}{2} {(\frac{x - μ}{} σ)}^{2}}$
( $μ$ , $σ^{2} > 0$ )	$S = R$
Laplace	$\frac{1}{2 b} e^{- \frac{\| x - μ \|}{b}}$
( $μ$ , $b^{2} > 0$ )	$S = R$

Notes

$B (a, b) = \int_{0}^{1} t^{a - 1} (1 - t)^{b - 1} d t$ is the Beta function.
$Γ (z) = \int_{0}^{\infty} x^{z - 1} e^{- x} d x$ is the Gamma function.

References

[1] H. Bhatia, W. Paul, F. Alajaji, B. Gharesifard, and P. Burlina (2020) Rényi Generative Adversarial Networks. ArXiv:2006.02479v1. Cited by: §I.
[2] H. Bhatia, W. Paul, F. Alajaji, B. Gharesifard, and P. Burlina (2021-08) Least kth-order and Rényi generative adversarial networks. Neural Computation 33 (9), pp. 2473–2510. External Links: ISSN 1530-888X, Link, Document Cited by: §I, §II, §II.
[3] G. Casella and R. L. Berger (2002) Statistical inference. Cengage Learning. Cited by: §III.
[4] R. G. Gallager (1996) Discrete stochastic processes. Springer. Cited by: §VI.
[5] M. Gil, F. Alajaji, and T. Linder (2013) Rényi divergence measures for commonly used univariate continuous distributions. Information Sciences 249, pp. 124–131. External Links: ISSN 0020-0255, Document, Link Cited by: §I.
[6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014) Generative adversarial nets. In Proceedings of Advances in Neural Information Processing Systems, Vol. 27, pp. 2672–2680. Cited by: §I.
[7] R. Gray (2001-10,) Toeplitz and circulant matrices: a review. Foundations and Trends® in Communications and Information Theory 2, pp. . External Links: Document Cited by: §V.
[8] P. A. Kluza (2019) On Jensen-Rényi and Jeffreys-Rényi type $f$ -divergences induced by convex functions. Physica A: Statistical Mechanics and its Applications. External Links: Document Cited by: §I.
[9] J. Lin (1991) Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 31, pp. 145–151. Cited by: §I.
[10] Z. Rached, F. Alajaji, and L. L. Campbell (2001) Rényi’s divergence and entropy rates for finite alphabet Markov sources. IEEE Transactions on Information theory 47 (4), pp. 1553–1561. Cited by: §VI, §VI.
[11] A. Rényi (1961) On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, pp. 547–561. Cited by: §I.
[12] A. Sarraf and Y. Nie (2021) RGAN: rényi generative adversarial network. SN Computer Science 2 (1), pp. 17. External Links: Document Cited by: §I, §I.
[13] E. Seneta (2006)
Non-negative matrices and markov chains
. Springer Science & Business Media. Cited by: §VI.
[14] K. Song (2001-02) Rényi information, loglikelihood and an intrinsic distribution measure. J. Statistical Planning and Inference 93. External Links: Document Cited by: §I.
[15] F. J. Valverde-Albacete and C. Peláez-Moreno (2019) The case for shifting the Rényi entropy. Entropy 21, pp. 1–21. External Links: Link Cited by: §I.
[16] S. Verdú (2015) $α$ -Mutual information. In Proceedings of the IEEE Information Theory and Applications Workshop, pp. 1–6. Cited by: §I.

	$\frac{1}{n} (\frac{ln \| Σ_{X^{n}} \| \| Σ_{X^{n}}^{- 1} B^{n} Σ_{Y^{n}}^{- 1} \|}{2 (α - 1)} + \frac{1}{2} ln \| Σ_{Y^{n}} \| + \frac{n}{} 2 ln 2 π)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n} (\frac{ln \| Σ_{X^{n}} \| \| Σ_{X^{n}}^{- 1} \| \| B^{n} \| \| Σ_{Y^{n}}^{- 1} \|}{(α - 1)} + ln \| Σ_{Y^{n}} \|)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n} (\frac{ln \| B^{n} \| - ln \| Σ_{Y^{n}} \|}{(α - 1)} + ln \| Σ_{Y_{n}} \|)$
	$= \frac{ln 2 π}{2} + \frac{1}{2 n (1 - α)} ((2 - α) ln \| Σ_{Y^{n}} \| - ln \| B^{n} \|) .$

On the Rényi Cross-Entropy

Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory

A Quantitative Comparison between Shannon and Tsallis Havrda Charvat Entropies Applied to Cancer Outcome Prediction

A Dual Process Model for Optimizing Cross Entropy in Neural Networks

Information Bottleneck Methods for Distributed Learning

Temperature check: theory and practice for training models with softmax-cross-entropy losses

Set Cross Entropy: Likelihood-based Permutation Invariant Loss Function for Probability Distributions

Probability Update: Conditioning vs. Cross-Entropy

I Introduction

Ii Basic Properties of the Rényi cross-entropy and differential cross-entropy

Iii Rényi Differential Cross-Entropy for Exponential Family Distributions

Lemma 1.

Proof.

Remark.

Iv Rényi differential Cross-Entropy between different distributions

Iv-a Distribution $q$ is uniform

Iv-B Distribution $p$ is uniform

Iv-C Distribution $q$ is exponentially distributed

Iv-D Distribution $q$ is Gaussian

V Rényi Differential Cross-Entropy Rate for Stationary Gaussian Processes

Lemma 2.

Proof.

Lemma 3.

Proof.

Vi Rényi cross-entropy rate for Markov sources

Lemma 4.

Proof.

Appendix A: Shannon-type information measures

Appendix B: Distributions listed in Table I

References

On the Rényi Cross-Entropy

Related Research

Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory

A Quantitative Comparison between Shannon and Tsallis Havrda Charvat Entropies Applied to Cancer Outcome Prediction

A Dual Process Model for Optimizing Cross Entropy in Neural Networks

Information Bottleneck Methods for Distributed Learning

Temperature check: theory and practice for training models with softmax-cross-entropy losses

Set Cross Entropy: Likelihood-based Permutation Invariant Loss Function for Probability Distributions

Probability Update: Conditioning vs. Cross-Entropy

I Introduction

Ii Basic Properties of the Rényi cross-entropy and differential cross-entropy

Iii Rényi Differential Cross-Entropy for Exponential Family Distributions

Lemma 1.

Proof.

Remark.

Iv Rényi differential Cross-Entropy between different distributions

Iv-a Distribution q is uniform

Iv-B Distribution p is uniform

Iv-C Distribution q is exponentially distributed

Iv-D Distribution q is Gaussian

V Rényi Differential Cross-Entropy Rate for Stationary Gaussian Processes

Lemma 2.

Proof.

Lemma 3.

Proof.

Vi Rényi cross-entropy rate for Markov sources

Lemma 4.

Proof.

Appendix A: Shannon-type information measures

Appendix B: Distributions listed in Table I

References

Iv-a Distribution $q$ is uniform

Iv-B Distribution $p$ is uniform

Iv-C Distribution $q$ is exponentially distributed

Iv-D Distribution $q$ is Gaussian