Objective Bayesian Analysis for the Differential Entropy of the Gamma Distribution

1 Introduction

In recent years, there has been a growing interest in estimating different metrics of information theory related to parametric distributions. The Shannon entropy, also known as differential entropy, introduced by Claude Shannon [30], is an essential quantity that measures the amount of available information or uncertainty outcome of a random process. Given a density function $f (x | α, β)$ , the differential entropy is given by

H (α, β) = E (- log f (x | α, β)) .

(2)

The differential entropy depends on the distribution parameters, and, given a sample, it is necessary to be estimated. The commonly used method to estimate the parameters is the maximum likelihood approach due to its one-to-one invariance property. Hence, we need only to estimate the parameters of the original model and plug-in the entropy function. Under this approach many authors have derived the estimators for different distributions such as, Weibull [9], Inverse Weibull [33], Log-logistic [13]

and for the exponential distribution with different shift origin

[20], to list a few.

A major drawback of the maximum likelihood inference is that the obtained estimates are usually biased for small samples [11]

. Another concern happens under small samples when constructing the confidence intervals for the parameters since such intervals are not precise and may not return good coverage probabilities. In this case, the maximum likelihood estimation (MLE) skewness study is essential to assess the quality of the interval

[12]. To overcome these limitations, we can use objective Bayesian methods. In this context, the inference for the parameters of the gamma distribution have been discussed earlier under this approach by Miller [23], Sun and Ye [31], Berger et al. [3], and Louzada and Ramos [21]. Moreover, Ramos et al. [27]

revised the most common objective priors and provided sufficient and necessary conditions for the obtained posteriors and their higher moments to be proper.

Although the authors have obtained different joint posterior distributions for the parameters of interest, the obtained posterior means can not be directly plunged in the Shannon entropy. Under the Bayesian approach, it is necessary to obtain the posterior distribution of the entropy measure. In this context, Shakhatreh [29] recently derived different posterior distributions using objective priors for the entropy assuming a Weibull distribution. On the other hand, the cited distribution’s entropy expression is not as complicated as the gamma distribution’s entropy expression. With this in mind, in this paper, focusing on the gamma distribution, we derive the posterior distributions using objective priors, such as Jeffreys prior [18], reference priors [7, 2, 3], and matching priors [32]

, and prove that the obtained posteriors are proper and can be used to construct the posterior distributions of the Shannon entropy. Moreover, even if the posterior distribution is proper, the posterior mean can be infinite, which is undesirable, and thus we shall also prove that the obtained posterior means for the entropy measure are finite. Finally, the credibility intervals are obtained to construct accurate interval estimates.

The gamma distribution considered here is a two-parameter family of distribution among the most well-known distribution used to model different stochastic processes and to make statistical inferences, and has received attention from different fields. It surfaces in many areas of applications, including financial analysis [10], climate analysis [17], reliability analysis [16]

, machine learning

[19], and physics [14]

. Particularly, the gamma distribution includes the exponential distribution, Erlang distribution, and chi-square distribution as special cases.

A random variable

X

follows a gamma distribution, if its probability density function, parametrized by a shape parameter

α > 0

and scale parameter

β > 0

, is given by,

f (x | α, β) = \frac{β^{α}}{Γ (α)} x^{α - 1} e^{- β x}, x > 0,

(3)

where $Γ (ϕ) = \int_{0}^{\infty} e^{- x} x^{ϕ - 1} d x$ is the gamma function.

The paper is organized as follows. Section 2 presents the maximum likelihood estimators for the gamma distribution parameters and the Shannon Entropy computation. Section 3 presents the objective Bayesian analysis using objective priors for the Shannon entropy parameter’s reparametrized posterior distribution. Section 4 provides a simulation study to select the best objective prior. In Section 5, the methodology is illustrated on a real dataset. Some final comments are given in Section 6.

2 Frequentist approach

The classical inference (frequentist) is a commonly used approach to conduct parameter estimation of a particular distribution. In this case, the parameter is treated as fixed, and the MLE is commonly used to obtain the estimates. The MLE has good asymptotic properties, such as invariance, consistency, and efficiency. This procedure search the parameter space of $θ$ where the maximum likelihood $^θ = {sup}_{θ} L (θ | x)$ is obtained. Here our main aim is to obtain the estimate of a function of the parameters. Hence, firstly we need to obtain the entropy measure, mathematically defined as $H (θ) = E (- l o g f (x | θ))$ , which quantifies the amount of uncertainty in the data $x$ . Besides, it should be noted that a higher realization of $H$ indicates more uncertainty.

The entropy $H$ of the gamma density is given by

	$H (α, β)$	$= - \int_{0}^{\infty} log (\frac{β^{α}}{Γ (α)} x^{α - 1} e x p {- β x}) f (x \| α, β) d x$		(4)
		$= α - log (β) + log (Γ (α)) + (1 - α) ψ (α),$		(4)

where $ψ (k) = \frac{\partial}{\partial k} log Γ (x)$ is the digamma function.

Now, consider a change of variable by setting $W = α$ , which implies $H = W - log (β) + log Γ (W) + (1 - W) ψ (W)$ . The aim of the transformation is to obtain a likelihood of $H$ and $W$ instead of $α$ and $β$ . Therefore, if $X_{1}$ , $\dots$ , $X_{n}$ are a complete sample from (3) then the likelihood function of $H$ and $W$ is given as

L (W, H | x) = \frac{δ (W, H)^{n W}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} exp {- δ (W, H) n \sum i = 1 x_{i}},

(5)

where $δ (W, H) = exp (W + log Γ (W) + (1 - W) ψ (W) - H)$ .

The log-likelihood function is given by

l (W, H | x) = W log (δ (W, H)) - log Γ (W) + W n \sum i = 1 log (x_{i}) - δ (W, H) n \sum i = 1 x_{i} .

(6)

The MLEs for the parameters are obtained by directly maximizing the log-likelihood function $ℓ (λ, ϕ; t)$ . Hence, after some algebraic manipulations the MLEs $^W$ and $^H$ are obtained from the solution of

\frac{\partial l (W, H | x)}{\partial W} = log (δ (W, H)) - ψ (W) - n \sum i = 1 log (x_{i}) + σ (W - δ (W, H) n \sum i = 1 x_{i})

\frac{\partial l (W, H | x)}{\partial H} = - W + δ (W, H) n \sum i = 1 x_{i}

where $σ = 1 + (1 - W) ψ^{'} (W)$ . The solutions for these equations provide the maximum likelihood estimators for the entropy of the gamma distributions, $ˆ H$ and $ˆ W$ . Since equation (3) cannot be solved easily using a closed-form solution, numerical techniques must estimate the true parameters.

Following [22]

, the MLEs are asymptotically normally distributed with a joint bivariate normal distribution given by

({^W}_{M L E}, {^H}_{M L E}) \sim N_{2} [(W, H), I^{- 1} (W, H)] as n \to \infty,

where $I (W, H)$ is the Fisher information matrix for the reparametrized model given by

(7)

and $ψ^{'}$ ( $W$ ) is the derivative of $ψ$ ( $W$ ), called the trigamma function.

In the present paper, we are only interested in $H$ , and thus, given $0 < a < 1$ and using the element $(I (W, H)^{- 1})_{22}$ , we can conclude that the confidence interval for the estimate of the entropy measure with a confidence level of $100 (1 - a) %$ for $H$ is given by

^H - Z_{\frac{a}{2}} \sqrt{(1 - W)^{2} ψ^{'} (W) + 2 - W} < H <^H + Z_{\frac{a}{2}} \sqrt{(1 - W)^{2} ψ^{'} (W) + 2 - W},

(8)

where $a$ is the significance level and $Z_{\frac{a}{2}}$ is the $\frac{a}{2}$ -th percentile of the standard normal distribution.

3 Bayesian Inference

Here, the parameter $θ$ is considered as a random variable and the distribution that represents knowledge about $θ$ is refereed as a prior distribution and defined by $π (θ)$ . The distribution $π (θ)$ provides the knowledge or uncertainty about $θ$ before obtaining the sample data $x$ . After the data $x$

is observed, a natural way of combining the resulting information from the a priori the distribution and the likelihood function is done by the Bayes’ theorem, resulting in the posterior distribution of

θ

given

x

. In a Bayesian framework, Ramos

e t a l .

[27] analyzed the properties of the posterior distribution of the gamma distribution parameters and stated the conditions for this distribution to have proper posterior and finite moments.

To obtain the posterior distributions for the $H$ parameter, we can consider the one-to-one invariance property of the Jeffreys prior, reference prior, and matching prior, and thus we only need to obtain the Jacobian matrix related to the reparametrization from $α$ and $β$ to $H$ and $W$ . After some algebraic manipulations, we can conclude that the parameters $β$ and $α$ can be written as

β = exp (W + log (Γ (W)) + (1 - W) ψ (W) - H) and α = W,

and thus, from the relations

\frac{\partial α}{\partial H} = 0, \frac{\partial α}{\partial W} = 1, \frac{\partial β}{\partial H} = - β and \frac{\partial β}{\partial W} = (1 + (1 - W) ψ^{(1)} (W)) β,

it follows that the Jacobian matriz (J) relative to the change of variable will be given by

J = ⎡ ⎢ ⎣ \begin{matrix} \frac{\partial α}{\partial H} & \frac{\partial α}{\partial W} \frac{\partial β}{\partial H} & \frac{\partial β}{\partial W} \end{matrix} ⎤ ⎥ ⎦ = [\begin{matrix} 0 & 1 - β & σ β \end{matrix}],

(9)

where $σ = 1 + (1 - W) ψ^{'} (W)$ .

The use of objective priors plays an essential role in Bayesian analysis where the data provide the dominant information, and the posterior distribution is not overshadowed by prior information. Such priors allow us to conduct objective Bayesian inference. On the other hand, in most situations, they are not proper prior distributions and may lead to improper posterior, invalidating the analysis since we cannot compute the normalizing constant. Therefore, we need to check if the obtained posterior (and posterior mean) is proper (or finite). The priors for the entropy and its related posterior distributions will be discussed in the next subsections.

Before we derive the priors and posterior distributions, hereafter, we shall always assume that there are at least two distinct data $t_{i}$ , that is, there exists $1 \leq i < j \leq n$ such that $t_{i} \neq t_{j}$ . Additionally, before we proceed, we present below a definition and proposition that will be used to prove that the obtained posteriors are proper. In the following let $¯ ¯¯ ¯ R = R \cup {- \infty, \infty}$ denote the extended real number line and let $R^{+}$ denote the strictly positive real numbers. The following definition is a special case from the one presented in [25] and will play an important role in proving that the analyzed posterior distributions and posterior means are proper.

Definition 3.1.

Let $a \in ¯ ¯¯ ¯ R$ , $g : U \to R^{+}$ and $h : U \to R^{+}$ , where $U \subset R$ and suppose that ${lim}_{x \to a} \frac{g (x)}{h (x)} = c \in R$ . Then, if $c > 0$ , we say that $g (x) \propto x \to a h (x)$ .

Regarding the above definition, we have the following proposition from [25].

Proposition 3.2.

Let $g : (a, b) \to R^{+}$ and $h : (a, b) \to R^{+}$ be continuous functions in $(a, b) \subset R$ , where $a \in ¯ ¯¯ ¯ R$ and $b \in ¯ ¯¯ ¯ R$ , and let $c \in (a, b)$ . Then $g (x) \propto x \to a h (x)$ implies in $\int_{a}^{c} g (t) d t \propto \int_{a}^{c} h (t) d t$ and $g (x) \propto x \to b h (x)$ implies in $\int_{c}^{b} g (t) d t \propto \int_{c}^{b} h (t) d t$ .

3.1 Jeffreys prior

Jeffreys [18] described a procedure to achieve an objective prior, which is invariant under one-to-one monotone transformations. The invariant property of the Jeffreys prior has been widely exploited to make statistical inferences from its posterior distribution numerical analysis. The prior construction is based on the square root of the determinant of the Fisher information matrix $I (α, β)$ . Thus, the Jeffreys prior to the gamma distribution is given by

π_{1} (α, β) \propto \frac{\sqrt{α ψ^{'} (α) - 1}}{β} .

(10)

Additionally, from the determinant of the Fisher information, or using the change of variables over the Jeffreys prior we have

π_{1} (H, W) \propto \sqrt{W ψ^{'} (W) - 1} .

(11)

Finally, the joint posterior distribution for $H$ and $W$ produced by the Jeffreys prior is

π_{1} (H, W | x) \propto \frac{δ (W, H)^{n W} \sqrt{W ψ^{'} (W) - 1}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} exp {- δ (W, H) n \sum i = 1 x_{i}} .

(12)

Theorem 3.3.

The posterior density (12) is proper for all $n \geq 2$ .

Proof.

Using the change of variables $exp (- H) = u \Leftrightarrow d u = - exp (- H) d H$ and denoting $δ_{1} (W) = exp (W + log (Γ (W)) + (1 - W) ψ (W))$ it follows that

	$d_{1} (x)$	$\propto \int_{0}^{\infty} \int_{- \infty}^{\infty} π_{1} (H, W \| x) d H d W$
		$\propto \int_{0}^{\infty} \int_{0}^{\infty} \frac{δ_{1} (W)^{n W} u^{n W - 1} \sqrt{W ψ^{'} (W) - 1}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} exp {- δ_{1} (W) u n \sum i = 1 x_{i}} d u d W$
		$= \int_{0}^{\infty} \frac{δ_{1} (W)^{n W} \sqrt{W ψ^{'} (W) - 1}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} \int_{0}^{\infty} u^{n W - 1} exp {- δ_{1} (W) (n \sum i = 1 x_{i}) u} d u d W$
		$= \int_{0}^{\infty} \sqrt{W ψ^{'} (W) - 1} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} d W = \int_{0}^{1} g_{1} (W) d w + \int_{1}^{\infty} g_{1} (W) d w,$

where $g_{1} (W) = \sqrt{W ψ^{'} (W) - 1} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} > 0$ for all $W \in (0, \infty)$ . Now, according to [25, 28], we have $\frac{Γ (n W)}{Γ (W)^{n}} \propto W \to 0^{+} W^{n - 1}$ and $\sqrt{W ψ^{'} (W) - 1} \propto W \to 0^{+} W^{- 1 / 2}$ and since

lim W \to 0^{+} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} = 1 \Rightarrow \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \propto W \to 0^{+} 1,

it follows by Proposition 3.2 that

\int_{0}^{1} g_{1} (W) d W \propto \int_{0}^{1} W^{- 1 / 2} \times 1 \times W^{n - 1} d W < \infty .

Moreover, due to [25, 28] we have $\frac{Γ (n W)}{Γ (W)^{n}} \propto W \to \infty n^{n W} W^{(n - 1) / 2}$ and $\sqrt{W ψ^{'} (W) - 1} \propto W \to \infty W^{- 1 / 2}$ , and since $x_{i}$

are not all equal, due to the inequality of the arithmetic and geometric means we have

q = log (\frac{\frac{1}{n} \sum_{i = 1}^{n} x_{i}}{\sqrt[n]{\prod_{i = 1}^{n} x_{i}}}) > 0

and thus it follows that

\frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} = {⎛ ⎜ ⎜ ⎝ \frac{\frac{1}{n} \sum_{i = 1}^{n} x_{i}}{\sqrt[n]{\prod_{i = 1}^{n} x_{i}}} ⎞ ⎟ ⎟ ⎠}^{- n W} n^{- n W} = exp (- n q W) n^{- n W} .

Therefore, from Proposition 3.2 it follows that

	$\int_{1}^{\infty} g_{1} (W) d W$	$\propto \int_{1}^{\infty} W^{- 1 / 2} \times exp (- n q W) n^{- n W} \times n^{n W} W^{(n - 1) / 2} d W$
		$= \int_{1}^{\infty} W^{n / 2 - 1} exp (- n q W) d W = \frac{Γ (n / 2)}{(n q)^{n / 2}} < \infty,$

which concludes the proof. ∎

Theorem 3.4.

The posterior mean of $H$ relative to (12) is finite for any $n \geq 2$ .

Proof.

Doing the change of variables $exp (- H) = u \Leftrightarrow d u = - exp (- H) d H$ and denoting $δ_{1} (W) = exp (W + log (Γ (W)) + (1 - W) ψ (W))$ , it follows that

	$E_{1} [H \| x]$	$\propto \int_{0}^{\infty} \int_{- \infty}^{\infty} H π_{1} (α, β \| x) d H d W$
		$= \int_{0}^{\infty} \int_{0}^{\infty} - log (u) \frac{δ_{1} (W)^{n W} u^{n W - 1} \sqrt{W ψ^{'} (W) - 1}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} exp {- δ_{1} (W) u n \sum i = 1 x_{i}} d u d W$
		$= \int_{0}^{\infty} \frac{δ_{1} (W)^{n W} \sqrt{W ψ^{'} (W) - 1}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} \int_{0}^{\infty} (- log (u)) u^{n W - 1} exp {- δ_{1} (W) (n \sum i = 1 x_{i}) u} d u d W .$

Moreover, from the identity $ψ (z) Γ (z) = Γ^{'} (z) = \int_{0}^{\infty} log (t) t^{z - 1} e^{- t} d z$ one obtains that

\int_{0}^{\infty} log (s) s^{z - 1} e^{- a s} d s = 1 / a^{z} \int_{0}^{\infty} log (t / a) t^{z - 1} e^{- t} d t = 1 / a^{z} (ψ (z) Γ (z) - log (a) Γ (z))

and thus, letting $| \cdot |$ denote the absolute value operator and letting $δ_{2} (W) = | ψ (n W) | + | log (Γ (W)) | + (1 + W) | ψ (W) | + W + ∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣$ for all $W > 0$ , and using the triangle inequality we have

	$\| E_{1} [H \| x] \|$	$\propto ∣ ∣ ∣ ∣ \int_{0}^{\infty} (ψ (n W) - log (δ_{1} (W) n \sum i = 1 x_{i})) \sqrt{W ψ^{'} (W) - 1} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} d W ∣ ∣ ∣ ∣$
		$\leq \int_{0}^{\infty} ∣ ∣ ∣ ∣ ψ (n W) - log (δ_{1} (W) n \sum i = 1 x_{i}) ∣ ∣ ∣ ∣ \sqrt{W ψ^{'} (W) - 1} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} d W$
		$\leq \int_{0}^{\infty} δ_{2} (W) \sqrt{W ψ^{'} (W) - 1} \frac{}{{\prod_{i = 1}^{n} x_{i}^{W}}} {(n \sum i = 1 x_{i})}_{i}^{n W} \frac{Γ (n W)}{Γ (W)^{n}} d W = \int_{0}^{1} h_{1} (W) d W + \int_{1}^{\infty} h_{1} (W) d W,$

where $h_{1} (W) = δ_{2} (W) \sqrt{W ψ^{'} (W) - 1} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}}$ for all $W > 0$ .

We shall now prove that $δ_{2} (W) \propto W \to 0^{+} W^{- 1}$ and $δ_{2} (W) \propto W \to \infty W log (W)$ . Indeed, notice that $δ_{2} (W) \geq W > 0$ for $W > 0$ . Moreover, since due to Abramowitz [1] we have ${lim}_{W \to 0^{+}} W Γ (W) = 1$ and ${lim}_{W \to 0^{+}} - W ψ (W) = 1$ it follows that

		$lim W \to 0^{+} \frac{\| ψ (n W) \|}{W^{- 1}} = lim W \to 0^{+} \frac{1}{n} \| (n W) ψ (n W) \| = \frac{1}{n}$
		$lim W \to 0^{+} \frac{\| log (Γ (W)) \|}{W^{- 1}} = lim W \to 0^{+} \| W log (Γ (W)) - W log (W) \| = \| 0 \cdot log (1) - 0 \| = 0$
		$lim W \to 0^{+} \frac{(1 + W) \| ψ (W) \|}{W^{- 1}} = lim W \to 0^{+} (1 + W) \| W ψ (W) \| = 1 and$
		$lim W \to 0^{+} \frac{W + ∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣}{W^{- 1}} = lim W \to 0^{+} (W^{2} + W ∣ ∣ ∣ ∣ log (n \sum i = 1 x_{i}) ∣ ∣ ∣ ∣) = 0$

and thus

lim W \to 0^{+} \frac{δ_{2} (W)}{W^{- 1}} = \frac{1}{n} + 1 \Rightarrow δ_{2} (W) \propto W \to 0^{+} \frac{1}{W} .

On the other hand, since due to Abramowitz [1] we have ${lim}_{W \to \infty} \frac{ψ (W)}{log (W)} = 1$ , it follows from the L’hopital rule that

lim W \to \infty \frac{log (Γ (W))}{W (log (W) + 1)} = lim W \to \infty \frac{(log (Γ (W))^{'}}{(W (log (W) + 1))^{'}} = lim W \to \infty \frac{ψ (W)}{log (W)} = 1,

and therefore, considering $W \geq 1$ we have

		$lim W \to \infty \frac{\| ψ (W) \|}{W (log (W) + 1)} = lim W \to \infty \frac{1}{W} \frac{1}{(1 + log (W)^{- 1})} ∣ ∣ ∣ \frac{ψ (W)}{log (W)} ∣ ∣ ∣ = 0,$
		$lim W \to \infty \frac{\| log (Γ (W)) \|}{W (log (W) + 1)} = lim W \to \infty ∣ ∣ ∣ \frac{log (Γ (W))}{W (log (W) + 1)} ∣ ∣ ∣ = 1,$
		$lim W \to \infty \frac{(1 + W) \| ψ (W) \|}{W (log (W) + 1)} = lim W \to \infty (1 + W^{- 1}) \frac{1}{(1 + log (W)^{- 1})} ∣ ∣ ∣ \frac{ψ (W)}{log (W)} ∣ ∣ ∣ = 1, and$
		$lim W \to \infty \frac{W + ∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣}{W (log (W) + 1)} = lim W \to \infty (\frac{1}{log (W) + 1} + \frac{}{∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣} W (log (W) + 1)) = 0,$

and thus

lim W \to \infty \frac{δ_{2} (W)}{W (log (W) + 1)} = 2 \Rightarrow δ_{2} (W) \propto W \to \infty W log (W) .

Therefore, combining the obtained proportionality $δ_{2} (W) \propto W \to 0^{+} W^{- 1}$ with the proportionalities proved in Theorem 3.3 and using Proposition 3.2 we have

\int_{0}^{1} h_{1} (W) d W \propto \int_{0}^{1} W^{- 1} \times W^{- 1 / 2} \times 1 \times W^{n - 1} d W < \infty .

Finally, using the proportionality $δ_{2} (W) \propto W \to \infty W log (W)$ , letting $q > 0$ be as in the proof of Theorem 3.3 and using that $log (W) + 1 \leq exp (log (W)) = W$ for $W \geq 1$ , it follows from the proportionalities proved during Theorem 3.3 and from Proposition 3.2 that

	$\int_{1}^{\infty} h_{1} (W) d W$	$\propto \int_{1}^{\infty} W (log (W) + 1) \times W^{- 1 / 2} \times exp (- n q W) n^{- n W} \times n^{n W} W^{(n - 1) / 2} d W$
		$\leq \int_{1}^{\infty} W^{(n / 2 + 2) - 1} exp (- n q W) d W = \frac{}{Γ (n / 2 + 2)} (n q)^{n / 2 + 2} < \infty,$

which concludes the proof. ∎

In order to sample for the posterior distribution we obtain that the marginal posterior distributions of $W$ is given by

π_{1} (W | x) \propto \sqrt{W ψ^{'} (W) - 1} \frac{Γ (n W)}{Γ (W)^{n}} {⎛ ⎜ ⎜ ⎝ \frac{\sqrt[n]{\prod_{i = 1}^{n} x_{i}}}{\sum_{i = 1}^{n} x_{i}} ⎞ ⎟ ⎟ ⎠}^{n W},

and the conditional posterior distribution of $H$ is given by

π_{1} (H | W, x) \propto exp {- n W H - δ (W, H) n \sum i = 1 x_{i}} .

3.2 Reference prior

Bernardo [7] discussed a different approach to obtain a new class of objective priors, named as reference priors. Further, many studies were presented to develop formal and rigorous definitions to derive such class of prior distributions under different contexts [4, 5, 6, 2, 3]. The reference prior is obtained by maximizing the Kullback-Leibler (KL) divergence assuming some regularity conditions. The idea of the expected posterior information to the prior allows the data to have the maximum influence on the posterior distributions. The reference priors have essential properties such as consistent sampling, consistent marginalization, and one-to-one transformation invariance [8]. The reference priors may depend on the order of the parameters of interest. Hence, for the gamma distribution, we have two distinct priors that are presented below.

3.2.1 Reference prior when $β$ is the parameter of interest

The reference prior when $β$ is the parameter of interest and $α$ is the nuisance parameter is given by

π_{2} (α, β) \propto \frac{\sqrt{ψ^{'} (α)}}{β} .

(13)

Thus, using the Jacobian transformation it follows that the related reference prior is given by

π_{2} (W, H) \propto \sqrt{ψ^{'} (W)} .

(14)

Finally, the joint posterior distribution for $H$ and $W$ , produced by the reference prior (21), is given by

π_{2} (W, H | x) \propto δ (W, H)^{n H} \frac{\sqrt{ψ^{'} (W)}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{H}} exp {- δ (W, H) n \sum i = 1 x_{i}} .

(15)

Theorem 3.5.

The posterior density (15) is proper for all $n \geq 2$ .

Proof.

Doing the change of variables $exp (- H) = u \Leftrightarrow d u = - exp (- H) d H$ , denoting $δ_{1} (W) = exp (W + log (Γ (W)) + (1 - W) ψ (W))$ and proceeding analogously as in the proof of Theorem 3.3 we have

d_{2} (w) \propto

\int_{0}^{\infty} \int_{- \infty}^{\infty} π_{2} (W, H | x) d H d W \propto \int_{0}^{1} g_{2} (W) d w + \int_{1}^{\infty} g_{2} (W) d w,

where for all $W \in (0, \infty)$ . Now, according to [25, 28], we have $\frac{Γ (n W)}{Γ (W)^{n}} \propto W \to 0^{+} W^{n - 1}$ and $\sqrt{ψ^{'} (W)} \propto W \to 0^{+} W^{- 1}$ , and since we proved in Theorem 3.3 that $\frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \propto W \to 0^{+} 1$ , it follows from Proposition 3.2 that

\int_{0}^{1} g_{2} (W) d W \propto \int_{0}^{1} W^{- 1} \times 1 \times W^{n - 1} d W < \infty .

Moreover, from Abramowitz [1] we have $\sqrt{ψ^{'} (W)} \propto W \to \infty W^{- 1 / 2}$ , which combined with $\sqrt{W ψ^{'} (W) - 1} \propto W \to \infty W^{- 1 / 2}$ implies in $\sqrt{ψ^{'} (W)} \propto W \to \infty \sqrt{W ψ^{'} (W) - 1}$ . Therefore it follows that $g_{2} (W) \propto W \to \infty g_{1} (W)$ . and by Proposition 3.2 it follows that

\int_{1}^{\infty} g_{2} (W) d W \propto \int_{1}^{\infty} g_{1} (W) d W < \infty,

which concludes the proof. ∎

Theorem 3.6.

The posterior mean of $H$ relative to (15) is finite for all $n \geq 2$ .

Proof.

Proceeding analogously as in the proof of Theorem 3.4 it follows that

	$\| E_{2} [H \| x] \|$	$\propto \int_{0}^{\infty} ∣ ∣ ∣ \int_{- \infty}^{\infty} H π_{2} (H, W \| x) d H d W ∣ ∣ ∣$
		$\leq \int_{0}^{\infty} δ_{2} (W) \sqrt{ψ^{'} (W)} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} d W = \int_{0}^{1} h_{2} (W) d W + \int_{1}^{\infty} h_{2} (W) d W,$

where $δ_{2} (W)$ is the same as defined in the proof of Theorem 3.4 and

h_{2} (W) = δ_{2} (W) \sqrt{ψ^{'} (W)} \frac{{\prod_{i = 1}^{n} x_{i}^{W}}}{{(\sum_{i = 1}^{n} x_{i})}_{i}^{n W}} \frac{Γ (n W)}{Γ (W)^{n}} \cdot

Since in the proof of Theorem 3.4 we showed that $δ_{2} (W) \propto W \to 0^{+} W^{- 1}$ , together with the proportionalities proved in Theorem 3.3 and Proposition 3.2 we have

\int_{0}^{1} h_{2} (W) d W \propto \int_{0}^{1} W^{- 1} \times W^{- 1} \times 1 \times W^{n - 1} d W < \infty .

Finally, from the proof of Theorem 3.5 we know that $\sqrt{ψ^{'} (W)} \propto W \to \infty \sqrt{W ψ^{'} (W) - 1}$ , which implies directly that $h_{2} (W) \propto W \to \infty h_{1} (W)$ , and thus from Proposition 3.2 it follows that

\int_{1}^{\infty} h_{2} (W) d W \propto \int_{1}^{\infty} h_{1} (W) d W < \infty,

which concludes the proof. ∎

The marginal posterior distributions of $W$ is given by

π_{2} (W | x) \propto \sqrt{ψ^{'} (W)} \frac{Γ (n W)}{Γ (W)^{n}} {⎛ ⎜ ⎜ ⎝ \frac{\sqrt[n]{\prod_{i = 1}^{n} x_{i}}}{\sum_{i = 1}^{n} x_{i}} ⎞ ⎟ ⎟ ⎠}^{n W} .

Moreover, the conditional posterior distribution of $H$ is given by

π_{2} (H | W, x) \propto exp {- n W H - δ (W, H) n \sum i = 1 x_{i}} .

3.2.2 Reference prior when $α$ is the parameter of interest

The reference prior when $α$ is the parameter of interest and $β$ is the nuisance parameter is given by

π_{3} (α, β) \propto \frac{1}{β} \sqrt{\frac{α ψ^{'} (α) - 1}{α}} .

(16)

Therefore, in terms of the reparametrized model, the reference prior when $W$ is the parameter of interest and $H$ is the nuisance parameter is given by

π_{3} (W, H) \propto \sqrt{\frac{W ψ^{'} (W) - 1}{W}} .

(17)

Finally, the joint posterior distribution for $α$ and $β$ , produced by the reference prior (17) is given by

π_{3} (W, H | x) \propto \sqrt{\frac{W ψ^{'} (W) - 1}{W}} \frac{δ (W, H)^{n W}}{Γ (W)^{n}} {n \prod i = 1 x_{i}^{W}} exp {- δ (W, H) n \sum i = 1 x_{i}} .

(18)

Theorem 3.7.

The posterior density (18) is proper for all $n \geq 2$ .

Proof.

Since $\sqrt{\frac{W ψ^{'} (W) - 1}{W}} = \sqrt{ψ^{'} (W) - \frac{1}{W}} \leq \sqrt{ψ^{'} (W)}$ it follows that $π_{3} (W, H) \leq π_{2} (W, H)$ for all $W \in (0, \infty)$ and $H \in (- \infty,$

		$lim W \to 0^{+} \frac{\| ψ (n W) \|}{W^{- 1}} = lim W \to 0^{+} \frac{1}{n} \| (n W) ψ (n W) \| = \frac{1}{n}$
		$lim W \to 0^{+} \frac{\| log (Γ (W)) \|}{W^{- 1}} = lim W \to 0^{+} \| W log (Γ (W)) - W log (W) \| = \| 0 \cdot log (1) - 0 \| = 0$
		$lim W \to 0^{+} \frac{(1 + W) \| ψ (W) \|}{W^{- 1}} = lim W \to 0^{+} (1 + W) \| W ψ (W) \| = 1 and$
		$lim W \to 0^{+} \frac{W + ∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣}{W^{- 1}} = lim W \to 0^{+} (W^{2} + W ∣ ∣ ∣ ∣ log (n \sum i = 1 x_{i}) ∣ ∣ ∣ ∣) = 0$

		$lim W \to \infty \frac{\| ψ (W) \|}{W (log (W) + 1)} = lim W \to \infty \frac{1}{W} \frac{1}{(1 + log (W)^{- 1})} ∣ ∣ ∣ \frac{ψ (W)}{log (W)} ∣ ∣ ∣ = 0,$
		$lim W \to \infty \frac{\| log (Γ (W)) \|}{W (log (W) + 1)} = lim W \to \infty ∣ ∣ ∣ \frac{log (Γ (W))}{W (log (W) + 1)} ∣ ∣ ∣ = 1,$
		$lim W \to \infty \frac{(1 + W) \| ψ (W) \|}{W (log (W) + 1)} = lim W \to \infty (1 + W^{- 1}) \frac{1}{(1 + log (W)^{- 1})} ∣ ∣ ∣ \frac{ψ (W)}{log (W)} ∣ ∣ ∣ = 1, and$
		$lim W \to \infty \frac{W + ∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣}{W (log (W) + 1)} = lim W \to \infty (\frac{1}{log (W) + 1} + \frac{}{∣ ∣ log (\sum_{i = 1}^{n} x_{i}) ∣ ∣} W (log (W) + 1)) = 0,$

Objective Bayesian Analysis for the Differential Entropy of the Gamma Distribution

Authors

Objective Bayesian analysis for spatial Student-t regression models

Power laws distributions in objective priors

Posterior properties of the Weibull distribution for censored data

Estimation of entropy measures for categorical variables with spatial correlation

Meta-Learning Conjugate Priors for Few-Shot Bayesian Optimization

Entropy, Information, and the Updating of Probabilities

A Proof of Entropy Minimization for Outputs in Deletion Channels via Hidden Word Statistics

1 Introduction

2 Frequentist approach

3 Bayesian Inference

Definition 3.1.

Proposition 3.2.

3.1 Jeffreys prior

Theorem 3.3.

Proof.

Theorem 3.4.

Proof.

3.2 Reference prior

3.2.1 Reference prior when $β$ is the parameter of interest

Theorem 3.5.

Proof.

Theorem 3.6.

Proof.

3.2.2 Reference prior when $α$ is the parameter of interest

Theorem 3.7.

Proof.

Objective Bayesian Analysis for the Differential Entropy of the Gamma Distribution

Authors

Related Research

Objective Bayesian analysis for spatial Student-t regression models

Power laws distributions in objective priors

Posterior properties of the Weibull distribution for censored data

Estimation of entropy measures for categorical variables with spatial correlation

Meta-Learning Conjugate Priors for Few-Shot Bayesian Optimization

Entropy, Information, and the Updating of Probabilities

A Proof of Entropy Minimization for Outputs in Deletion Channels via Hidden Word Statistics

This week in AI

1 Introduction

2 Frequentist approach

3 Bayesian Inference

Definition 3.1.

Proposition 3.2.

3.1 Jeffreys prior

Theorem 3.3.

Proof.

Theorem 3.4.

Proof.

3.2 Reference prior

3.2.1 Reference prior when β is the parameter of interest

Theorem 3.5.

Proof.

Theorem 3.6.

Proof.

3.2.2 Reference prior when α is the parameter of interest

Theorem 3.7.

Proof.

3.2.1 Reference prior when $β$ is the parameter of interest

3.2.2 Reference prior when $α$ is the parameter of interest