A Note on New Bernstein-type Inequalities for the Log-likelihood Function of Bernoulli Variables

1 Introduction

Let $X_{1}, X_{2}, . . ., X_{n}$

be independent Bernoulli random variables, where

X_{i}

takes the value 1 with probability

p_{i}

, denoted by Ber(

p_{i}

). We are interested in deriving a concentration bound, which decays exponentially and is independent of parameters

p_{i}

, for the joint log-likelihood function of

X_{1}, X_{2}, . . ., X_{n}

. That is,

P (∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 (X_{i} log p_{i} + (1 - X_{i}) log (1 - p_{i})) - \frac{1}{n} n \sum i = 1 (p_{i} log p_{i} + (1 - p_{i}) log (1 - p_{i})) ∣ ∣ ∣ ∣ \geq ϵ) \leq c_{1} e^{- c_{2} n},

(1)

where $c_{1}$ and $c_{2}$ are constants that only depend on $ϵ$ .

This research is motivated by theoretical studies of likelihood-based methods for binary data, in particular likelihood-based methods for community detection in networks. For example, Theorem 2 in Choi et al. (2012) uses on an inequality of this type and so does Theorem 2 in Paul and Chen (2016).

We begin with classical results. By symmetry, we only consider

P (∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 X_{i} log p_{i} - \frac{1}{n} n \sum i = 1 p_{i} log p_{i} ∣ ∣ ∣ ∣ \geq ϵ) .

Since $X_{i} log p_{i} \equiv p_{i} log p_{i}$ almost surely when $p_{i} = 1$ or 0 (using the convention $0 log 0 = 0$ ), the term can be dropped. Without loss of generality, assume $p_{i} \in (0, 1)$ for $i = 1, . . ., n$ . Noticing that $X_{i} log p_{i} \in [log p_{i}, 0]$ for $i = 1, . . ., n$ , we have Hoeffding’s inequality (Hoeffding, 1963): for all $ϵ > 0$ ,

P (\frac{1}{n} ∣ ∣ ∣ ∣ n \sum i = 1 (X_{i} - p_{i}) log p_{i} ∣ ∣ ∣ ∣ \geq ϵ) \leq 2 exp {- \frac{2 n^{2} ϵ^{2}}{\sum_{i = 1}^{n} (log p_{i})^{2}}} .

Let $p_{(1)}$ be the smallest value among $p_{1}, . . ., p_{n}$ . Then $| (X_{i} - p_{i}) log p_{i} | \leq | log p_{(1)} |$ for $i = 1, . . ., n$ . Bernstein’s inequality (see Dubhashi and Panconesi (2009), Theorem 1.2) gives: for all $ϵ > 0$ ,

P (\frac{1}{n} ∣ ∣ ∣ ∣ n \sum i = 1 (X_{i} - p_{i}) log p_{i} ∣ ∣ ∣ ∣ \geq ϵ) \leq 2 exp {- \frac{n^{2} ϵ^{2} / 2}{\sum_{i = 1}^{n} Var (X_{i} log p_{i}) + | log p_{(1)} | n ϵ / 3}} .

Note that both inequalities depend on $p_{1}, . . ., p_{n}$ . As a result, when $p_{(1)}$ goes to 0 fast enough as $n$ grows, the bounds can be trivial due to the divergence of $| log p_{(1)} |$ . When applying these inequalities, technical assumptions are therefore needed to control the rate of the parameters going to the boundaries, for example, the condition on $P_{i j}$ in Theorem 2 of Choi et al. (2012).

In this note, we prove a Bernstein-type inequality where the bound is independent of $p_{1}, . . ., p_{n}$ . In other words, we show that $\sum_{i = 1}^{n} (X_{i} - p_{i}) log p_{i}$ is in fact well-behaved when the parameters are near the boundary. The results such as in Choi et al. (2012) and Paul and Chen (2016) can therefore be strengthened by removing the technical assumptions. The new inequality is particularly useful in cases where those assumptions are not convenient to be made.

2 Main Result

Theorem 1.

Let $X_{i}$ be independent $Ber (p_{i})$ for $i = 1, . . ., n$ where $p_{i} \in [0, 1]$ . For all $t > 0$ ,

(2)

Proof.

Let $Y_{i} = (X_{i} - p_{i}) log p_{i}$ . Let $G (p_{i}, λ)$ be the moment generating function of $Y_{i}$ , which is

G (p_{i}, λ) = E [e^{λ Y_{i}}] = p_{i} e^{λ (1 - p_{i}) log p_{i}} + (1 - p_{i}) e^{- λ p_{i} log p_{i}} .

The key step is to derive an exponential upper bound for $G (p_{i}, λ)$ which is independent of $p_{i}$ . First consider the case where $p_{i} \in (0, 1)$ .

We prove a Bernstein’s condition (see Wainwright (2019), p. 27 for an introduction) for the moments of $Y_{i}$ . That is, find constants $σ^{2}$ and $b$ , such that

| E [Y_{i}^{m}] | \leq \frac{1}{2} m! σ^{2} b^{m - 2} for m = 3, 4, . . . .

(3)

Different from Wainwright (2019), here we look for constants $σ^{2}$ and $b$ which are independent of $p_{i}$ .

Consider

E [Y_{i}^{m}] = p_{i} (1 - p_{i})^{m} (log p_{i})^{m}      A_{1} + (1 - p_{i}) (- p_{i} log p_{i})^{m}      A_{2} .

By taking the first and the second derivatives of $p_{i} (log p_{i})^{m}$ , one can easily check that its optimum is achieved at $p_{i} = e^{- m}$ . Therefore,

| A_{1} | \leq | p_{i} (log p_{i})^{m} | \leq {(\frac{m}{e})}^{m} \leq \frac{m!}{\sqrt{2 π m}},

where the last inequality follows from Stirling’s formula (Robbins, 1955). Similarly,

| A_{2} | \leq (1 - p_{i}) (- p_{i} log p_{i})^{m} \leq e^{- m} .

It follows that

| E [Y_{i}^{m}] | \leq \frac{m!}{\sqrt{2 π m}} + \frac{1}{e^{m}} \leq \frac{1}{2} m! for m = 3, 4, . . . .

Therefore, the Bernstein’s condition (3) holds when $σ^{2} = 1$ and $b = 1$ .

We now use the Bernstein’s condition to derive an upper bound for $G (p_{i}, λ)$ . The argument is similar to Wainwright (2019), pp. 27-28. We give the details for completeness.

By the power series expansion of the exponential function and Fubini’s theorem (for exchanging the expectation and summation),

	$G (p_{i}, λ) = E [e^{λ Y_{i}}]$	$= 1 + \frac{λ^{2} Var (Y_{i})}{2} + \infty \sum m = 3 λ^{m} \frac{E [Y_{i}^{m}]}{m!}$
		$\leq 1 + \frac{λ^{2}}{2} + \frac{λ^{2}}{2} \infty \sum m = 1 \| λ \|^{m},$

where the inequality follows from the Bernstein’s condition (3) and $Var (Y_{i}) = p_{i} (1 - p_{i}) (log p_{i})^{2} \leq 1$ . For any $| λ | < 1$ , the geometrics series converges, and

G (p_{i}, λ) \leq 1 + \frac{λ^{2}}{2} \frac{1}{1 - | λ |} \leq exp {\frac{λ^{2}}{2 (1 - | λ |)}},

(4)

where the second inequality follows from $1 + s \leq e^{s}$ . Notice that $G (p_{i}, λ) \equiv 1$ for $p_{i} = 0$ or $1$ so the inequality holds for all $p_{i} \in [0, 1]$ .

The rest of the proof follows from a standard argument using the Chernoff bound, which can be found in a standard textbook on concentration inequalities, for example, Dubhashi and Panconesi (2009), Chapter 1. We give the details for readers who are unfamiliar with this technique. For $- 1 < λ < 0$ ,

P (n \sum i = 1 Y_{i} \leq - t) = P (e^{λ \sum_{i = 1}^{n} Y_{i}} \geq e^{- λ t}) \leq \frac{\prod_{i = 1}^{n} E [e^{λ Y_{i}}]}{e^{- λ t}} \leq exp {\frac{n λ^{2}}{2 (1 - | λ |)} + λ t},

where the first inequality is Markov’s inequality and the second inequality follows from (4). By setting $λ = - \frac{t}{t + n} \in (- 1, 0)$ , we obtain

P (n \sum i = 1 Y_{i} \leq - t) \leq exp {- \frac{t^{2}}{2 (n + t)}} .

The bound for the right tail can be obtained similarly by setting $λ = \frac{t}{t + n}$ . ∎

Remark 1.

$E [Y_{i}^{m}]$ is dominated by the term $p_{i} (1 - p_{i})^{m} (log p_{i})^{m}$ , which has a bump near the boundary, – that is, its value achieves the order of $(m / e)^{m}$ at $p_{i} = e^{- m}$ . This value is, however, still bounded by $m!$ , which implies the left-tail bound of $\sum_{i = 1}^{n} Y_{i}$ is well-behaved when the parameters are near the boundary.

Remark 2.

The constant $σ^{2} = 1$ in the Bernstein’s condition (3) is not the optimal value. We simply choose this value for obtaining a nice form in (2). On the contrary, $b = 1$ is optimal because $1 / \sqrt{2 π m}$ dominates $b^{m - 2}$ for any $0 < b < 1$ . This fact can also be seen from the following proposition:

Proposition 1.

For $λ < - 1$ , ${lim}_{p \to 0^{+}} G (p, λ) = \infty$ , which implies $G (p, λ)$ cannot be bounded by any function that takes finite values. For $λ \geq - 1$ , ${lim}_{p \to 0^{+}} G (p, λ) < \infty$ .

Proof.

The result is obvious by noticing that $p e^{λ log p} = p^{λ + 1}$ . ∎

We now prove (1). We state a slightly more general result for multinoulli variables. Let $X_{i} = (X_{i 1}, . . ., X_{i K})$ be a multinoulli variable with $p_{i k} = P (X_{i k} = 1)$ , and assume $X_{1}, . . ., X_{n}$ are independent.

Corollary 1.

For $p_{i k} \in [0, 1]$ , $i = 1, . . ., n$ , $k = 1, . . ., K$ , and all $ϵ > 0$ ,

P (∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 K \sum k = 1 (X_{i k} - p_{i k}) log p_{i k} ∣ ∣ ∣ ∣ \geq ϵ) \leq 2 K exp {- \frac{n ϵ^{2}}{2 K (K + ϵ)}} .

Proof.

The result is obvious by noticing that

P (∣ ∣ ∣ ∣ \frac{1}{n} n \sum i = 1 K \sum k = 1 (X_{i k} - p_{i k}) log p_{i k} ∣ ∣ ∣ ∣ \geq ϵ) \leq K \sum k = 1 P (∣ ∣ ∣ ∣ n \sum i = 1 (X_{i k} - p_{i k}) log p_{i k} ∣ ∣ ∣ ∣ \geq \frac{n ϵ}{K}),

and setting $t = n ϵ / K$ in (2). ∎

3 Extension to Grouped Observations

We now extend our result to a setup where the observations are grouped into different classes. In fact, this is the setup that can be directly applied to the community detection literature, for example, Theorem 2 in Choi et al. (2012) and Theorem 2 in Paul and Chen (2016). We will also apply the result in a working paper by the author and collaborators on the theory of hub models, a special latent class model for binary data proposed by Zhao and Weko (2019).

Let $X_{1}^{(1)}, X_{2}^{(1)}, . . ., X_{n_{1}}^{(1)}, X_{1}^{(2)}, X_{2}^{(2)}, . . ., X_{n_{2}}^{(2)}, . . ., X_{1}^{(I)}, X_{2}^{(I)}, . . ., X_{n_{I}}^{(I)}$ be independent Bernoulli variables, where $p_{j}^{(i)}$ is the parameter for $X_{j}^{(i)}$ . Let $\sum_{i = 1}^{I} n_{i} = n$ . And let ${¯ p}^{(i)} = \frac{1}{n_{i}} \sum_{j = 1}^{n_{i}} p_{j}^{(i)}$ for $i = 1, . . ., I$ , where ${¯ p}^{(i)} \in [0, 1]$ .

Theorem 2.

For all $t > 0$ ,

P (∣ ∣ ∣ ∣ I \sum i = 1 n_{i} \sum j = 1 (X_{j}^{(i)} - p_{j}^{(i)}) log {¯ p}^{(i)} ∣ ∣ ∣ ∣ \geq t) \leq 2 exp {- \frac{t^{2}}{2 (n + t)}} .

(5)

Note that here the model assumption on $X_{j}^{(i)}$ is identical to the setup in Section 2, where each Bernoulli variable has its own parameter. The function we consider in the inequality is, however, defined differently. Moreover, this theorem reduces to Theorem 1 when $n_{i} \equiv 1$ for $i = 1, . . ., I$ .

Proof.

Let $Z^{(i)} = \sum_{j = 1}^{n_{i}} (X_{j}^{(i)} - p_{j}^{(i)}) log {¯ p}^{(i)} = \sum_{j = 1}^{n_{i}} (X_{j}^{(i)} - {¯ p}^{(i)}) log {¯ p}^{(i)}$ . Consider the moment generating function $E [e^{λ Z^{(i)}}]$ for ${¯ p}^{(i)} \in (0, 1)$ .

	$E [e^{λ Z^{(i)}}] =$	$n_{i} \prod j = 1 (p_{j}^{(i)} e^{λ (1 - {¯ p}^{(i)}) log {¯ p}^{(i)}} + (1 - p_{j}^{(i)}) e^{- λ {¯ p}^{(i)} log {¯ p}^{(i)}})$
	$\leq$	${({¯ p}^{(i)} e^{λ (1 - {¯ p}^{(i)}) log {¯ p}^{(i)}} + (1 - {¯ p}^{(i)}) e^{- λ {¯ p}^{(i)} log {¯ p}^{(i)}})}^{n_{i}} = (G ({¯ p}^{(i)}, λ))^{n_{i}},$

where the inequality follows from the inequality of arithmetic and geometric means:

\sqrt[n]{\prod_{i = 1}^{n} a_{i}} \leq \sum_{i = 1}^{n} a_{i} / n

for non-negative

a_{1}, . . ., a_{n}

. From (4),

G ({¯ p}^{(i)}, λ) \leq exp {\frac{λ^{2}}{2 (1 - | λ |)}}

. It follows that

E [e^{λ Z^{(i)}}] \leq exp {\frac{n_{i} λ^{2}}{2 (1 - | λ |)}}

. The inequality also holds for

{¯ p}^{(i)} = 0

or 1 as

E [e^{λ Z^{(i)}}] \equiv 1

. The rest of the proof follows from the standard argument using the Chernoff bound as shown in the proof of Theorem 1. ∎

We conclude this note with a corollary that is easily proved by the same argument for Corollary 1. Let $X_{1}^{(1)}, X_{2}^{(1)}, . . ., X_{n_{1}}^{(1)}, X_{1}^{(2)}, X_{2}^{(2)}, . . ., X_{n_{2}}^{(2)}, . . ., X_{1}^{(I)}, X_{2}^{(I)}, . . ., X_{n_{I}}^{(I)}$ be independent multinoulli variables, where each $X_{j}^{(i)} = (X_{j 1}^{(i)}, . . ., X_{j K}^{(i)})$ , and $p_{j k}^{(i)} = P (X_{j k}^{(i)} = 1)$ for $k = 1, . . ., K$ . As before, let $n = \sum_{i = 1}^{I} n_{i}$ . And let ${¯ p}_{k}^{(i)} = \frac{1}{n_{i}} \sum_{j = 1}^{n_{i}} p_{j k}^{(i)}$ for $i = 1, . . ., I$ and $k = 1, . . . K$ , where ${¯ p}_{k}^{(i)} \in [0, 1]$ .

Corollary 2.

For all $ϵ > 0$ ,

P (\frac{1}{n} ∣ ∣ ∣ ∣ I \sum i = 1 n_{i} \sum j = 1 K \sum k = 1 (X_{j k}^{(i)} - p_{j k}^{(i)}) log {¯ p}_{k}^{(i)} ∣ ∣ ∣ ∣ \geq ϵ) \leq 2 K exp {- \frac{n ϵ^{2}}{2 K (K + ϵ)}} .

Acknowledgements

This research was supported by the National Science Foundation grant DMS-1840203.

References

Choi et al. (2012) Choi, D. S., Wolfe, P. J., and Airoldi, E. M. (2012). Stochastic blockmodels with a growing number of classes. Biometrika, 99(2):273–284.
Dubhashi and Panconesi (2009) Dubhashi, D. P. and Panconesi, A. (2009). Concentration of measure for the analysis of randomized algorithms. Cambridge University Press.
Hoeffding (1963) Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30.
Paul and Chen (2016) Paul, S. and Chen, Y. (2016). Consistent community detection in multi-relational data through restricted multi-layer stochastic blockmodel. Electronic Journal of Statistics, 10(2):3807–3870.
Robbins (1955) Robbins, H. (1955). A remark on stirling’s formula. The American mathematical monthly, 62(1):26–29.
Wainwright (2019) Wainwright, M. J. (2019). High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
Zhao and Weko (2019) Zhao, Y. and Weko, C. (2019). Network inference from grouped observations using hub models. Statistica Sinica, 29(1):225–244.

A Note on New Bernstein-type Inequalities for the Log-likelihood Function of Bernoulli Variables

Authors

On Optimal Uniform Concentration Inequalities for Discrete Entropies in the High-dimensional Setting

Empirical Analysis of Sampling Based Estimators for Evaluating RBMs

Computing the Dirichlet-Multinomial Log-Likelihood Function

Accelerate iterated filtering

Parameter Learning for Log-supermodular Distributions

Concave likelihood-based regression with finite-support response variables

Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss

1 Introduction

2 Main Result

Theorem 1.

Proof.

Remark 1.

Remark 2.

Proposition 1.

Proof.

Corollary 1.

Proof.

3 Extension to Grouped Observations

Theorem 2.

Proof.

Corollary 2.

Acknowledgements

References

A Note on New Bernstein-type Inequalities for the Log-likelihood Function of Bernoulli Variables

Authors

Related Research

On Optimal Uniform Concentration Inequalities for Discrete Entropies in the High-dimensional Setting

Empirical Analysis of Sampling Based Estimators for Evaluating RBMs

Computing the Dirichlet-Multinomial Log-Likelihood Function

Accelerate iterated filtering

Parameter Learning for Log-supermodular Distributions

Concave likelihood-based regression with finite-support response variables

Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss

This week in AI

1 Introduction

2 Main Result

Theorem 1.

Proof.

Remark 1.

Remark 2.

Proposition 1.

Proof.

Corollary 1.

Proof.

3 Extension to Grouped Observations

Theorem 2.

Proof.

Corollary 2.

Acknowledgements

References