Variational Approximation Accuracy in Bayesian Non-negative Matrix Factorization

1 Introduction

1.1 Algorithms for Non-negative Matrix Factorization

Non-negative matrix factorization (NMF) [16, 4] has been applied to text mining [21], signal processing [14], bioinformatics [11], consumer analysis [12], and recommender systems [3]. NMF experiments discover the knowledge and predict the future unknown structures in the real world, however, the method suffers from many local minima and seldom reaches the global minimum. In addition, the results of numerical experiments strongly depend on the initial values; a rigorous method has yet to be established.

In order to resolve this difficulty, Bayesian inference for NMF has been established

[4]

. It uses, for numerical calculation of the Bayesian posterior distribution, Gibbs sampling method which is a kind of Markov chain Monte Carlo method (MCMC). Bayesian NMF is more robust than usual recursive methods of NMF since it numerically realizes the posterior distribution; the parameters are subject to a probability distribution and that makes it possible to grasp the degree of fluctuation of the learning/inference result. As is described later, in general, Bayesian method has higher estimation accuracy than maximum likelihood estimation and maximum posterior estimation if the model has hierarchical structures or hidden variables, like NMF.

On the other hand, the variational Bayesian algorithm (VB) for NMF has also been established [4], with being inspired the mean field approximation. The variational Bayesian NMF algorithm (VBNMF) also results more numerically stable than usual recursive algorithms as VB approximates the Bayesian posterior distribution. Moreover, VBNMF computes faster than usual Bayesian inference such a MCMC. However, its free energy (called the variational free energy) is larger than the Bayesian free energy, since VB ascends the evidence lower bound (ELBO) but it is not the true model evidence. Note that the marginal likelihood is also called the model evidence and the negative log ELBO is equal to the variational free energy. From the above, it is important to clarify the approximation accuracy of the variational free energy for not only theoretical reasons but also practical points of view.

1.2 Learning Theory of Bayesian and Variational Inference

A statistical model is called regular if a function from a parameter set to a probability density function set is one-to-one and if the likelihood function can be approximated by a Gaussian function. It is proved that, if a statistical model is regular and if a true distribution is realizable by a statistical model, then the generalization error is asymptotically equal to

d / (2 n)

, where

d

n

, and the generalization error are the dimension of the parameter, the sample size, and the expected Kullback-Leibler divergence of the true distribution and the estimated learning machine, respectively. Moreover, the negative log marginal likelihood or the free energy asymptotically behaves

(d / 2) log n + O_{p} (1)

. However, the statistical model used in NMF is not regular because the map from a parameter to a probability density function is not injective. As a result, its generalization error and the free energy is still unknown.

There are many non-regular statistical models in machine learning. For example, reduced rank regressions, normal mixture models, neural networks, hidden Markov models, and Boltzmann machines are such examples. From the theoretical point of view, the free energy and the generalization error of a non-regular learning machine in Bayesian learning was proved to be asymptotically equal to

λ log n

and

λ / n

, where

λ

is a real log canonical threshold (RLCT), respectively [18, 19]. Moreover, in non-regular cases,

λ < d / 2

holds and

λ

is also much less than the learning coefficients of maximum likelihood/posterior methods [20]. The RLCTs for several learning machines, have been clarified. In fact, reduced rank regressions [1], normal mixture models [22], and hidden Markov models [23], they are clarified by using resolution of singularities. A statistical model selection method sBIC using RLCTs has also been proposed [6].

On the other hand, for several statistical models, the variational free energy was proved that it asymptotically equals $λ_{v b} log n + O_{p} (1)$ ,where $λ_{v b}$ is a learning coefficient and it depends on the model. Normal mixture models [17], hidden Markov models [10], and NMF [13] are such examples. In general, the learning coefficient of VB may not be equal to but becomes an upper bound of the RLCT: $λ_{v b} ≧ λ$ , since the variational free energy is larger than the usual free energy even if the sample size diverges infinity. Unfortunately, the variational generalization error is not equal to $λ_{v b} / n$ , asymptotically.

VBNMF has been devised [4], and the exact learning coefficient of VBNMF has been derived [13]. Nevertheless, the variational approximation accuracy has not been clarified since the RLCT of NMF has been unknown. If the prior distribution is strictly and entirely positive and bounded analytic function on the domain, then an upper bound of the RLCT of NMF has been proved [9, 8]. If the non-negative restriction is not assumed for matrix factorization, then the exact value of the RLCT has been clarified as reduced rank regression models [1]

. However, the RLCT has been unknown in the case of that the prior is gamma distributions, which may be zero.

In this paper, we theoretically derive the lower bound of the RLCT difference $λ_{v b} - λ$ in NMF, by which we can derive the lower bound of the approximation accuracy of VBNMF for Bayesian NMF.

The rest of this paper is organized as follows. In the second section, we briefly explain Bayesian inference and Variational Bayesian algorithm. In the third section, we present the main theorems and their proofs. In the fourth section, we conclude this paper.

2 Inference Frameworks

In this section, we explain the framework of Bayesian inference, VB algorithm.

2.1 Framework of Bayesian Inference

First, we explain the general theory of Bayesian inference.

Let $q (x)$ and $p (x | w)$ be probability density functions on a finite dimensional real Euclidean space, where $w$ is a parameter. In learning theory, $q (x)$ and $p (x | w)$ represent a true distribution and a learning machine with $w$ respectively. A probability density function on a set of parameters $φ (w)$ is called a prior. Usually, the prior has parameters $ϕ$ that is called hyperparameter. Let $X^{n} = (X_{1}, X_{2}, . . ., X_{n})$

be a set of random variables that are independently subject to

q (x)

, where

n

and

X^{n}

are referred to as the sample size and training data. The posterior of

w

is defined by

ψ (w | X^{n}) = \frac{1}{Z_{n}} φ (w) n \prod i = 1 p (X_{i} | w),

where $Z_{n}$ is the normalizing constant that is determined by the condition $\int ψ (w | X^{n}) = 1$ . $Z_{n}$ is called the marginal likelihood, evidence, or partition function, and it is also a probability density function of the training data: $Z_{n} = Z_{n} (X^{n})$ . The Bayesian predictive distribution is also defined by

p (x | X^{n}) = \int p (x | w) ψ (w | X^{n}) d w .

The negative log marginal likelihood (or free energy) is defined by

F_{n} = - log Z_{n} = - log \int φ (w) n \prod i = 1 p (X_{i} | w) .

The generalization error $G_{n}$ is also defined by the Kullback-Leibler divergence from the true distribution $q (x)$ and the predictive one $p (x | X^{n})$ :

G_{n} = \int q (x) log \frac{q (x)}{p (x | X^{n})} d x .

Note that $F_{n}$ and $G_{n}$ are functions of $X^{n}$ hence it is also a random variable. These expected value overall training data $E [F_{n}]$ and $E [G_{n}]$ are called the expected free energy and generalization error, respectively, where this expectation $E [\cdot]$ is subject to the true distribution of the training data

Q (X^{n}) = Q (X_{1}, \dots, X_{n}) = n \prod i = 1 q (X_{i})

and defined by

E [\cdot] = \int Q (x_{1}, \dots, x_{n}) (\cdot) d x_{1} \dots d x_{n}

Assume there exists a parameter $w_{0}$ that satisfies $q (x) = p (x | w_{0})$ . By the singular learning theory [18, 19, 20], it was proven that

	$F_{n}$	$= n S_{n} + λ log n - (m - 1) log log n + O_{p} (1),$		(1)
	$E [G_{n}]$	$= \frac{λ}{n} + o (\frac{1}{n})$		(2)

hold when $n$ tends to infinity, even if the posterior distribution can not

be approximated by any normal distribution, where

S_{n} = - \frac{1}{n} \sum_{i = 1}^{n} log q (X_{i})

is the empirical entropy. The constant

λ

is the real log canonical threshold (RLCT) which is an important birational invariant in algebraic geometry. The constant

m

is called the multiplicity and also a birational invariant. From the mathematical point of view, RLCT is characterized by the following property. We defined a zeta function by

ζ (z) = \int K (w)^{z} φ (w) d w,

where

K (w) = \int q (x) log \frac{q (x)}{p (x | w)} d x .

Then this is holomorphic in $R e (z) > 0$ which can be analytically continued to a unique meromorphic function on the entire complex plane [2]. The poles of this extended function are all negative rational numbers. Let $(- λ)$ be the nearest pole to the origin; $λ$ is then equal to the RLCT. The multiplicity $m$ is denoted by the order of the nearest pole. If $p (x | w)$ is regular then $λ = d / 2$ ; however, it is not usually general.

2.2 Variational Bayesian Algorithm

Variational Bayesian algorithm (VB) or variational approximation is an approximation method for Bayesian inference. VB is based on the mean field approximation, and possible to make the numerical calculation cost less than usual Bayesian inference.

Let the training data be $X^{n} = (X_{1}, \dots, X_{n})$ and the posterior be $ψ (w | X^{n}) = ψ (w^{1}, \dots, w^{d} | X^{n})$ . In general, the posterior distribution cannot be found analytically thus we assume that the parameters are independent:

ψ (w | X^{n}) = ψ_{v b}^{1} (w^{1} | X^{n}) \dots ψ_{v b}^{d} (w^{d} | X^{n}) .

This assumption is just an approximation but useful if $ψ_{v b}^{1} (w^{1} | X^{n}) \dots ψ_{v b}^{d} (w^{d} | X^{n})$ are more simple form than the original posterior.

This approximation is meaning as Kullback-Leibler divergence; put

ψ_{v b} (w | X^{n}) := d \prod j = 1 ψ_{v b}^{j} (w^{1} | X^{n})

and minimize

K L (ψ_{v b} ∥ ψ) = \int ψ_{v b} (w | X^{n}) log \frac{ψ_{v b} (w | X^{n})}{ψ (w | X^{n})} d w .

The problem of Bayesian inference is numerical realization of the posterior and VB solves as the above optimization. In the practical cases, the parameters are often decomposed by several, especially just two, parts and they are assumed to be independent:

ψ (w | X^{n}) \approx ψ_{v b} (w | X^{n}) = p_{v b}^{a} (w_{a} | X^{n}) p_{v b}^{b} (w_{b} | X^{n}),

where $w = (w_{a}, w_{b})$ , $w_{a} = (w_{a_{1}}, \dots, w_{a_{k}})$ , $w_{b} = (w_{b_{1}}, \dots, w_{b_{d - k}}) .$

VB optimizes the above Kullback-Leibler divergence by searching $ψ_{v b}$ , however, the objective function may be not calculated analytically. This is because the marginal likelihood is contained:

	$K L (ψ_{v b} ∥ ψ)$	$= \int ψ_{v b} (w \| X^{n}) log \frac{ψ_{v b} (w \| X^{n})}{ψ (w \| X^{n})} d w$
		$= \int ψ_{v b} (w \| X^{n}) log \frac{ψ_{v b} (w \| X^{n})}{P (X^{n} \| w) φ (w) / Z_{n} (X^{n})} d w$
		$= \int ψ_{v b} (w \| X^{n}) log (ψ_{v b} (w \| X^{n}) - P (X^{n} \| w) φ (w)) d w$
		$+ \int ψ_{v b} (w \| X^{n}) log Z_{n} (X^{n}) d w$
		$= \int ψ_{v b} (w \| X^{n}) log (ψ_{v b} (w \| X^{n}) - P (X^{n} \| w) φ (w)) d w + log Z_{n} (X^{n}),$

where $P (X^{n} | w) = \prod_{i = 1}^{n} p (X_{i} | w)$ .

$p (x | w)$ and $φ (w)$ are designed by the human thus the first term only contains known variables. The minimization $K L (ψ_{v b} ∥ ψ)$ problem returns to minimization the following functional of the mean field approximation $ψ_{v b}$ :

ϝ (ψ_{v b}) := \int ψ_{v b} (w | X^{n}) log (ψ_{v b} (w | X^{n}) - P (X^{n} | w) φ (w)) d w .

If there exists ${^ψ}_{v b}$ such that ${^ψ}_{v b} = ψ$ , then the Kullback-Leibler divergence is equal to 0 i.e. the functional is equal to the free energy: $ϝ ({^ψ}_{v b}) = - log Z_{n} (X^{n}) = F_{n}$ . Nevertheless, in general, there may be no ${^ψ}_{v b}$ such that ${^ψ}_{v b} = ψ$ . The variational free energy ${¯ F}_{n}$ is defined by the minimum of $ϝ$ :

{¯ F}_{n} = min ψ_{v b} ϝ (ψ_{v b}) .

The inequality ${¯ F}_{n} ≧ F_{n}$ is immediately derived by the definition.

For example, if the approximation

ψ (w | X^{n}) \approx ψ_{v b} (w | X^{n}) = p_{v b}^{a} (w_{a} | X^{n}) p_{v b}^{b} (w_{b} | X^{n})

is applied, then

	${¯ F}_{n}$	$= min ψ_{v b} = ψ_{v b}^{a} ψ_{v b}^{b} {\int ψ_{v b}^{a} (w_{a} \| X^{n}) log ψ_{v b}^{a} (w_{a} \| X^{n}) d w_{a}$
		$+ \int ψ_{v b}^{b} (w_{b} \| X^{n}) log ψ_{v b}^{b} (w_{b} \| X^{n}) d w_{b}$
		$+ \iint ψ_{v b}^{a} (w_{a} \| X^{n}) ψ_{v b}^{b} (w_{b} \| X^{n}) log (P (X^{n} \| w) φ (w)) d w_{a} d w_{b}} .$

Besides, $ψ_{v b}^{a}$ and $ψ_{v b}^{b}$ satisfy the following self-consistency condition:

	$log ψ_{v b}^{a} (w_{a} \| X^{n}) = C_{1} + \int ψ_{v b}^{b} (w_{b} \| X^{n}) log ψ (w_{a}, w_{b} \| X^{n}) d w_{b},$
	$log ψ_{v b}^{b} (w_{b} \| X^{n}) = C_{2} + \int ψ_{v b}^{a} (w_{a} \| X^{n}) log ψ (w_{a}, w_{b} \| X^{n}) d w_{a},$

where $C_{1}, C_{2}$ are the normalizing constants.

In this way, VB is an approximation of Bayesian and its accuracy can be expressed by ${¯ F}_{n} - F_{n} ≧ 0$ , the less the difference is, the more valid the approximation is. From the theoretical point of view, ${¯ F}_{n}$ has an asymptotic behavior which is similar to the one of $F_{n}$ . For example, in VBNMF, the following theorem is proved [13].

Theorem 2.1 (Kohjima).

Let the elements of the data matrices $x_{i j}$ $(i = 1, \dots, M;$ $j = 1, \dots, N)$

be independently generated from the Poisson distribution whose mean is equal to the

(i, j)

element of

U_{0} V_{0}

, where the number of columns in

U_{0}

(= the number of rows in V_{0})

H_{0}

; called the non-negative rank of

U_{0} V_{0}

[5].

Let the likelihood model and the prior be the following Poisson and gamma distributions, respectively:

	$p (X \| U, V) = M \prod i = 1 N \prod j = 1 \frac{((U V)_{i j})^{x_{i j}}}{x_{i j}!} e^{- (U V)_{i j}},$

where $ϕ_{U}, θ_{U}, ϕ_{V}, θ_{V} > 0$ , the size of $U$ and $V$ are $M \times H$ and $H \times N$ , and $(U V)_{i j}$ is the $(i, j)$ entry of $U V$ , respectively.

Then, the variational free energy ${¯ F}_{n}$ satisfies the following asymptotic equality:

{¯ F}_{n} = n S_{n} + λ_{v b} log n + O_{p} (1) (n \to \infty),

where

λ_{v b} = {\begin{matrix} (H - H_{0}) (M ϕ_{U} + N ϕ_{V}) + \frac{1}{2} H_{0} (M + N), & if M ϕ_{U} + N ϕ_{V} < \frac{M + N}{2} \frac{1}{2} H (M + N), & otherwise . \end{matrix}

In this paper, we mathematically show an upper bound $¯ λ$ of the RLCT $λ$ of the NMF in the case that is same as VBNMF; the model is Poisson ditributions and the prior is gamma distributions; the prior may have zero or divergence points in $K^{- 1} (0)$ . By using the upper bound, we also derive a lower bound of the approximation accuracy of VBNMF.

3 Main Theorems and Proof

In this section, we introduce our main results and prove them.

3.1 Main Theorems

Third, we introduce the main result of this paper. In the followings, $w = (U, V)$ is a parameter and $x = X$ is an observed random variable.

Let $M (M, N, C)$ be a set of $M \times N$ matrices whose elements are in $C$ , where $C$ is a subset of $R$ . Let $K$ be a compact subset of $R_{≧ 0} = {x \in R | x ≧ 0}$ and let $K_{0}$ be a compact of subset $R_{> 0} = {x \in R | x > 0}$ . We denote that $U \in M (M, H, K)$ , $V \in M (H, N, K)$ and $U_{0} \in M (M, H_{0}, K_{0})$ , $V_{0} \in M (H_{0}, N, K_{0})$ are the NMF result of $U_{0} V_{0}$ such that $H_{0}$ is the non-negative rank [5] of $U_{0} V_{0}$ , i.e. they give the minimal $H_{0}$ , where $H ≧ H_{0}$ and ${(x, y, a, b) \in K \times K_{0} | x y = a b} \neq \emptyset$ .

Definition 3.1 (RLCT of NMF).

Assume that the largest pole of the function of one complex variable $z$ ,

ζ (z) = \int_{M (M, H, K)} d U \int_{M (H, N, K)} d V (∥ U V - U_{0} V_{0} ∥^{2})^{z} φ (U, V)

is equal to $(- λ)$ . Then $λ$ is said to be the RLCT of NMF.

We have already derived an upper bound of the RLCT of NMF in the case that the prior $φ (U, V)$ is strictly positive and bounded [9, 8]. However, to compare with VBNMF, we set the prior as gamma distributions:

φ (U, V) = G a m (U | ϕ_{U}, θ_{U}) G a m (V | ϕ_{V}, θ_{U}),

where

	$G a m (U \| ϕ_{U}, θ_{U}) = M \prod i = 1 H \prod k = 1 \frac{θ_{U}^{ϕ_{U}}}{Γ (θ_{U})} u_{i k}^{ϕ_{U} - 1} e^{- θ_{U} u_{i k}},$
	$G a m (V \| ϕ_{V}, θ_{V}) = H \prod k = 1 N \prod j = 1 \frac{θ_{V}^{ϕ_{V}}}{Γ (θ_{V})} v_{i k}^{ϕ_{V} - 1} e^{- θ_{V} v_{i k}},$

and $ϕ_{U}, θ_{U}, ϕ_{V}, θ_{V} > 0$ .

In this paper, we prove the following theorems.

Theorem 3.1.

If the prior is the above gamma distributions, then the RLCT of NMF $λ$ satisfies the following inequality:

λ ≦ \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] .

The equality holds when $M = N = 1$ .

We prove this theorem in the next section. As two applications of this theorem, we obtain an upper bounds of the free energy and Bayesian generalization error of NMF in this case. The following theorem shows a statistical bound of Bayesian estimation of NMF.

Theorem 3.2.

Let the probability density functions of $X \in M (M, N, K)$ be $q (X)$ and $p (X | U, V)$ , which represent a true distribution and a learning machine respectively defined by

	$q (X)$	$= P o i (X \| U_{0} V_{0}),$
	$p (X \| U, V)$	$= P o i (X \| U V),$

where

P o i (X | A) = M \prod i = 1 N \prod j = 1 \frac{(a_{i j})^{x_{i j}}}{x_{i j}!} e^{- a_{i j}}, X = (x_{i j})_{i = 1, j = 1}^{M, N}, A = (a_{i j})_{i = 1, j = 1}^{M, N} .

Also let $φ (U, V) = G a m (U | ϕ_{U}, θ_{U}) G a m (V | ϕ_{V}, θ_{U})$ . Then, the free energy $F_{n}$ and the expected generalization error $E [G_{n}]$ satisfies the following inequality:

	$F_{n} ≦ n S_{n} + \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] log n + O_{p} (1),$
	$E [G_{n}] ≦ \frac{1}{2 n} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] + o (\frac{1}{n}) .$

In Theorem 3.2, we study a case when a set of random matrices $X^{n} = X_{1}, X_{2}, . . ., X_{n}$ are observed and the true decomposition $U_{0}$ and $V_{0}$ are statistically estimated. Actually sometimes NMF has studied in the case when only one target matrix is decomposed, however, in general, decomposition of a set of independent matrices should be studied because target matrices are often obtained daily, monthly, or different places [12]. In such cases, decomposition of a set of matrices results in statistical inference. We consider this situation and it is common to [13] and Theorem 2.1. A statistical model $p (X | U, V)$ which has parameters $(U, V)$ are employed for estimation. Then the free energy and generalization error of Bayesian estimation is given by this theorem.

Theorem 3.3.

Let the variational free energy of VBNMF be ${¯ F}_{n}$ . Then, the following inequality is attained:

{¯ F}_{n} - F_{n} ≧ ~ λ log n + O_{p} (1),

where

~ λ = {\begin{matrix} \frac{1}{2} [(H - H_{0}) (M ϕ_{U} + N ϕ_{V} + max {M ϕ_{U}, N ϕ_{V}}) + H_{0}], & if M ϕ_{U} + N ϕ_{V} < \frac{M + N}{2} \frac{1}{2} [(H - H_{0}) (M + N - min {M ϕ_{U}, N ϕ_{V}}) + H_{0}], & otherwise% . \end{matrix}

Theorem 3.3 gives a lower bound of the difference of the free energy between the variational approximation and the true.

3.2 Proof of Main Theorems

In orfer to prove Theorem 3.1, we use the following three lemmas which are proved in Appendix A.

Lemma 3.1.

Let $λ$ be the absolute of the maximum pole of

ζ (z) = \iint d U d V {(∥ U V ∥^{2})}^{z} G a m (U | ϕ_{U}, θ_{U}) G a m (V | ϕ_{V}, θ_{V}) .

Then,

λ = \frac{H min {M ϕ_{U}, N ϕ_{V}}}{2}

holds; this is the equality of Theorem 3.1 in the case $H_{0} = 0$ .

Lemma 3.2.

If $H_{0} = H = 1$ , the equal sign of the Theorem3.1 holds:

λ = \frac{M + N - 1}{2} .

Lemma 3.3.

If $H = H_{0}$ , the Theorem3.1 is attained:

λ ≦ \frac{H_{0} (M + N - 1)}{2} .

Let the entries of the matrices $(U, V)$ be

U = (u_{1}, \dots, u_{H}), u_{k} = (u_{i k})_{i = 1}^{M},

V = (v_{1}, \dots, v_{H})^{T}, v_{k} = (v_{k j})_{j = 1}^{N},

and the ones of $(U_{0}, V_{0})$ be

U_{0} = (u_{1}^{0}, \dots, u_{H_{0}}^{0}), u_{k}^{0} = (u_{i k}^{0})_{i = 1}^{M},

V_{0} = (v_{1}^{0}, \dots, v_{H_{0}}^{0})^{T}, v_{k}^{0} = (v_{k j}^{0})_{j = 1}^{N},

respectively.

Now, we prove Theorem3.1 using the above lemmas.

Proof of Theorem3.1.

Let $K$ and $L$ be non-negative analytic functions on a finite dimensional Euclid space. A binomial relation $\sim$ is defined by

K (w) \sim L (w) \Leftrightarrow_{d e f} the RLCT of K (w) is equal to the one% of L (w) .

We have

			$∥ U V - U_{0} V_{0} ∥^{2}$
		$=$	$M \sum i = 1 N \sum j = 1 (u_{i 1} v_{1 j} + . . . + u_{i H} v_{H j} - u_{i 1}^{0} v_{1 j}^{0} - . . . - u_{i H_{0}}^{0} v_{H_{0} j}^{0})^{2}$
		$=$	$M \sum i = 1 N \sum j = 1 (H \sum k = 1 u_{i k} v_{k j} - H_{0} \sum k = 1 u_{i k}^{0} v_{k j}^{0})^{2}$
		$=$	$M \sum i = 1 N \sum j = 1 (H_{0} \sum k = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0}) + H \sum k = H_{0} + 1 u_{i k} v_{k j})^{2}$
		$≦$	$C M \sum i = 1 N \sum j = 1 ⎛ ⎝ H_{0} \sum k = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0})^{2} + H \sum k = H_{0} + 1 u_{i k}^{2} v_{k j}^{2} ⎞ ⎠ f o r \exists C > 0 (c o n s t .)$
		$\sim$	$M \sum i = 1 N \sum j = 1 ⎛ ⎝ H_{0} \sum k = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0})^{2} + H \sum k = H_{0} + 1 u_{i k}^{2} v_{k j}^{2} ⎞ ⎠$
		$=$	$H_{0} \sum k = 1 (M \sum i = 1 N \sum j = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0})^{2}) + H \sum k = H_{0} + 1 (M \sum i = 1 N \sum j = 1 u_{i k}^{2} v_{k j}^{2})$
		$=$	$H_{0} \sum k = 1 {∥ ∥ u_{k} (v_{k})^{T} - u_{k}^{0} (v_{k}^{0})^{T} ∥ ∥}_{k}^{2}$
			$+ {∥ ∥ ∥ ∥ ∥ ∥ ⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} u_{1 (H_{0} + 1)} & \dots & u_{1 H} ⋮ & ⋱ & ⋮ u_{M (H_{0} + 1)} & \dots & u_{M H} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠ ⎛ ⎜ ⎜ ⎝ \begin{matrix} v_{(H_{0} + 1) 1} & \dots & v_{(H_{0} + 1) N} ⋮ & ⋱ & ⋮ v_{H 1} & \dots & v_{H N} \end{matrix} ⎞ ⎟ ⎟ ⎠ ∥ ∥ ∥ ∥ ∥ ∥}^{2} .$

Put the first and second terms of above

and

K_{2} = {∥ ∥ ∥ ∥ ∥ ∥ ⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} u_{1 (H_{0} + 1)} & \dots & u_{1 H} ⋮ & ⋱ & ⋮ u_{M (H_{0} + 1)} & \dots & u_{M H} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠ ⎛ ⎜ ⎜ ⎝ \begin{matrix} v_{(H_{0} + 1) 1} & \dots & v_{(H_{0} + 1) N} ⋮ & ⋱ & ⋮ v_{H 1} & \dots & v_{H N} \end{matrix} ⎞ ⎟ ⎟ ⎠ ∥ ∥ ∥ ∥ ∥ ∥}^{2},

respectively. Because of the prior $φ (U, V) ≧ 0$ , all we have to do is find the RLCT of

K_{1} φ (U, V) + K_{2} φ (U, V)

to derive an upper bound.

Let $¯ λ$ be an RLCT of the right-most side, $λ_{1}$ be an RLCT of first term in the right-most side, and $λ_{2}$ be an RLCT of the second one. If $φ (U, V) = 0$ then $K_{1} \neq 0$ thus the first prior term can be ignored to calculate the RLCT; we have

K_{1} + K_{2} φ (U, V)

Because of that variables are independent and RLCTs are order isomorphic,

λ ≦ ¯ λ = λ_{1} + λ_{2} .

(3)

Since ‘ $K_{1}$ corresponds to the proof of Lemma 3.3 in the case of $H \leftarrow H_{0}$ ,

λ_{1} = H_{0} \frac{M + N - 1}{2} .

In contrast, $K_{2}$ corresponds to Lemma 3.1 in the case of $H \leftarrow H - H_{0}$ . That causes

λ_{2} = \frac{(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}}}{2} .

Using the above inequality and equality for inequality(3),

$λ$	$≦$	$~ λ$
	$=$	$λ_{1} + λ_{2}$
	$=$	$H_{0} \frac{M + N - 1}{2} + \frac{(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}}}{} 2 .$
$∴ λ$	$≦$	$\frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] .$

∎

Remark 3.1.

Under the same assumption of the Theorem 3.1, suppose

f_{i j}^{k} = {\begin{matrix} u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0} & (k \in {1, \dots, H_{0}}) u_{i k} v_{k j} & (k \in {H_{0} + 1, \dots, H}) \end{matrix} .

If $f_{i j}^{k} ≧ 0$ , i.e. , $u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0} ≧ 0 (k \in {1, \dots, H_{0}})$ ,

λ = \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] .

Proof.

Owing to $f_{i j}^{k} ≧ 0$ ,

H \sum k = 1 (f_{i j}^{k})^{2} ≦ {(H \sum k = 1 f_{i j}^{k})}^{2} ≦ H H \sum k = 1 (f_{i j}^{k})^{2} .

Thus,

H \sum k = 1 (f_{i j}^{k})^{2} \sim {(H \sum k = 1 f_{i j}^{k})}^{2} .

Using the above relation,

			$∥ U V - U_{0} V_{0} ∥^{2}$
		$=$	$M \sum i = 1 N \sum j = 1 (H_{0} \sum k = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0}) + H \sum k = H_{0} + 1 u_{i k} v_{k j})^{2}$
		$=$	$M \sum i = 1 N \sum j = 1 (H \sum k = 1 f_{i j}^{k})^{2}$
		$\sim$	$M \sum i = 1 N \sum j = 1 (H \sum k = 1 (f_{i j}^{k})^{2})$
		$=$	$M \sum i = 1 N \sum j = 1 ⎛ ⎝ H_{0} \sum k = 1 (u_{i k} v_{k j} - u_{i k}^{0} v_{k j}^{0})^{2} + H \sum k = H_{0} + 1 u_{i k}^{2} v_{k j}^{2} ⎞ ⎠$
		$=$	$H_{0} \sum k = 1 {∥ ∥ u_{k} (v_{k})^{T} - u_{k}^{0} (v_{k}^{0})^{T} ∥ ∥}_{k}^{2} + {∥ ∥ ∥ ∥ ∥ ∥ ⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} u_{1 (H_{0} + 1)} & \dots & u_{1 H} ⋮ & ⋱ & ⋮ u_{M (H_{0} + 1)} & \dots & u_{M H} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠ ⎛ ⎜ ⎜ ⎝ \begin{matrix} v_{(H_{0} + 1) 1} & \dots & v_{(H_{0} + 1) N} ⋮ & ⋱ & ⋮ v_{H 1} & \dots & v_{H N} \end{matrix} ⎞ ⎟ ⎟ ⎠ ∥ ∥ ∥ ∥ ∥ ∥}^{2} .$

∴ λ = \frac{1}{2} [(H - H_{0}) min {M, N} + H_{0} (M + N - 1)] .

∎

Second, Theorem 3.2 and 3.3 are derived by using Theorem 3.1.

Proof of Theorem 3.2 and 3.3.

First, we prove Theorem 3.2. Owing to the equality (1) and (2), we have

	$F_{n} = n S_{n} + λ log n - (m - 1) log log n + O_{p} (1),$
	$E [G_{n}] = \frac{λ}{n} + o (\frac{1}{n}) .$

On account of Theorem 3.1,

λ ≦ \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] .

holds. From the above three formulas and $m ≧ 1$ , Theorem 3.2 is attained:

	$F_{n} ≦ n S_{n} + \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] log n + O_{p} (1),$
	$E [G_{n}] ≦ \frac{1}{2 n} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)] + o (\frac{1}{n}) .$

Next, we show Theorem 3.3. Let $¯ λ$ be the upper bound of $λ$ in Theorem 3.1:

¯ λ = \frac{1}{2} [(H - H_{0}) min {M ϕ_{U}, N ϕ_{V}} + H_{0} (M + N - 1)]

In the same way as the above, we have

F_{n} ≦ n S_{n} + ¯ λ log n + O_{p} (1) .

Also, because of Theorem 2.1,

{¯ F}_{n} = n S_{n} + λ_{v b} log n + O_{p} (1)

holds, where

λ_{v b} = {\begin{matrix} (H - H_{0}) (M \end{matrix}

	$K L (ψ_{v b} ∥ ψ)$	$= \int ψ_{v b} (w \| X^{n}) log \frac{ψ_{v b} (w \| X^{n})}{ψ (w \| X^{n})} d w$
		$= \int ψ_{v b} (w \| X^{n}) log \frac{ψ_{v b} (w \| X^{n})}{P (X^{n} \| w) φ (w) / Z_{n} (X^{n})} d w$
		$= \int ψ_{v b} (w \| X^{n}) log (ψ_{v b} (w \| X^{n}) - P (X^{n} \| w) φ (w)) d w$
		$+ \int ψ_{v b} (w \| X^{n}) log Z_{n} (X^{n}) d w$
		$= \int ψ_{v b} (w \| X^{n}) log (ψ_{v b} (w \| X^{n}) - P (X^{n} \| w) φ (w)) d w + log Z_{n} (X^{n}),$

	${¯ F}_{n}$	$= min ψ_{v b} = ψ_{v b}^{a} ψ_{v b}^{b} {\int ψ_{v b}^{a} (w_{a} \| X^{n}) log ψ_{v b}^{a} (w_{a} \| X^{n}) d w_{a}$
		$+ \int ψ_{v b}^{b} (w_{b} \| X^{n}) log ψ_{v b}^{b} (w_{b} \| X^{n}) d w_{b}$
		$+ \iint ψ_{v b}^{a} (w_{a} \| X^{n}) ψ_{v b}^{b} (w_{b} \| X^{n}) log (P (X^{n} \| w) φ (w)) d w_{a} d w_{b}} .$

Variational Approximation Accuracy in Bayesian Non-negative Matrix Factorization

Rapid Posterior Exploration in Bayesian Non-negative Matrix Factorization

Beta Process Non-negative Matrix Factorization with Stochastic Structured Mean-Field Variational Inference

Asymptotic Bayesian Generalization Error in a General Stochastic Matrix Factorization for Markov Chain and Bayesian Network

Fast Bayesian Non-Negative Matrix Factorisation and Tri-Factorisation

A sampling algorithm to compute the set of feasible solutions for non-negative matrix factorization with an arbitrary rank

A Comparative Study of Temporal Non-Negative Matrix Factorization with Gamma Markov Chains

A Variational Autoencoder for Probabilistic Non-Negative Matrix Factorisation

1 Introduction

1.1 Algorithms for Non-negative Matrix Factorization

1.2 Learning Theory of Bayesian and Variational Inference

2 Inference Frameworks

2.1 Framework of Bayesian Inference

2.2 Variational Bayesian Algorithm

Theorem 2.1 (Kohjima).

3 Main Theorems and Proof

3.1 Main Theorems

Definition 3.1 (RLCT of NMF).

Theorem 3.1.

Theorem 3.2.

Theorem 3.3.

3.2 Proof of Main Theorems

Lemma 3.1.

Lemma 3.2.

Lemma 3.3.

Proof of Theorem3.1.

Remark 3.1.

Proof.

Proof of Theorem 3.2 and 3.3.

Variational Approximation Accuracy in Bayesian Non-negative Matrix Factorization

Related Research

Rapid Posterior Exploration in Bayesian Non-negative Matrix Factorization

Beta Process Non-negative Matrix Factorization with Stochastic Structured Mean-Field Variational Inference

Asymptotic Bayesian Generalization Error in a General Stochastic Matrix Factorization for Markov Chain and Bayesian Network

Fast Bayesian Non-Negative Matrix Factorisation and Tri-Factorisation

A sampling algorithm to compute the set of feasible solutions for non-negative matrix factorization with an arbitrary rank

A Comparative Study of Temporal Non-Negative Matrix Factorization with Gamma Markov Chains

A Variational Autoencoder for Probabilistic Non-Negative Matrix Factorisation

1 Introduction

1.1 Algorithms for Non-negative Matrix Factorization

1.2 Learning Theory of Bayesian and Variational Inference

2 Inference Frameworks

2.1 Framework of Bayesian Inference

2.2 Variational Bayesian Algorithm

Theorem 2.1 (Kohjima).

3 Main Theorems and Proof

3.1 Main Theorems

Definition 3.1 (RLCT of NMF).

Theorem 3.1.

Theorem 3.2.

Theorem 3.3.

3.2 Proof of Main Theorems

Lemma 3.1.

Lemma 3.2.

Lemma 3.3.

Proof of Theorem3.1.

Remark 3.1.

Proof.

Proof of Theorem 3.2 and 3.3.