Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences

1 Introduction

Randomized sequential algorithms abound in machine learning and optimization. The most famous is the stochastic gradient method (see Bottou, 1998; Bertsekas, 2012; Nemirovski et al., 2009; Shalev-Shwartz and Srebro, 2008), but other popular methods include algorithms for alternating projections (see Strohmer and Vershynin, 2009; Leventhal and Lewis, 2010), proximal point methods (see Bertsekas, 2011), coordinate descent (see Nesterov, 2010) and derivative free optimization (see Nesterov, 2011; Nemirovski and Yudin, 1983). In all of these cases, an iterative procedure is derived where, at each iteration, an independent sample from some distribution determines the action at the next stage. This sample is selected with-replacement from a pool of possible options.

In implementations of many of these methods, however, practitioners often choose to break the independence assumption. For instance, in stochastic gradient descent, many implementations pass through each item exactly once in a random order (i.e., according to a random permutation). In randomized coordinate descent, one can cycle over the coordinates in a random order. These strategies, employing without-replacement sampling, are often easier to implement efficiently, guarantee that every item in the data set is touched at least once, and often have better empirical performance than their with-replacement counterparts (see Bottou, 2009; Recht and Ré, 2011; Feng et al., 2012).

Unfortunately, the analysis of without-replacement sampling schemes are quite difficult. The independence assumption underlying with-replacement sampling provides an elegant Markovian framework for analyzing incremental algorithms. The iterates in without-replacement sampling are correlated, and studying them requires sophisticated probabilistic tools. Consequently, most of the analyses without-replacement optimization assume that the iterations are assigned deterministically. Such deterministic orders might incur exponentially worse convergence rates than randomized methods (Nedic and Bertsekas, 2000)

, and deterministic orders still require careful estimation of accumulated errors

(see Luo, 1991; Tseng, 1998). The goal of this paper is to make progress towards patching the discrepancy between theory and practice of without-replacement sampling in randomized algorithms.

In particular, in many cases, we demonstrate that without-replacement sampling outperforms with-replacement sampling provided a noncommutative version of the arithmetic-geometric mean inequality holds. Namely, if $A_{1}, \dots, A_{n}$ are a collection of $d \times d$ positive semidefinite matrices, we define the arithmetic and (symmetrized) geometric means to be

M_{A} := \frac{1}{n} n \sum i = 1 A_{i}, and M_{G} := \frac{1}{n!} \sum σ \in S_{n} A_{σ (1)} \times \dots \times A_{σ (n)}

where $S_{n}$ denotes the group of permutations. Our conjecture is that the norm of $M_{G}$ is always less than the norm of $(M_{A})^{n}$ . Assuming this inequality, we show that without-replacement sampling leads to faster convergence for both the least mean squares and randomized Kaczmarz algorithms of Strohmer and Vershynin (2009).

Using established work in matrix analysis, we show that these noncommutative arithmetic-geometric mean inequalities hold when there are only two matrices in the pool. We also prove that the inequality is true when all of the matrices commute. We demonstrate that if we don’t symmetrize, there are deterministically ordered products of $n$ matrices whose norm exceeds $∥ M_{A} ∥^{n}$ by an exponential factor. That is, symmetrization is necessary for the noncommutative arithmetic-geometric mean inequality to hold.

While we are unable to prove the noncommutative arithmetic-geometric mean inequality in full generality, we verify that it holds for many classes of random matrices. Random matrices are, in some sense, the most interesting case for machine learning applications. This is particularly evident in applications such as empirical risk minimization and online learning where the the data are conventionally assumed to be generated by some i.i.d random process. In Section 4, we show that if $A_{1}, \dots, A_{n}$ are generated i.i.d. from certain distributions, then the noncommutative arithmetic-geometric mean inequality holds in expectation with respect to the $A_{i}$ . Section 4.1 assumes that $A_{i} = Z_{i} Z_{i}^{T}$ w.b.here $Z_{i}$ have independent entries, identically sampled from some symmetric distribution. In Section 4.2, we analyze the random matrices that commonly arise in stochastic gradient descent and related algorithms, again proving that without-replacement sampling exhibits faster convergence than with-replacement sampling. We close with a discussion of other open conjectures that could impact machine learning theory, algorithms, and software.

2 Sampling in incremental gradient descent

To illustrate how with- and without-replacement sampling methods differ in randomized optimization algorithms, we focus on one core algorithm, the Incremental Gradient Method (IGM). Recall that the IGM minimizes the function

minimize x f (x) = n \sum i = 1 f_{i} (x)

(2.1)

via the iteration

x_{k} = x_{k - 1} - γ_{k} \nabla f_{i_{k}} (x_{k - 1}) .

(2.2)

Here, $x_{0}$

is an initial starting vector,

γ_{k}

are a sequence of nonnegative step sizes, and the indices

i_{k}

are chosen using some (possibly deterministic) sampling scheme. When

f

is strongly convex, the IGM iteration converges to a near-optimal solution of (2.1) for any

x_{0}

under a variety of step-sizes protocols and sampling schemes including constant and diminishing step-sizes (see Anstreicher and Wolsey, 2000; Bertsekas, 2012; Nemirovski et al., 2009). When the increments are selected uniformly at random at each iteration, IGM is equivalent to stochastic gradient descent. We use the term IGM here to emphasize that we are studying many possible orderings of the increments. In the next examples, we study the specialized case where the

f_{i}

are quadratic and the IGM is equivalent to the least mean squares algorithm of Widrow and Hoff (1960).

2.1 One-dimensional Examples

First consider the following toy one-dimensional least-squares problem

minimize x \frac{1}{2} n \sum i = 1 (x - y_{i})^{2} .

(2.3)

where $y_{i}$ is a sequence of scalars with mean $μ_{y}$

and variance

σ^{2}

. Applying (2.2) to (2.3) results in the iteration.

x_{k} = x_{k - 1} - γ_{k} (x_{k - 1} - y_{i_{k}})

If we initialize the method with $x_{0} = 0$ and take $n$ steps of incremental gradient with stepsize $γ_{k} = 1 / k$ , we have

x_{n} = \frac{1}{n} n \sum j = 1 y_{i_{j}}

where $i_{j}$ is the index drawn at iteration $j$ . If the steps are chosen using a without-replacement sampling scheme, $x_{n} = μ_{y}$ , the global minimum. On the other hand, using with-replacement sampling, we will have

E [(x_{n} - μ_{y})^{2}] = \frac{σ^{2}}{n},

which is a positive mean square error.

Another toy example that further illustrates the discrepancy is the least-squares problem

minimize x \frac{1}{2} n \sum i = 1 β_{i} (x - y)^{2}

where $β_{i}$ are positive weights. Here, $y$ is a scalar, and the global minimum is clearly $y$ . Let’s consider the incremental gradient method with constant stepsize $γ_{k} = γ < min β_{i}^{- 1}$ . Then after $n$ iterations we will have

| x_{n} - y | = | y | n \prod j = 1 (1 - γ β_{i_{j}})

If we perform without-replacement sampling, this error is given by

| x_{n} - y | = | y | n \prod i = 1 (1 - γ β_{i}) .

On the other hand, using with-replacement sampling yields

E [| x_{n} - y |] = | y | {(1 - \frac{γ}{n} n \sum i = 1 β_{i})}_{i}^{n} .

By the arithmetic-geometric mean inequality, we then have that the without-replacement sample is always closer to the optimal value in expectation. This sort of discrepancy is not simply a feature of these toy examples. We now demonstrate that similar behavior arises in multi-dimensional examples.

2.2 IGM in more than one dimension

Now consider IGM in higher dimensions. Let $x_{⋆}$ be a vector in $R^{d}$ and set

y_{i} = a_{i}^{T} x_{⋆} + ω_{i} for i = 1, \dots, n

where $a_{i} \in R^{d}$ are some test vectors and $ω_{i}$

are i.i.d. Gaussian random variables with mean zero and variance

ρ^{2}

We want to compare with- vs without-replacement sampling for IGD on the cost function

minimize x n \sum i = 1 (a_{i}^{T} x - y_{i})^{2} .

(2.4)

Suppose we walk over $k$ steps of IGD with constant stepsize $γ$ and we access the terms $i_{1}, \dots, i_{k}$ in that order. Then we have

x_{k} = x_{k - 1} - γ a_{i_{k}} (a_{i_{k}}^{T} x_{i_{k - 1}} - y_{i_{k}}) = (I - γ a_{i_{k}} a_{i_{k}}^{T}) x_{i_{k - 1}} + γ a_{i_{k}} y_{i_{k}} .

Subtracting $x_{⋆}$ from both sides of this equation then gives

	$x_{k} - x_{⋆}$	$= (I - γ a_{i_{k}} a_{i_{k}}^{T}) (x_{k - 1} - x_{⋆}) + γ a_{i_{k}} ω_{i_{k}}$		(2.5)
		$= k \prod j = 1 (I - γ a_{i_{j}} a_{i_{j}}^{T}) (x_{0} - x_{⋆}) + k \sum ℓ = 1 \prod k \geq j > ℓ (I - γ a_{i_{j}} a_{i_{j}}^{T}) γ a_{i_{ℓ}} ω_{i_{ℓ}} .$

Here, the product notation means we multiply by the matrix with smallest index first, then left multiply by the matrix with the next index and so on up to the largest index.

Our goal is to estimate the risk after $k$ steps, namely $E [∥ x_{k} - x_{⋆} ∥^{2}]$ , and demonstrate that this error is smaller for the without-replacement model. The expectation is with respect to the IGM ordering and the noise sequence $ω_{i}$ . To simplify things a bit, we take a partial expectation with respect to $ω_{i}$ :

(2.6)

In this case, we need to compare the expected value of matrix products under with or without-replacement sampling schemes in order to conclude which is better. Is there a simple conjecture, analogous to the arithmetic-geometric mean inequality, that would guarantee without-replacement sampling is always better?

2.3 The Randomized Kaczmarz algorithm

As another high-dimensional example in the same spirit, we consider the randomized Kaczmarz algorithm of Strohmer and Vershynin (2009). The Kaczmarz algorithm is used to solve the over-determined linear system $Φ x = y$ . Here $Φ$ is an $n \times d$ matrix with $n > d$ and we assume there exists an exact solution $x_{⋆}$ satisfying $Φ x_{⋆} = y$ . Kaczmarz’s method solves this system by alternating projections (Kaczmarz, 1937) and was implemented in the earliest medical scanning devices (Hounsfield, 1973). In computer tomography, this method is called the Algebraic Reconstruction Technique (Herman, 1980; Natterer, 1986) or Projection onto Convex Sets (Sezan and Stark, 1987).

Kaczmarz’s algorithm consists of iterations of the form

x_{k + 1} = x_{k} + \frac{y_{i} - ϕ_{i_{k}}^{T} x_{k}}{∥ ϕ_{i_{k}} ∥^{2}} ϕ_{i_{k}} .

(2.7)

where the rows of $Φ$ are accessed in some deterministic order. This sequence can be interpreted as an incremental variant of Newton’s method on the least squares cost function

minimize x n \sum i = 1 (ϕ_{i}^{T} x_{k} - y_{i})^{2}

with step size equal to $1$ (see Bertsekas, 1999).

Establishing the convergence rate of this method proved difficult in imaging science. On the other hand, Strohmer and Vershynin (2009)

proposed a randomized variant of the Kaczmarz method, choosing the next iterate with-replacement with probability proportional to the norm of

ϕ_{i}

. Strohmer and Vershynin established linear convergence rates for their iterative scheme. Expanding out (2.7) for

k

iterations, we see that

x_{k} - x_{⋆} = k \prod j = 1 ⎛ ⎝ I - \frac{ϕ_{i_{j}} ϕ_{i_{j}}^{T}}{∥ ϕ_{i_{j}} ∥^{2}} ⎞ ⎠ (x_{0} - x_{⋆}) .

Let us suppose that we modify Strohmer and Vershynin’s procedure to employ without-replacement sampling. After $k$ steps is the with-replacement or without-replacement model closer to the optimal solution?

3 Conjectures concerning the norm of geometric and arithmetic means of positive definite matrices

To formulate a sufficient conjecture which would guarantee that without-replacement sampling outperforms with-replacement, let us first formalize some notation. Throughout, $[n]$ denotes the set of integers from $1$ to $n$ . Let $D$ be some domain, $f : D^{k} \to R$ , and $(x_{1}, \dots, x_{n})$ a set of $n$ elements from $D$ . We define the without-replacement expectation as

E_{w o} [f (x_{i_{1}}, \dots, x_{i_{k}})] = \frac{(n - k)!}{n!} \sum j_{1} \neq j_{2} \neq \dots \neq j_{k} f (x_{j_{1}}, \dots, x_{j_{k}}) .

That is, we average the value of $f$ over all ordered tuples of elements from $(x_{1}, \dots, x_{n})$ . Similarly, the with-replacement expectation is defined as

E_{w r} [f (x_{i_{1}}, \dots, x_{i_{k}})] = n^{- k} n \sum (j_{1}, \dots, j_{k}) = 1 f (x_{j_{1}}, \dots, x_{j_{k}}) .

With these conventions, we can list our main conjectures as follows:

Conjecture 3.1 (Operator Inequality of Noncommutative Arithmetic and Geometric Means)

Let $A_{1}, \dots, A_{n}$ be a collection of positive semidefinite matrices. Then we conjecture that the following two inequalities always hold:

	$∥ ∥ ∥ ∥ E_{w o} [k \prod j = 1 A_{i_{j}}] ∥ ∥ ∥ ∥$	$\leq ∥ ∥ ∥ ∥ E_{w r} [k \prod j = 1 A_{i_{j}}] ∥ ∥ ∥ ∥$		(3.1)
	$∥ ∥ ∥ ∥ E_{w o} [k \prod j = 1 A_{i_{k - j + 1}} k \prod j = 1 A_{i_{j}}] ∥ ∥ ∥ ∥$	$\leq ∥ ∥ ∥ ∥ E_{w r} [k \prod j = 1 A_{i_{k - j + 1}} k \prod j = 1 A_{i_{j}}] ∥ ∥ ∥ ∥$		(3.2)

Note that in (3.1), we have $E_{w r} [\prod_{j} A_{i_{j}}] = (\frac{1}{n} \sum_{i} A_{i})^{k} = (M_{A})^{k}$ .

Assuming this conjecture holds, let us return to the analysis of the IGM (2.6). Assuming that $x_{0} - x_{⋆}$ is an arbitrary starting vector and that (3.2) holds, we have that each term in this summation is smaller for the without-replacement sampling model than for the with-replacement sampling model. In turn, we expect the without-replacement sampling implementation will return lower risk after one pass over the data-set. Similarly, for the randomized Kaczmarz iteration (2.7), Conjecture 3.1 implies that a without-replacement sample will have lower error after $k < n$ iterations.

In the remainder of this document we provide several case studies illustrating that these noncommutative variants of the arithmetic-geometric mean inequality hold in a variety of settings, establishing along the way tools and techniques that may be useful for proving Conjecture 3.1 in full generality.

3.1 Two matrices and a search for the geometric mean

Both of the inequalities (3.1) and (3.2) are true when $n = 2$ . These inequalities all follow from an well-estabilished line of research in estimating the norms of products of matrices, started by the seminal work of Bhatia and Kittaneh (1990).

Proposition 3.2

Both (3.1) and (3.2) hold when $n = 2$

Proof Let $A$ and $B$ be positive definite matrices. Both of our arithmetic-geometric mean inequalities follow from the stronger inequality

∥ A B ∥ \leq {∥ ∥ \frac{1}{2} A + \frac{1}{2} B ∥ ∥}^{2} .

(3.3)

This bound was proven by Bhatia and Kittaneh (2000). In particular, since

∥ \frac{1}{2} A B + \frac{1}{2} B A ∥ \leq ∥ A B ∥,

(3.1) is immediate. For (3.2), note that

E_{w o} [A_{i_{1}} A_{i_{2}}^{2} A_{i_{1}}] = \frac{1}{2} A B^{2} A + \frac{1}{2} B A^{2} B

(3.4)

and

E_{w r} [A_{i_{1}} A_{i_{2}}^{2} A_{i_{1}}] = \frac{1}{4} A^{4} + \frac{1}{4} A B^{2} A + \frac{1}{4} B A^{2} B + \frac{1}{4} B^{4} .

We can bound (3.4) by

\frac{1}{2} ∥ A B^{2} A + B A^{2} B ∥ \leq ∥ A B^{2} A ∥ = ∥ A B ∥^{2} \leq ∥ \frac{1}{2} A + \frac{}{1} 2 B ∥^{4} = ∥ (\frac{1}{2} A + \frac{1}{2} B)^{4} ∥ .

Here, the first inequality is the triangle inequality and the subsequent equality follows because the norm of $X^{T} X$ is equal to the squared norm of $X$ . The second inequality is (3.3).

To complete the proof we show

X_{L} := (\frac{1}{2} A + \frac{1}{2} B)^{4} ⪯ \frac{1}{4} A^{4} + \frac{1}{4} A B^{2} A + \frac{1}{4} B A^{2} B + \frac{1}{4} B^{4} := X_{R}

in the semidefinite ordering. But this follows by observing

X_{R} - X_{L} = \sum p \in [2]^{2} \sum q \in [2]^{2} Q (p, q) A_{p (2)} A_{p (1)} A_{q (1)} A_{q (2)}

(3.5)

where $Q (p, q) = 3 / 16$ if $p = q$ and $- 1 / 16$ otherwise. Since $p$ and $q$ both take $4$ possible values, the matrix $Q$ is positive definite which means that $X_{R} - X_{L}$ can be written as a nonnegative sum of products of the form $Y Y^{T}$ . We conclude that $X_{R} - X_{L}$ must be positive define and hence $∥ X_{R} ∥ \geq ∥ X_{L} ∥$ , completing the proof¹¹1An explicit decomposition (3.5) into Hermitian squares was initially found using the software NCSOSTools by Cafuta et al. (2011). This software finds decompositions of matrix polynomials into sums of Hermitian squares. Our argument was constructed after discovering this decomposition..

Note that this proposition actually verifies a stronger statement: for two matrices, the arithmetic-geometric mean inequality holds for deterministic orderings of two matrices. We will discuss below how symmetrization is necessary for more than two matrices. In fact, considerably stronger inequalities hold for symmetrized products of two matrices. As a striking example, the symmetrized geometric mean actually precedes the square of the arithmetic mean in the positive definite order. Let $A$ and $B$ be positive semidefinite. Then we have

{(\frac{1}{2} A + \frac{1}{2} B)}^{2} - (\frac{1}{2} A B + \frac{1}{2} B A) = \frac{1}{4} A^{2} + \frac{1}{4} B^{2} - \frac{1}{4} A B - \frac{1}{4} B A = {(\frac{1}{2} A - \frac{1}{2} B)}^{2} ⪰ 0 .

This ordering breaks for $3$ matrices as evinced by the counterexample

A_{1} = [\begin{matrix} 7 & 0 0 & 0 \end{matrix}], A_{2} = [\begin{matrix} 1 & 1 1 & 1 \end{matrix}], A_{3} = [\begin{matrix} 1 & 1 1 & 1 \end{matrix}] .

The interested reader should consult Bhatia and Kittaneh (2008) for a comprehensive list of inequalities concerning pairs of positive semidefinite matrices.

Unfortunately, these techniques are specialized to the case of two matrices, and no proof currently exists for the inequalities when $n \geq 3$ . There have been a varied set of attempts to extend the noncommutative arithmetic-geometric mean inequalities to more than two matrices. Much of the work in this space has focused on how to properly define the geometric mean of a collection of positive semidefinite matrices. For instance, Ando et al. (2004) demarcate a list of properties desirable by any geometric mean, with one of the properties being that the geometric mean must precede the arithmetic mean in the positive-definite ordering. Ando et al derive a geometric mean satisfying all of these properties, but the resulting mean in no way resembles the means of matrices discussed in this paper. Instead, their geometric mean is defined as a fixed point of a nonlinear map on matrix tuples. Bhatia and Holbrook (2006) and Bonnabel and Sepulchre (2009) propose geometric means based on geodesic flows on the Riemannian manifold of positive definite matrices, however these means also do not correspond to the averaged matrix products that we study in this paper.

3.2 When is it not necessary to symmetrize the order?

When the matrices commute, Conjecture 3.1 is a consequence of the standard arithmetic-geometric mean inequality (more precisely, a consequence of Maclaurin’s inequalities).

Theorem 3.3 (Maclaurin’s Inequalities)

Let $x_{1}, \dots, x_{n}$ be positive scalars. Let

s_{k} = {(\frac{n}{k})}^{- 1} \sum \begin{matrix} Ω \subset [n] | Ω | = k \end{matrix} \prod i \in Ω x_{i}

be the normalized $k$ th symmetric sum. Then we have

s_{1} \geq \sqrt{s_{2}} \geq \dots \geq \sqrt[n - 1]{s_{n - 1}} \geq \sqrt[n]{s_{n}}

Note that $s_{1} \geq \sqrt[n]{s_{n}}$ is the standard form of the arithmetic-geometric mean inequality. See Hardy et al. (1952) for a discussion and proof of this chain of inequalities.

To see that these inequalities immediately imply Conjecture 3.1 when the matrices $A_{i}$ are mutually commutative, note first that when $d = 1$ , we have

E_{w o} [k \prod i = 1 a_{i}] = {(\frac{n}{k})}^{- 1} \sum \begin{matrix} Ω \subset [n] | Ω | = k \end{matrix} \prod i \in Ω a_{i} \leq {(\frac{1}{n} n \sum i = 1 a_{i})}_{i}^{k} = E_{w r} [k \prod i = 1 a_{i}] .

The higher dimensional analogs follow similarly. If all of the $A_{i}$ commute, then the matrices are mutually diagonalizable. That is, we can write $A_{i} = U Λ_{i} U^{T}$ where $U$

is an orthogonal matrix, and the

Λ_{i} = diag (λ_{1}^{(i)}, \dots, λ_{d}^{(i)})

are all diagonal matrices of the eigenvalues of

A_{i}

in descending order. Then we have

∥ ∥ ∥ ∥ E_{w o} [k \prod i = 1 A_{i}] ∥ ∥ ∥ ∥ = ∥ ∥ ∥ ∥ E_{w o} [k \prod i = 1 Λ_{i}] ∥ ∥ ∥ ∥ = E_{w o} k \prod i = 1 λ_{1}^{(i)} \leq E_{w r} k \prod i = 1 λ_{1}^{(i)} = ∥ ∥ ∥ ∥ E_{w r} [k \prod i = 1 A_{i}] ∥ ∥ ∥ ∥

and we also have

∥ ∥ ∥ ∥ E_{w o} [k \prod i = 1 A_{i}] ∥ ∥ ∥ ∥ = E_{w o} [∥ ∥ ∥ ∥ k \prod i = 1 A_{i} ∥ ∥ ∥ ∥] and ∥ ∥ ∥ ∥ E_{w r} [k \prod i = 1 A_{i}] ∥ ∥ ∥ ∥ = E_{w r} [∥ ∥ ∥ ∥ k \prod i = 1 A_{i} ∥ ∥ ∥ ∥]

verifying our conjecture. In fact, in this case, any order of the matrix products will satisfy the desired arithmetic-geometric mean inequalities.

3.3 When is it necessary to symmetrize the order?

In contrast, symmetrizing over the order of the product is necessary for noncommutative operators. The following example, communicated to us by Aram Harrow , provides deterministic without-replacement orderings that have exponentially larger norm than the with-replacement expectation. Let $ω_{n} = π / n$ . For $n \geq 3$ , define the collection of vectors

a_{k; n} = [\begin{matrix} cos (k ω_{n}) sin (k ω_{n}) \end{matrix}] .

(3.6)

Note that all of the $a_{k; n}$ have norm $1$ and, for $1 \leq k < n$ , $⟨ a_{k; n}, a_{k + 1; n} ⟩ = cos (ω_{n})$ . The matrices $A_{k} := a_{k; n} a_{k; n}^{T}$ are all positive semidefinite for $1 \leq k \leq n$ , and we have the identity

\frac{1}{n} n \sum k = 1 A_{k} = \frac{1}{2} I .

(3.7)

Any set of unit vectors satisfying (3.7) is called a normalized tight frame, and the vectors (3.6) form a harmonic frame due to their trigonometric origin (see Hassibi et al., 2001; Goyal et al., 2001). The product of the $A_{i}$ is given by

and hence

∥ ∥ ∥ ∥ k \prod i = 1 A_{i} ∥ ∥ ∥ ∥ = {cos}^{k - 1} (ω_{n}) \geq 2^{k} {cos}^{k - 1} (ω_{n}) ∥ ∥ ∥ ∥ {(\frac{1}{n} n \sum k = 1 A_{k})}_{k}^{k} ∥ ∥ ∥ ∥ .

Therefore, the arithmetic mean is less than the deterministically ordered matrix product for all $n \geq 3$ .

It turns out that this harmonic frame example is in some sense the worst case. The following proposition shows that the geometric mean is always within a factor of $d^{k}$ of the arithmetic mean for any ordering of the without-replacement matrix product.

Proposition 3.4

Let $A_{1}, \dots, A_{n}$ be $d \times d$ positive semidefinite matrices. Then

∥ ∥ ∥ ∥ E_{w o} [k \prod i = 1 A_{j_{i}}] ∥ ∥ ∥ ∥ \leq d^{k} ∥ ∥ ∥ ∥ E_{w r} [k \prod i = 1 A_{j_{i}}] ∥ ∥ ∥ ∥

Proof If we sample $j_{1}, \dots, j_{k}$ uniformly from $[n]$ , then we have

	$∥ ∥ ∥ ∥ E_{w o} [k \prod i = 1 A_{j_{i}}] ∥ ∥ ∥ ∥$	$\leq E_{w o} [∥ ∥ ∥ ∥ k \prod i = 1 A_{j_{i}} ∥ ∥ ∥ ∥] \leq E_{w o} [k \prod i = 1 ∥ ∥ A_{j_{i}} ∥ ∥] \leq E_{w o} [k \prod i = 1 trace (A_{j_{i}})]$
		$\leq {(\frac{1}{n} n \sum i = 1 trace (A_{i}))}_{i}^{k} = trace {(\frac{1}{n} n \sum i = 1 A_{i})}_{i}^{k} \leq {∥ ∥ ∥ ∥ \frac{d}{n} n \sum i = 1 A_{i} ∥ ∥ ∥ ∥}_{i}^{k}$

Here, the first inequality follows from the triangle inequality. The second, because the operator norm is submultiplicative. The third inequality follows because the trace dominates the operator norm. The fourth inequality is Maclaurin’s. The fifth inequality follows because the trace of a $d \times d$ positive semidefinite matrix is upper bounded by $d$ times the operator norm. The final inequality is again the triangle inequality.

Note that this worst-case bound holds for deterministic orders of matrix products as well. Once we apply the submultiplicative property of the operator norm, all of the non-commutativity is washed out of the problem. Examples of deterministic matrix products saturating this upper bound can be constructed in higher dimensions using frames. If $d$ even, set

f_{k + 1}^{T} = \sqrt{\frac{2}{d}} [a_{k; n}^{T}, a_{3 k; n}^{T}, \dots, a_{(d - 1) k; n}^{T}] for k = 0, 1, \dots, n - 1,

(3.8)

and for odd

d

f_{k + 1}^{T} = \sqrt{\frac{2}{d}} [\frac{1}{\sqrt{2}}, a_{2 k; n}^{T} a_{4 k; n}^{T}, \dots, a_{(d - 1) k; n}^{T}] for k = 0, 1, \dots, n - 1 .

(3.9)

Then one can verify again using standard trigonometric identities that

\frac{1}{n} n \sum k = 1 f_{k} f_{k}^{T} = \frac{1}{d} I,

and that the inner products of adjacent $f_{i}$ are

f_{i}^{T} f_{i + 1}

= ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} \frac{2}{d} cos ((d / 2 - 1) ω_{n}) \frac{sin ((d / 2 + 1) ω_{n})}{sin (ω_{n})} - \frac{2}{d} cos (ω_{n}) & d even \frac{2}{d} cos ((d - 1) / 2 ω_{n}) \frac{sin ((d + 1) / 2 ω_{n})}{sin (ω_{n})} - \frac{1}{d} & d odd \end{matrix} .

These inner products are approximately $1 - \frac{π^{2} (d^{2} - 1)}{6 n^{2}}$ for large $n$ . Thus, each of these cases violate the arithmetic-geometric mean inequality for the order $(1, 2, \dots, k)$ by a factor of approximately $d^{k}$ provided $n \geq d$ .

At first glance, the harmonic frames example appears to cast doubt on the validity of Conjecture 3.1. However, after symmetrizing over the symmetric group, we can show that the $d = 2$ harmonic frames do obey (3.1).

Theorem 3.5

Let $λ (n) =_{2} F_{3} [\begin{matrix} 1 & - n / 2 + 1 / 2 & - n / 2 1 / 2 & - n + 1 \end{matrix}; 1]$ . With the $a_{k; n}$ defined in (3.6),

\frac{1}{n!} \sum σ \in S_{n} n \prod i = 1 a_{σ (i); n} a_{σ (i + 1); n}^{T} = - λ (n) 2^{- n} I, and 1 \geq λ (n) = O (n^{- 1}) .

This theorem additionally verifies that there is an asymptotic gap between the arithmetic and geometric means of the harmonic frames example after symmetrization. We include a full proof of this result in Appendix B. The proof treats the norm variationally using the identity that $∥ X ∥_{2}$ is the maximum of $v^{T} X v$ over all unit vectors $v$

. Our computation then reduces to effectively computing a Fourier transform of the function of

v

in an appropriately defined finite group. We show that the Fourier coefficients can be viewed as enumerating sets, and we compute them exactly using generating functions.

The combinatorial argument that we use to prove Theorem 3.5 is very specialized. To provide a broader set of examples, we now turn to show that Conjecture 3.1 does in fact hold for many classes of random matrices.

4 Random matrices

In this section, we show that if $A_{1}, \dots, A_{n}$ are generated i.i.d. from certain distributions, then Conjecture 3.1 holds in expectation with respect to the $A_{i}$ . Section 4.1 assumes that $A_{i} = Z_{i} Z_{i}^{T}$ where $Z_{i}$ have independent entries, identically sampled from some symmetric distribution. In Section 4.2, we explore when the matrices $A_{i}$ are random rank-one perturbations of the identity as was the case in the IGM and Kaczmarz examples.

4.1 Random matrices satisfy the noncommutative arithmetic-geometric mean inequality

In this section, we prove the following

Proposition 4.1

For each $i = 1, \dots, n$ , suppose $A_{i} = Z_{i} Z_{i}^{T}$ with $Z_{i}$ a $d \times r$ random matrix whose entries are i.i.d. samples from some symmetric distribution. Then Conjecture 3.1 holds in expectation.

Proof Suppose the entries of each $Z_{i}$ have finite variance $σ^{2}$ (the theorem would be otherwise vacuous if we assumed infinite variance). Let the $(a, b)$ entry of $Z_{i}$ be denoted by $Z_{a, b}^{(i)}$ . Also, denote by $W$ the matrix with all of the $Z_{i}$ stacked as columns: $W = σ^{- 1} [Z_{1}, \dots, Z_{n}]$ .

Let’s first prove that (3.1) holds in expectation for these matrices. First, consider the without-replacement samples, which are considerably easy to analyze. Let $(j_{1}, \dots, j_{k})$ be a without-replacement sample from $[n]$ . Then

∥ ∥ ∥ ∥ E [k \prod i = 1 A_{j_{i}}] ∥ ∥ ∥ ∥ = ∥ E [A_{1}]^{k} ∥ = r^{k} σ^{2 k} .

For the arithmetic mean, we can compute

	$r^{- k} σ^{- 2 k} ∥ ∥ ∥ ∥ E ⎡ ⎣ {(\frac{1}{n} n \sum i = 1 A_{i})}_{i}^{k} ⎤ ⎦ ∥ ∥ ∥ ∥ \geq \frac{1}{r^{k} σ^{2 k} d} trace ⎛ ⎝ E ⎡ ⎣ {(\frac{1}{n} n \sum i = 1 A_{i})}_{i}^{k} ⎤ ⎦ ⎞ ⎠$
$=$	$E ⎡ ⎣ d^{- 1} trace ⎛ ⎝ {(\frac{1}{n r σ^{2}} n \sum i = 1 A_{i})}_{i}^{k} ⎞ ⎠ ⎤ ⎦ = E [d^{- 1} trace ({(\frac{1}{n r} W W^{T})}^{k})]$	(4.1)
$=$	$d^{- 1} (n r)^{- k} d \sum {a_{1}, \dots, a_{k}} = 1 n r \sum {b_{1}, \dots, b_{k}} = 1 E [W_{a_{1}, b_{1}} W_{a_{2}, b_{1}} W_{a_{2}, b_{2}} W_{a_{3}, b_{2}} \dots W_{a_{k}, b_{k}} W_{a_{1}, b_{k}}]$
$=$	$(n r)^{- k} d \sum {a_{2}, \dots, a_{k}} = 1 n r \sum {b_{1}, \dots, b_{k}} = 1 E [W_{1, b_{1}} W_{a_{2}, b_{1}} W_{a_{2}, b_{2}} W_{a_{3}, b_{2}} \dots W_{a_{k}, b_{k}} W_{1, b_{k}}] .$	(4.2)

Note that since $W_{i j}$ are iid, symmetric random variables, each term in this sum is zero if it contains an odd power of $W_{i j}$ for some $i$ and $j$ . If all of the powers in a summand are even, its expected value is bounded below by $1$ . A simple lower bound for this final term (4.2) thus looks only at the contribution from when all of the indices $a_{i}$ are set equal to $1$ .

(n r)^{- k} n r \sum {b_{1}, \dots, b_{k}} = 1 E [W_{1, b_{1}}^{2} W_{1, b_{2}}^{2} \dots W_{1, b_{k}}^{2}] = E ⎡ ⎣ {(\frac{1}{n r} n r \sum b = 1 W_{1, b}^{2})}^{k} ⎤ ⎦ \geq {(E [\frac{1}{n r} n r \sum b = 1 W_{1, b}^{2}])}^{k} = 1 .

Here the inequality is Jensen’s. This calculation proves (3.1) for our family of random matrices. That is, we have demonstrated that the expected value of the with-replacement sample has greater norm than the expected value of the without-replacement sample.

To verify that (3.2

) holds for our random matrix model, we first record the following property about the fourth moments of the entries of the

A_{i}

. Let

ξ := E [G_{i j}^{4}]^{1 / 4}

. Then we can verify by direct calculation that

(4.3)

A consequence of this lemma is that $E [A_{i}^{2}] = (r (r + d - 1) σ^{4} + r ξ^{4}) I_{d}$ . Using this identity, we can set $ζ := r (r + d - 1) σ^{4} + r ξ^{4}$ and we then have

E [V (i_{1}, \dots, i_{k})] = E [A_{i_{k}} \dots A_{i_{1}}^{2} \dots A_{i_{k}}] = ζ E [A_{i_{k}} \dots A_{i_{2}}^{2} \dots A_{i_{k}}] = \dots = ζ^{k} I_{d}

We compute this identity in a second way that describes its combinatorics more explicitly, which we will use as to derive our lower bound.

	$E [V_{u, v} (i_{1}, \dots, i_{k})] =$	$E ⎡ ⎢ ⎣ \sum p \in [d]^{2 k} A_{u, p_{1}}^{(i_{1})} A_{p_{2}, v}^{(i_{2 n})} k - 1 \prod j = 2 A_{p_{i_{j - 1}}, p_{i_{j}}}^{(i_{j})} A_{p_{i_{2 k - j}}, p_{i_{2 k - j + 1}}}^{(i_{2 k - j + 1})} ⎤ ⎥ ⎦$
	$=$	$\sum p \in [d]^{2 k} E [A_{u, p_{1}}^{(i_{1})} A_{p_{2}, v}^{(i_{2 n})}] k - 1 \prod j = 2 E [A_{p_{i_{j - 1}}, p_{i_{j}}}^{(i_{j})} A_{p_{i_{2 k - j}}, p_{i_{2 k - j + 1}}}^{(i_{2 k - j + 1})}]$

The second equality uses linearity coupled with the fact that $i_{1}, \dots, i_{k}$ are distinct, hence $E [A_{u, v}^{(i_{j})} A_{u^{'}, v^{'}}^{(i_{l})}] = E [A_{u, v}^{(i_{j})}] E [A_{u^{'}, v^{'}}^{(i_{l})}]$ since elements from distinct matrices are independent. Many of the terms in this sum contain odd powers which are zero. Using the fact that $A = A^{T}$ , we see that all terms that are non-zero must contain only products of two forms: $A_{u u}^{2}$ or $A_{u v}^{2}$ . Then, we can write the sum:

V_{u, v} (i_{1}, \dots, i_{k}) = \sum p \in [d]^{k} E [(A_{u, p_{1}}^{(i_{1})})^{2}] k \prod j = 2 E [(A_{p_{j - 1}, p_{j}}^{(i_{j})})^{2}]

Now consider the case that some index may be repeated (i.e., there exist $k, l$ such that $i_{j} = i_{l}$ for $j \neq l$ ). The key observation is the following. Let $w$ be a real-valued random variable with a finite second moment. Then,

E [w^{2 p}] \geq E [w^{2}]^{p} % f o r p = 0, 1, \dots, n

(4.4)

With equality only for $p = 0, 1$ . This is Jensen’s inequality applied to $x^{p}$ for $x \geq 0$ (since $w$ is real then $w^{2}$ is positive, and $x^{p}$ is convex on $[0, \infty)$ for $p = 0, 1, 2, \dots$ ). To verify the inequality, let $n_{i}$ be the number of times index $i$ is repeated and observe

$E [V_{u, v} (i_{1}, \dots, i_{k})]$	$= E [\sum ¯ p \in [d]^{2} A_{u, p (1)}^{(i_{1})} A_{p (2), v}^{(i_{2 N})} k - 1 \prod j = 2 A_{p (i_{j - 1}), p (i_{j})}^{(i_{j})} A_{p (i_{2 k - j}), p (i_{2 k - j + 1})}^{(i_{2 k - j + 1})}]$
	$\geq \sum p \in [d]^{k} E [(A_{u, p (1)}^{(i_{1})})^{2} k \prod j = 2 (A_{p (j - 1), p (j)}^{(i_{j})})^{2}]$	(4.5)
	$\geq \sum p \in [d]^{k} E [(A_{u, p (1)}^{(i_{1})})^{2}] k \prod j = 2 E [(A_{p (j - 1), p (j)}^{(i_{j})})^{2}] .$	(4.6)

(4.5) follows from (4.3), since all terms are non-negative. (4.6) inequality is repeated application of (4.4). The final expression is precisely equal to the without-replacement average. Now, the with replacement average can be bounded as

∥ ∥ ∥ ∥ E_{w r} [k \prod j = 1 A_{i_{k - j + 1}} k \prod j = 1 A_{i_{j}}] ∥ ∥ ∥ ∥ \geq \frac{1}{d} E_{w r} [trace (k \prod j = 1 A_{i_{k - j + 1}} k \prod j = 1 A_{i_{j}}]) = E_{w r} [V_{1, 1} (i_{1}

Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences

Authors

Recht-Ré Noncommutative Arithmetic-Geometric Mean Conjecture is False

Berry–Esseen Bounds for Multivariate Nonlinear Statistics with Applications to M-estimators and Stochastic Gradient Descent Algorithms

Exponential inequality for chaos based on sampling without replacement

Without-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm

Incremental Sampling Without Replacement for Sequence Models

1 Introduction

2 Sampling in incremental gradient descent

2.1 One-dimensional Examples

2.2 IGM in more than one dimension

2.3 The Randomized Kaczmarz algorithm

3 Conjectures concerning the norm of geometric and arithmetic means of positive definite matrices

Conjecture 3.1 (Operator Inequality of Noncommutative Arithmetic and Geometric Means)

3.1 Two matrices and a search for the geometric mean

Proposition 3.2

3.2 When is it not necessary to symmetrize the order?

Theorem 3.3 (Maclaurin’s Inequalities)

3.3 When is it necessary to symmetrize the order?

Proposition 3.4

Theorem 3.5

4 Random matrices

4.1 Random matrices satisfy the noncommutative arithmetic-geometric mean inequality

Proposition 4.1

Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences

Authors

Related Research

Recht-Ré Noncommutative Arithmetic-Geometric Mean Conjecture is False

Berry–Esseen Bounds for Multivariate Nonlinear Statistics with Applications to M-estimators and Stochastic Gradient Descent Algorithms

Exponential inequality for chaos based on sampling without replacement

Without-Replacement Sampling for Stochastic Gradient Methods: Convergence Results and Application to Distributed Optimization

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm

Incremental Sampling Without Replacement for Sequence Models

This week in AI

1 Introduction

2 Sampling in incremental gradient descent

2.1 One-dimensional Examples

2.2 IGM in more than one dimension

2.3 The Randomized Kaczmarz algorithm

3 Conjectures concerning the norm of geometric and arithmetic means of positive definite matrices

Conjecture 3.1 (Operator Inequality of Noncommutative Arithmetic and Geometric Means)

3.1 Two matrices and a search for the geometric mean

Proposition 3.2

3.2 When is it not necessary to symmetrize the order?

Theorem 3.3 (Maclaurin’s Inequalities)

3.3 When is it necessary to symmetrize the order?

Proposition 3.4

Theorem 3.5

4 Random matrices

4.1 Random matrices satisfy the noncommutative arithmetic-geometric mean inequality

Proposition 4.1