Novel Near-Optimal Scalar Quantizers with Exponential Decay Rate and Global Convergence

10/29/2018 ∙ by Vijay Anavangot, et al. ∙ 0

Many modern distributed real-time signal sensing/monitoring systems require quantization for efficient signal representation. These distributed sensors often have inherent computational and energy limitations. Motivated by this concern, we propose a novel quantization scheme called approximate Lloyd-Max that is nearly-optimal. Assuming a continuous and finite support probability distribution of the source, we show that our quantizer converges to the classical Lloyd-Max quantizer with increasing bitrate. We also show that our approximate Lloyd-Max quantizer converges exponentially fast with the number of iterations. The proposed quantizer is modified to account for a relatively new quantization model which has envelope constraints, termed as the envelope quantizer. With suitable modifications we show optimality and convergence properties for the equivalent approximate envelope quantizer. We illustrate our results using extensive simulations for different finite support source distributions on the source.



There are no comments yet.


page 1

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The widespread deployment of sensors for monitoring systems (such as pollution, weather) will generate gigantic amounts of (discrete) signals/data. For efficient storage and communication, data compression methods such as quantization will play a vital role. In order to account for limited computation and energy available at these sensor nodes (due to large scale of IoT devices and mobile sampling), novel quantization algorithms will be necessary.

Scalar quantization of a signal with known probability distribution is studied in the well-known works of Lloyd and Max [1, 2]. With the advent of distributed signal processing and in-network computations [3, 4, 5]

in large scale sensor deployments, this (locally) optimal quantization scheme is infeasible due to limited energy, bandwidth and computational power at the terminal sensor nodes. The classical Lloyd-Max algorithm requires integral computations in the centroid (conditional mean) update step. In this work we introduce a nearly-optimal scalar quantization algorithm, known as Approximate Lloyd-Max (ALM) that bypasses these computationally complex operations. We show exponentially fast convergence of ALM to the near global optima for a class of source distributions where Lloyd-Max quantizer is globally optimal. Our algorithm uses vectorized update rules that are governed by linear transforms derived from localized mean square error optimizations.

The approximate Lloyd-Max algorithm deals with the mean square error cost function. We also consider a second cost function, known as envelope constrained mean square error quantization or shortened as envelope quantizer, which is relatively new and applicable to domains spanning, environmental monitoring, protection region database for TV whitespace and others [6]. The equivalent of ALM in this case is known as Approximate Envelope Quantizer (AEQ). We show that the AEQ scheme inherits all the properties of the ALM, by suitable modifications to accomodate the envelope constraint. The classical Lloyd-Max algorithm is known to have global convergence to a unique local minima, under certain restrictive class of cost function and the probaility distribution [7, 8]. Our algorithms give a generalized proof method to establish the global convergence for the class of continuously distributed sources on a finite support. The convergence result is hinged on the linear level update rule obtained as a result of the cost minimization in a local neighborhood. The same proof methodology applies to AEQ, where the additional envelope constraint is handled suitably through level shifting.

The main contributions of this work are summarized below.

  1. Linear approximation based quantization schemes - Approximate Lloyd-Max (ALM) and Approximate Envelope Quantizer (AEQ). A vectorized algorithm termed as Alternating Between Evens and Odds (ABEO) is proposed, which performs parallel computation of all the quantization levels in each iteration.

  2. Convergence of ALM and AEQ is established using linear matrix transformations and the Perron-Frobenius theory [28, 29]. We show that both the algorithms convege to a unique global minima, at exponential decay rate convergence.

  3. Near optimality of the approximation based quantization scheme is analytically established.

  4. Simulations on source models with finite support are performed to characterize the error vs bitrate tradeoff. Experiments also confirms the near-optimality of the quantization schemes.

Key ideas : The application of function approximations for quantizer error cost optimization forms the main theme of this work. The approximation scheme proposed here, simultaneously satisfy computational simplicity and accuracy of the quantization level computation. The level updates thus obtained, is represented as a sequence of linear (matrix) transformations that satisfy row-stochastic property. The structure of these matrices enables us to use ideas from Perron-Frobenius theory to establish global convergence of the proposed algorithms. The row-stochastic nature of the vectorized update rule also finds connections in gossip algorithms and consensus models [9, 10].

Ii Related Literature

Fixed rate optimal scalar quantization with known data distribution and mean square error cost function, was first studied in the independent works by Lloyd and Max [1, 2]. This iterative scheme, popularly known as the Lloyd-Max algorithm, minimizes the mean square error using alternate update of the the decision boundaries and the quantization levels. Sharma extended the Lloyd-Max method to a general class of (convex/semiconvex) distortion measures [11]. This work employs a combination of dynamic programming and fast search in order to iteratively update the quantization levels. An algorithm for quantizer design considering vector data was first introduced by Linde, Buzo and Gray [12]. Quantizer design based on a known probabilistic model or on a sequence of long training data is demonstrated here. Vector quantization follows from an efficient extension of the Lloyd-Max algorithm to higher dimensions. Another work related to vector quantization, considers predictive quantizer design hinging on tree based search methods for the class of Gauss-Markov sources [13]. Ziv proposed a variable rate universal quantizer for vector data, that achieves the optimal performance within a constant gap [14]. Other variants of high dimensional quantization using lattices and Voronoi tessellations are widely studied in mathematical literature [15, 16, 17]. Gray and Neuhoff have summarized the historical evolution of the quantization schemes, both scalar and vector cases, in their comprehensive review paper [18]. A practical approach to implement quantizers under limited computational power and memory constraints are addressed by Gersho [19]. In this respect, suboptimal and asymptotically optimal (with respect to the number of quantization levels) schemes are proposed. The computational constraints discussed here differ from the envelope constraints motivated in the current paper. In the former case the constraints are due to computational costs at higher dimension, while in this paper the constraints arise from application specific design requirements.

The convergence analysis of the Lloyd-Max algorithm is widely studied in literature. Convergence with exponential decay rate to a unique local minima, under the assumptions of a convex cost function and a log-concave probability distribution [20]. The work by Sabin and Gray explains the absolute convergence of the Lloyd algorithm and its empirical density consistency on training data [7]. The correspondence by Wu shows the convergence of the Lloyd method I for continuous, positive density function defined over a finite interval using the idea of finite state machines [8]. Another variant of this work explores dynamic programming based global optimum search methods based on the monotony properties of the error function [21]. The authors show a quadratic time algorithm that converges to the global minima. A more recent work provides a linear time accelerated multigrid search algorithm that applies to both continuous and discrete scalar probability densities [22]

. Several works have studied the quantization problem in relation to the K-means clustering framework 

[23, 24, 25]. Pollard introduced a novel approach to show the consistency theorem for k-means cluster centers and its relation on data centric quantization. Bottou and Bengio established the optimality of the K-means algorithm using gradient descent and fast Newton algorithm [24]. High dimensional Voronoi tessellation under generalized assumptions, such as compact support are covered in some relatively recent works [15, 26].

Our contributions differ from the previous literature in the following aspects. We show an exponentially fast, globally optimal convergence result for the class of finitely supported quantizers. Also the proposed algorithm is not restricted to the class of log concave (unimodal) distributions, as assumed in many of the earlier works. Our method is analytically and computationally efficient, as it involves only the use of a sequence of linear transformations and convergence based on the Perron Frobenius theory. The proposed scheme extends to more generic cost functions and optimization constraints. This is possible since the update rules are based on local optimization of the cost function.

Notations : We use the notation,

to represent a random variable and

to denote the realization of the random variable. represents the density function of . For ease of exposition, we use the phrase ”quantization for continuous distribution” to indicate ”quantization for continuously distributed sources”.

Organization : We first develop the cost function, optimality criteria and update rule corresponding to nearly optimal quantizers, ALM and AEQ (see III). In Section IV, the main analytical result relating to the global optimality and exponential convergence of the two proposed algorithms is presented. Section V discusses the simulation experiments and convergence trends of our algorithms.

Iii Nearly Optimal Quantizers for Continuous Probability Distribution

This section covers the system assumptions, cost formulation, and quantizer level updates rules for the two nearly optimal quantization algorithms proposed, viz. Approximate Lloyd Max (ALM) and the Approximate Envelope Quantizer (AEQ). The cost function for ALM is based on the mean square error, while the cost function for AEQ is the envelope constraint mean square error. The two cost functions are iteratively minimized in a local neighborhood of the existing quantization levels, so as to obtain a vectorized update policy (see Sec. III-B, Sec. III-C). Sec. III-B2

provides insights into the proposed approximate quantization scheme using the example of uniformly distributed sources. We characterize the

closeness of the proposed approximate schemes to their respective true (non-approximated) counterparts in an asymptotic sense, later in this section (see Sec. III-C2).

Iii-a Assumptions on the source distribution

Let the data observations be generated from a known continuous probability distribution . Without loss of generality, we assume that is supported on a unit interval, 111For all practical data distributions, the support is an interval . Using scale and shift operations it can be transferred to .. The following smoothness criterion is assumed over the slope of the distribution function:


The above condition states that the density function has a slope bounded by . A scalar quantizer is defined by a map . The image set of the map, represent the discrete quantization levels. For elucidating our quantization algorithm, we introduce two reference levels, and , which are fixed at the endpoints of the interval . In addition, we will assume the following ordering, on the quantization levels. The parameter above, is a fixed positive integer denoting the number of quantization levels. In this paper the condition is assumed. The subsequent discussion will elaborate on the cost functions and their minimization techniques, that lead us to the two algorithms proposed in this paper - ALM and AEQ.

Iii-B ALM cost minimization and level updates

The classical Lloyd-Max quantizer minimizes the mean square error (MSE) cost function, by alternating the updates between the quantization level set and the boundary level set . The above minimization can be equivalently performed by minimizing a collection of local cost functions in the nearest left and right neighborhood of each quantization level. This alternative approach is used to develop the ALM algorithm. The following cost function decomposition illustrates our approach:


In the above equation, the boundary set defined as,

We perform minimization of (2) by taking partial derivatives with respect to each level in the set . Using Leibniz rule of differentiation under the integral sign we get the following condition for the optimal levels [27]:


for . The equation above, in general, does not give rise to a closed form expression for . Thus, we apply a piecewise linear approximation of the density function to determine the approximate solution of (3).

Iii-B1 Optimal levels for approximate density

We consider the first order approximation of the density function in between the nearest neighbor quantization levels. That is,


where and corresponds to the slope and the intercept of the approximation. These parameters are determined using the end point conditions and . The linear approximation described above helps us to obtain a computable expression for the optimal in (3). On replacing the density function by its approximation in the optimality condition (in (3)), a cubic equation, is obtained, which has a real root in the interval (See Appendix B-A for the proof). For , the equation becomes quadratic as . The coefficients and are tabulated for the different quantization levels in Table. I

TABLE I: Coefficients of the cubic polynomial equation , to determine optimal level updates of ALM

Iii-B2 Insights into Approximate Lloyd-Max (ALM) quantization of uniformly distributed sources

Let us consider a continuous source having a uniform distribution in the interval . The optimal mean square error (MSE) quantizer for uniform distribution is trivially obtained when the levels are fixed at equispaced locations on the unit interval. However, we draw useful insights on the ALM algorithm when the levels are initialized to random points. Starting from an initialized quantization vector , the ALM algorithm minimizes the local cost function in the neighborhood interval, of each level . This results in a level update that consistently reduces the overall MSE. In this specific example of sources with uniform distribution, it is seen that the piecewise linear approximation exactly represents the true distribution. The ALM cost minimization hence follows the updates given as (see Section.III-B,III-B1),


It is noted that the same updates are obtained for the Lloyd-Max algorithm. However, for general continuous distributions, the Lloyd-Max algorithm incur additional computational expense due the evaluation of an integral for the centroid update. Also, showing the global optimality of the Lloyd-Max algorithm is analytically cumbersome for a general class of probability distributions. It will be later shown that the proposed algorithm is efficient than the conventional Lloyd-Max update, since ALM gives a vectorized rule using a series of matrix products. The motivation for our vectorized method is derived from the even odd algorithm due to Maheshwari and Kumar [6]. The original method although proposed for the data driven quantizer design, extends to the model driven case that is considered in this paper. The even odd algorithm updates the quantization levels in two steps. In the first step, we modify the values of the odd set of levels, while keeping the even indexed levels, viz. , fixed. This is followed by updating the even set, fixing the odd set. This method speeds up the computation in each iteration of the algorithm, as there are two sets of parallel update. Another advantage of the even odd algorithm is its analytical simplicity. In each level modification step, only the local neighbor points are considered. The above two properties of the even odd algorithm, helps us to represent the overall vector update as a matrix transform. For example, consider the case where . Let represent the (randomly) initialized quantization levels. Then the modified level after first iteration,


The product matrix has the structure,


The matrices and are observed to be row stochastic with non-negative entries. Hence, we can use the Perron-Frobenius theory [28, 29] to determine the fixed point of the product matrix . The fixed point determined from , corresponds the optimal solution which conforms with the optimal levels obtained using the Lloyd-Max algorithm. The above result is realized on repeated application of the update rule (6). The quantization level at the -th stage is given by update, . As , converges to a rank 2 matrix with two non-zero columns. It will be later shown that these non-zero column vectors corresponds to the fixed points of . A unique optima is obtained upon imposing an ordering on the quantization levels. We show the following properties for the matrix of interest, .

  1. is row stochastic.

  2. Eigenvalues of satisfy .

  3. is an eigenvalue and

    is a corresponding eigenvector.

  4. All eigenvectors of are either symmetric or antisymmetric.

  5. The geometric multiplicity of is 2; ie there are 2 eigenvectors corresponding to the eigenvalue .

  6. If is an eigenvector of , then is an independent eigenvector of .

The proof of the above results are discussed in Appendix A. Using these properties we can show that there exists a fixed point such that and . The proposed method is noted for its exponential rate of convergence. For the example above, the decay rate is , as the second largest eigenvalue of is .

For an envelope constrained uniform quantizer, the update rules are similar to the MSE quantizer, except for a level shift. In the following section we will discuss the envelope quantization algorithm for continuous probability density, based on insights derived from the quantization of uniform distributions.

Iii-C AEQ cost minimization and level updates

Let the mean square error cost with the envelope constraint imposed be denoted as . The following splitting of terms is possible on the terms of :


The simplification in the cost function above is performed by substituting for . It is observed that the total cost can be minimized with respect to the each quantization level . The minima corresponds to equating the partial derivative to zero. That is,


In the above equation, it is noted that corresponding to , the nearest neighbor levels and are fixed. This implies that the quantizer level updates can be performed simultaneously for all even (or odd) indices, while fixing the odd (or even) indices. Since the modified quantization levels can be determined by two separate parallel updates of even and odd sets, we term this procedure as Alternating Between Evens and Odds (ABEO). This update rule will considerably speed up the proposed quantization approach. It is observed that, in general (9) does not ensure a closed-form solution of . Hence we provide a linear approximation based algorithm for envelope quantization. This is considered in the next section.

Iii-C1 Linear Approximation Based Algorithm

The linear approximation described for ALM (in (4)) helps us to obtain a closed-form expression for the optimal for the AEQ levels. We rewrite the sufficient condition for optimality using the approximate density function .


We substitute (4) in the above equation, to obtain a third order polynomial equation, . The coefficients depends on the nearest neighbor levels, and . We list these coefficients in Table II.

TABLE II: Coefficients of the cubic polynomial equation , to determine optimal level updates of AEQ

The roots of the cubic equation in the interval , corresponds to the optimum level update of . The existence of atleast one real root in is shown in the Appendix B-B. The proposed linear approximation based quantization scheme is described in a step-wise manner in Algorithm. 1.

Input : Input distribution , # of levels, MaxIter, Threshold
Output : List of quantization levels,
Initialization : , stop condition = False, , dist = 0
1 while !stop condition do
4       % where
5       for (In Parallel) in  do
7             Set linear approximation parameters :
9       end for
10      for (In Parallel) in  do
12             Update with steps in loop above
14       end for
15      ; if (dist Threshold) or (iter MaxIter) then
16             stop condition = True
17       end if
19 end while
Algorithm 1 Scalar Envelope Quantizer Algorithm

For the ALM scheme, the steps of Algorithm. 1 is valid, when the level modification step in line 8 is performed using the cubic polynomial in Table. I instead of . The ALM and AEQ levels obtained using the piecewise linear approximation will be shown to be arbitrarily close to their respective true levels as .

Iii-C2 Asymptotic Optimality of Piecewise Linear Approximation

Let be the optimal quantizer with respect to the true density function and be the quantizer obtained by the linear approximation of the density . The asymptotic convergence (as quantization levels ) of the linear approximation scheme can be established by using the Taylor series expansion. At , the Taylor approximation around the interval and , is given by


For the simplifying the notations in the analysis, we restrict our attention to the value of . The two neighboring levels of interest are and . Let and denote the optimal level updates of the true density and the approximated density respectively (see (9) and (10)). Then, the Taylor expansion at is,


where . Using the above fact, for all . For the specific example of a uniformly distributed source, . The asymptotic optimality of the ALM and AEQ schemes, as is summarized in the following result.

Theorem 3.1 (Asymptotic optimality of ALM and AEQ).

The approximate solution of the quantization level update (see Table. I for ALM and Table. II for AEQ), converges to the true solution, as the number of levels . That is, there exists a such that for all .


The proofs for ALM and AEQ schemes are dealt separately below.

ALM Optimality : Consider the level update expression in (3) evaluated at the true solution . Using the fact that , the expressions corresponding to the true density, and its approximation, are given by,


On subtracting the (13) from (14) and using the Taylor approximation in (12), we show,


We use the fact that the local MSE cost function about is continuous and has a positive second derivative. By the continuity of the cost function, we infer that as .

AEQ Optimality : We note that the true solution satisfies the optimality equation in (9). On applying in (10), the rule is satisfied with an offset value . That is,


Subtracting (16) from (17), we get


The above result shows that the offset as a function of the optimal solution eventually converges to zero. Since the approximate solution, is the root of the left hand side of (17), we argue that approaches arbitrarily close, as . The fact is true since the AEQ optimality condition (10) is continuous in and has a positive derivative at (see Appendix C for proof). The above properties ensure that, as . ∎

Remark 1 (An alternate bound on ALM and AEQ near optimality result).

The absolute difference of the nearly optimal and the true optimal values of are bounded by the maximum length of the interval . That is,

This gives a loose bound on the near optimality, which follows directly from the decreasing interval length with .

The above result holds true when is replaced by any . Hence, we see that the approximated quantization vector converges to the true quantization vector under the norm. That is,

In simulations (shown in Fig. 4(g)-(i)) , it is observed that the quantization levels obtained from the approximation schemes are close to the true optima, computed using the original density function. The ALM and AEQ schemes proposed here achieves a nearly optimal solution, with a reduced computational burden.

From this point, we treat the analysis of ALM and AEQ algorithms in a common framework. We abstract out the solution of the cost minimization procedure, that is the roots of the polynomials in Table I and Table II, and express the resulting levels shifts as linear transformations.

Iii-D Level shifts as linear updates

The optimal solution for the iterative update of level is given by the roots of (3) or (10) in the interval . The solution at -th iteration can be expressed as a convex combination,


The above update equation will aid in the convergence analysis of the proposed algorithm. In vector notation the ABEO update rule can be expressed as,


In the above equation and are square matrices having dimension . Note that for the ALM scheme and for AEQ . These square matrices determines the optimal level updates obtained using (19). For instance, for ,


We note that the two matrices, and are row stochastic. A (row) symmetry on the location of the zeros is also observed. The matrix operators preserve the values of reference levels, and in every iteration. This is attributed to the first and the last rows of the (21) The vector update of the quantizer explained in (20)-(21), has got the required structure to apply convergence using the Perron Frobenius theory [28, 29].

Iv Convergence Analysis of Near Optimal Quantizers

This section describes the analysis for convergence of the linear approximation based method in Algorithm 1. Using the fact that, the product of two row stochastic matrices is row stochastic, we show that has every row that sums to unity. For the case, the above product matrix has the following structure:


where is used for concise notation. Other properties from the individual matrices, such as (row) symmetry on zero locations, are carried forward to the matrix. The first and last rows of the matrix are independent of the scale parameters . An important observation is regarding the zero vectors, that appear alternatively in the columns of the above matrix. This occurs due to the fact that the linear updates and , acts only on the alternate entries (nearest neighbors) of the quantization vector . An important observation on is that for all entries . Further, we show that the coefficient of the linear combination for are bounded away from the extremes and . This is shown in the Fact below.

Proposition 1 (’s are bounded away from extremes).

The coefficients for in (22) satisfy the criteria for all iteration count .


From the smoothness assumption (1), we deduce the fact that the linear approximation slope is bounded, that is, . First, we consider the case when . From the ALM and AEQ optimality conditions (see (3) and (10)) we observe that . In other words, a flat density approximation results in .

In the second case, when and , we show that . To establish the result, we first observe the fact that the ALM and AEQ solutions always lie in the interval (see Appndix B-A, B-B). Thus . We now consider the boundary cases corresponding to and . These are equivalent to the solution and respectively. It is observed that under these extreme cases the ALM and AEQ optimality conditions (see (3) and (10)) are valid only when ; which corresponds to a trivial case. The resulting contradiction, hence shows that , for all bounded values of the slope . ∎

In accordance with Algorithm 1, the quantization levels after iterations is . We show the following convergence property of the product below.

Proposition 2 (Convergence of columns of product matrix).

The odd columns of the sequence converges to zero as .


We consider the first two terms of the product, viz. and . The two matrices are row stochastic. Any element of the product is an inner product between a row vector of and a column vector of . Let and be the (column vector) representation of the -th row of and the -th column of respectively. Then, , the entry of the product , is the inner product between and . That is, . The following facts hold true for and :

  1. for every and . (See Proposition. 1)

  2. where . This is true since

    is a row stochastic matrix.

Since the inner product is a (non-zero) convex combination (from (F2) and due to Proposition. 1) of components of , from (F1) we assert that


In other words, all the elements of columns are strictly less than the maximum element in the corresponding columns of . Alternatively, we can represent this as a contraction, , where . Using the fact that product is row stochastic, we can extend the same argument to the product of three matrices, that is . We make use of an induction argument to show the property in the limiting case. Let be the -th element of the product matrix . Then using (23), we have the contraction of the sequence,


where . From above we observe that, the sequence of inner product terms, is a monotonically decreasing sequence for every and . Using monotone convergence theorem [30] on the above (bounded) sequence, we show that the columns of the product sequence eventually decreases to zero. ∎

Theorem 4.1 (Convergence to global minima).

The iteration in Algorithm. 1 converges to a quantization vector,


where , and is independent of the initialization (except for the reference levels).


We note that the limiting product matrix converges to a matrix with columns (See Proposition. 2). The first and last columns, viz. and , are non-zero vectors, as the matrix transformation