(Learned) Frequency Estimation Algorithms under Zipfian Distribution

1 Introduction

The last few years have witnessed a rapid growth in using machine learning methods to solve “classical” algorithmic problems. For example, they have been used to improve the performance of data structures [KBC $^{+}$ 18, Mit18], online algorithms [LV18, PSK18, GP19]

, combinatorial optimization

[KDZ

^{+}

17, BDSV18], similarity search [WLKC16], compressive sensing [MPB15, BJPD17] and streaming algorithms [HIKV19]. Multiple frameworks for designing and analyzing such algorithms were proposed [ACC

^{+}

11, GR17, BDV18, AKL

^{+}

19]. The rationale behind this line of research is that machine learning makes it possible to adapt the behavior of the algorithms to inputs from a specific data distribution, making them more efficient or more accurate in specific applications.

In this paper we focus on learning-augmented streaming algorithms for frequency estimation. The latter problem is formalized as follows: given a sequence $S$ of elements from some universe $U$ , construct a data structure that for any element $i \in U$ computes an estimation ${~ f}_{i}$ of $f_{i}$ , the number of times $i$ occurs in $S$ . Since counting data elements is a very common subroutine, frequency estimation algorithms have found applications in many areas, such as machine learning, network measurements and computer security. Many of the most popular algorithms for this problem, such as Count-Min (CM) [CM05] or Count-Sketch (CS) [CCFC02] are based on hashing. Specifically, these algorithms hash stream elements into $B$ buckets, count the number of items hashed into each bucket, and use the bucket value as an estimate of item frequency. To improve the accuracy, the algorithms use $k > 1$ such hash functions and aggregate the answers. These algorithms have several useful properties: they can handle item deletions (implemented by decrementing the respective counters), and some of them (Count-Min) never underestimate the true frequencies, i.e., ${~ f}_{i} \geq f_{i}$ .

In a recent work [HIKV19]

, the authors showed that the aforementioned algorithm can be improved by augmenting them with machine learning. Their approach is as follows. During the training phase, they construct a classifier (neural network) to detect whether an element is “heavy”, i.e., whether

f_{i}

exceeds some threshold. After such a classifier is trained, they scan the input stream, and apply the classifier to each element

i

. If the element is predicted to be heavy, it is allocated a unique bucket, so that an exact value of

f_{i}

is computed. Otherwise, the element is forwarded to a “standard” hashing data structure

C

, e.g., CM or CS. To estimate

{~ f}_{i}

, the algorithm either returns the exact count

f_{i}

(if

i

is allocated a unique bucket) or an estimate provided by the data structure

C

.¹¹1See Figure 1 for a generic implementation of the learning-based algorithms of [HIKV19]. An empirical evaluation, on networking and query log data sets, shows that this approach can reduce the overall estimation error.

The paper also presents a preliminary analysis of the algorithm. Under the common assumption that the frequencies follow the Zipfian law, i.e.,²²2In fact we will assume that $f_{i} = 1 / i$ . This is just a matter of scaling and is convenient as it removes the dependence of the length of the stream in our bounds $f_{i} \propto 1 / i$ , and further that item $i$

is queried with probability proportional to its frequency, the expected error incurred by the learning-augmented version of CM is shown to be asymptotically lower than that of the “standard” CM.

³³3This assumes that the error rate for the “heaviness” predictor is low enough. Aiming at a theoretical understanding, in this paper we focus on the case where the error rate is zero. However, the magnitude of the gap between the error incurred by the learned and standard CM algorithms has not been established. Specifically, [HIKV19] only shows that the expected error of standard CM with

k

hash functions and a total of

B

buckets is between

\frac{k}{B log (k)}

and

\frac{k {log}^{(k + 2) / (k - 1)} (k n / B)}{B}

. Furthermore, no such analysis was presented for CS.

1.1 Our results

In this paper we resolve the aforementioned questions left open in [HIKV19]. Assuming that the frequencies follow a Zipfian law, we show:

An asymptotically tight bound of $Θ (\frac{k log (k n / B)}{B})$ for the expected error incurred by the CM algorithm with $k$ hash functions and a total of $B$ buckets. Together with a prior bound for Learned CM (Table 1), this shows that learning-augmentation improves the error of CM by a factor of $Θ (log (n) / log (n / B))$ if the heavy hitter oracle is perfect.
The first error bounds for CS and Learned CS (see Table 1). In particular, we show that for Learned CS, using a single hash function as in [HIKV19] leads to an asymptotically optimal error bound, improving over standard CS by a factor of $Θ (log (n) / log (n / B))$ (same as CM).

	$k = 1$	$k > 1$
Count-Min (CM)	$Θ (\frac{log n}{B})$ [HIKV19]	$Θ (\frac{k \cdot log (\frac{k n}{B})}{B})$
Learned Count-Min (L-CM)	$Θ (\frac{{log}^{2} (\frac{n}{B})}{B log n})$ [HIKV19]	$Ω (\frac{{log}^{2} (\frac{n}{B})}{B log n})$ [HIKV19]
Count-Sketch (CS)	$Θ (\frac{log B}{B})$	$Ω (\frac{k^{1 / 2}}{B log k})$ and $O (\frac{k^{1 / 2}}{B})$
Learned Count-Sketch (L-CS)	$Θ (\frac{log \frac{n}{B}}{B log n})$	$Ω (\frac{log \frac{n}{B}}{B log n})$

Table 1: This table summarizes our and previously known results on the expected frequency estimation error of Count-Min (CM), Count-Sketch (CS) and their learned variants (i.e., L-CM and L-CS) that use

k

functions and overall space

k \times \frac{B}{k}

under Zipfian distribution. For CS, we assume that

k

is odd (so that the median of

k

values is well defined). The lower bounds for L-CM and L-CS are assuming that we use the information from a perfect heavy hitter oracle (i.e., the oracle makes no mistake in its predictions) to place the heavy hitters in separate buckets.

In addition to clarifying the gap between the learned and standard variants of popular frequency estimation algorithms, our results provide interesting insights about the algorithms themselves. For example, for both CM and CS, the number of hash functions $k$ is often selected to be $Θ (log n)$ , in order to guarantee that every frequency is estimated up to a certain error bound. In contrast, we show that if instead the goal is to bound the expected error, then setting $k$ to a constant (strictly greater than $1$ ) leads to the asymptotic optimal performance. We remark that the same phenomenon holds not only for a Zipfian query distribution but in fact for an arbitrary distribution on the queries, e.g. the uniform (see Remark 2.2).

1.2 Related work

In addition to the aforementioned hashing-based algorithms [CM05, CCFC02], multiple non-hashing algorithms were also proposed, e.g., [MG82, MM02, MAEA05]. These algorithms often exhibit better accuracy/space tradeoffs, but do not posses many of the properties of hashing-based methods, such as the ability to handle deletions as well as insertions.

Zipf law is a common modeling tool used to evaluate the performance of frequency estimation algorithms, and has been used in many papers in this area, including [MM02, MAEA05, CCFC02]. In its general form it postulates that $f_{i}$ is proportional to $1 / i^{s}$ for some exponent parameter $s > 0$ . In this paper we focus on the “original” Zipf law where $s = 1$ . However, the techniques introduced in this paper can be applied to other values of the exponent $s$ as well.

1.3 Our techniques

Our main contribution is our analysis of the standard Count-Min and Count-Sketch algorithms for Zipfians with $k > 1$ hash functions. Showing the improvement for the learned counterparts is relatively simple (for Count-Min it was already done in [HIKV19]). In both of these analyses we consider a fixed item $i$ and bound $E [| f_{i} - {~ f}_{i} |]$ whereupon linearity of expectation leads to the desired results. In the following we assume that $f_{j} = 1 / j$ for each $j \in [n]$ and describe our techniques for bounding $E [| f_{i} - {~ f}_{i} |]$ for each of the two algorithms.

Count-Min. With a single hash function and $B$ buckets the head of the Zipfian distribution, namely the items of frequencies $(f_{i})_{i \in [B]}$ , contribute with $log B / B$ to the expected error. Our main observation is that the fast decay of $(f_{i})_{i \in [B]}$ reduces this to $1 / B$ for $k \geq 2$ and that in fact the main contribution to the error comes from the light items $(f_{i})_{i \in [n] ∖ [B]}$ . The expected contribution of these items is easily upper bounded and can be lower bounded using Bennett’s concentration inequality. In contrast to the analysis from [HIKV19] which is technical and leads to suboptimal bounds, our analysis is short, simple, and yields completely tight bounds in terms of all of the parameters $k, n$ and $B$ .

Count-Sketch.

Simply put, our main contribution is an improved understanding of the distribution of random variables of the form

S = \sum_{i = 1}^{n} f_{i} η_{i} σ_{i}

. Here the

η_{i} \in {0, 1}

are i.i.d Bernouilli random variables and the

σ_{i} \in {- 1, 1}

are independent Rademachers, that is,

Pr [η_{i} = 1] = Pr [η_{i} = - 1] = 1 / 2

. Note that the counters used in CS are random variables having precisely this form. Usually such random variables are studied for the purpose of obtaining large deviation results. In contrast, in order to analyze CS, we are interested in a fine-grained picture of the distribution within a “small” interval

I

around zero, say with

Pr [S \in I] = 1 / 2

. For example when proving a lower bound on

E [| f_{i} - {~ f}_{i} |]

we must prove a certain anti-concentration of

S

around

0

. More precisely we find an interval

J \subset I

centered at zero such that

Pr [S \in J] = O (1 / \sqrt{k})

. Combined with the fact that we use

k

independent hash functions as well as properties of the median and the binomial distribution, this gives that

E [| f_{i} - {~ f}_{i} |] = Ω (| J |)

. Anti-concentration inequalities of this type are in general notoriously hard to obtain but it turns out that we can leverage the properties of the Zipfian distribution, specifically its heavy head. For our upper bounds on

E [| f_{i} - {~ f}_{i} |]

we need strong lower bounds on

Pr [S \in J]

for intervals

J \subset I

centered at zero. Then using concentration inequalities we can bound the probability that half of the

k

relevant counters are smaller (larger) than the lower (highter) endpoint of

J

, i.e., that the median does not lie in

J

. Again this requires a precise understanding of the distribution of

S

within

I

1.4 Structure of the paper

In Section 2 we describe the algorithms Count-Min and Count-Sketch. We also formally define the estimation error that we will study as well as the Zipfian distribution. In Sections 4 and 3 we provide our analyses of the expected error of Count-Min and Count-Sketch. In Section 5 we analyze the performance of learned Count-Sketch.

2 Preliminaries

We start out by recapping the sketching algorithms Count-Min and Count-Sketch. Common to both of these algorithms is that we sketch a stream $S$ of elements coming from some universe $U$ of size $n$ . For notational convenience we will assume that $U = [n] := {1, \dots, n}$ . If item $i$ occurs $f_{i}$ times then either algorithm outputs an estimate ${~ f}_{i}$ of $f_{i}$ .

Count-Min.

We use $k$ independent and uniformly random hash functions $h_{1}, \dots, h_{k} : [n] \to [B]$ . Letting $C$ be an array of size $[k] \times [B]$ we let $C [ℓ, b] = \sum_{j \in [n]} [h_{ℓ} (j) = b] f_{j}$ . When querying $i \in [n]$ the algorithm returns ${~ f}_{i} = {min}_{ℓ \in [k]} C [ℓ, h_{ℓ} (i)]$ . Note that we always have that ${~ f}_{i} \geq f_{i}$ .

Count-Sketch.

We pick independent and uniformly random hash functions $h_{1}, \dots, h_{k} : [n] \to [B]$ and $s_{1}, \dots, s_{k} : [n] \to {- 1, 1}$ . Again we initialize an array $C$ of size $[k] \times [B]$ but now we let $C [ℓ, b] = \sum_{j \in [n]} [h_{ℓ} (j) = b] s_{ℓ} (j) f_{j}$ . When querying $i \in [n]$ the algorithm returns the estimate ${~ f}_{i} = {m e d i a n}_{ℓ \in [k]} s_{ℓ} (i) \cdot C [ℓ, h_{ℓ} (i)]$ .

Remark 2.1.

The bounds presented in Table 1 assumes that the hash functions have codomain $[B / k]$ and not $[B]$ , i.e., that the total number of buckets is $B$ . In the proofs to follows we assume for notational ease that the hash functions take value in $[B]$ and the claimed bounds follows immediately by replacing $B$ by $B / k$ .

Estimation Error.

To measure and compare the overall accuracy of different frequency estimation algorithms, we will use the expected estimation error which is defined as follows: let $F = {f_{1}, \dots, f_{n}}$ and ${~ F}_{A} = {{~ f}_{1}, \dots, {~ f}_{n}}$ respectively denote the actual frequencies and the estimated frequencies obtained from algorithm $A$ of items in the input stream. We remark that when $A$ is clear from the context we denote ${~ F}_{A}$ as $~ F$ . Then we define

E r r (F, {~ F}_{A}) := E_{i \sim D} | f_{i} - {~ f}_{i} |,

(1)

where $D$ denotes the query distribution of the items. Here, similar to previous work (e.g., [RKA16, HIKV19]), we assume that the query distribution $D$ is the same as the frequency distribution of items in the stream, i.e., for any $i^{*} \in [n]$ , ${Pr}_{i \sim D} [i = i^{*}] \propto f_{i^{*}}$ (more precisely, for any $i^{*} \in [n]$ , ${Pr}_{i \sim D} [i = i^{*}] = f_{i^{*}} / N$ where $N = \sum_{i \in [n]} f_{i}$ denotes the total sum of all frequencies in the stream).

Remark 2.2.

As all upper/lower bounds in this paper are proved by bounding the expected error of estimating the frequency a single item, $E [| {~ f}_{i} - f_{i} |]$ , then using linearity of expectation, in fact we obtain analogous bounds for any query distribution $(p_{i})_{i \in [n]}$ . In particularly this means that the bounds of Table 1 for CM and CS hold for any query distribution. For L-CM and L-CS the factor of $log (n / B) / log n$ gets replaced by $\sum_{i = B_{h} + 1}^{n} p_{i}$ where $B_{h} = Θ (B)$ is the number of buckets reserved for heavy hitters.

Zipfian Distribution

In our analysis we assume that the frequency distribution of items follows Zipf’s law. That is, if we sort the items according to their frequencies with no loss of generality assuming that $f_{1} \geq f_{2} \geq \dots \geq f_{n}$ , then for any $j \in [n]$ , $f_{j} \propto 1 / j$ . Given that the frequencies of items follow Zipf’s law and assuming that the query distribution is the same as the distribution of the frequency of items in the input stream (i.e., ${Pr}_{i \sim D} [i^{*}] = f_{i^{*}} / N = 1 / (i^{*} \cdot H_{n})$ where $H_{n}$ denotes the $n$ -th harmonic number), we can write the expected error defined in eq. 1 as follows:

E r r (F, {~ F}_{A}) = E_{i \sim D} [| f_{i} - {~ f}_{i} |] = \frac{1}{N} \cdot \sum i \in [n] | {~ f}_{i} - f_{i} | \cdot f_{i} = \frac{1}{H_{n}} \cdot \sum i \in [n] | {~ f}_{i} - f_{i} | \cdot \frac{1}{i}

(2)

Throughout this paper, we present our results with respect to the objective function in the right hand side of eq. 2, i.e., $(1 / H_{n}) \cdot \sum_{i = 1}^{n} | {~ f}_{i} - f_{i} | \cdot f_{i}$ . We further assume that in fact $f_{i} = 1 / i$ . At first sight this assumption may seem strange since it says that item $i$ appears a non-integral number of times in the stream. This is however just a matter of scaling and the assumption is convenient as it removes the dependence on the length of the stream in our bounds.

1:procedure LearnedSketch(

B

B_{h}

H H - O r a c l e

S k e t c h - A l g

)

2: for each stream element

i

3: if

H H - O r a c l e (i) = 1

then

▹

the oracle predicts

i

as heavy (one of

B_{h}

most frequent items)

4: if a unique bucket is already assigned to item

i

then

{c o u n t e r}_{i} \leftarrow {c o u n t e r}_{i} + 1

6: else

7: allocate a new unique bucket to item

i

and

{c o u n t e r}_{i} \leftarrow 1

8: end if

9: else

10: feed

i

S k e t c h - A l g (B - B_{h})

▹

an instance of

S k e t c h - A l g

with

B - B_{h}

many buckets

11: end if

12: end for

13:end procedure

Algorithm 1 Learning-Based Frequency Estimation

Figure 1: A generic learning augmented algorithm for the frequency estimation problem.

H H - O r a c l e

denotes a given learned oracle for detecting whether the item is among the top

B_{h}

frequent items of the stream and

S k e t c h - A l g

is a given (sketching) algorithm (e.g., CM or CS) for the frequency estimation problem.

Learning Augmented Sketching Algorithms for Frequency Estimation.

In this paper, following the approach of [HIKV19], the learned variants of CM and CS are algorithms augmented with a machine learning based heavy hitters oracle. More precisely, we assume that the algorithm has access to an oracle $H H - O r a c l e$ that predicts whether an item is “heavy” (i.e., is one of the $B_{h}$ most frequent items) or not. Then, the algorithm treats heavy and non-heavy items differently: (a) a unique bucket is allocated to each heavy item and their frequencies are computed with no error, (b) the rest of items are fed to the given (sketching) algorithm $S k e t c h - A l g$ using the remaining $B - B_{h}$ buckets and their frequency estimates are computed via $S k e t c h - A l g$ . See Figure 1.

Note that, in general the oracle $H H - O r a c l e$ can make errors. Aiming at a theoretical understanding, in this paper we focus on the case where the oracle is perfect, i.e., the error rate is zero. We also assume that $B_{h} = Θ (B - B_{h}) = Θ (B)$ , that is, we use approximately the same number of buckets for the heavy items as for the sketching of the light items. One justification for this assumption is that in any case we can increase both the number of buckets for heavy and light items to $B$ without affecting the overall asymptotic space usage.

3 Tight Bounds for Count-Min with Zipfians

For both Count-Min and Count-Sketch we aim at analyzing the expected value of the variable $\sum_{i \in [n]} f_{i} \cdot | {~ f}_{i} - f_{i} |$ where $f_{i} = 1 / i$ and $_{i}$ is the estimate of $f_{i}$ output by the relevant sketching algorithm. Throughout this paper we use the following notation: For an event $E$ we denote by $[E]$ the random variable in ${0, 1}$ which is $1$ if and only if $E$ occurs. We begin by presenting our improved analysis of Count-Min with Zipfians. The main theorem is the following.

Theorem 3.1.

Let $n, B, k \in N$ with $k \geq 2$ and $B \leq n / k$ . Let further $h_{1}, \dots, h_{k} : [n] \to [B]$ be independent and truly random hash functions. For $i \in [n]$ define the random variable $_{i} = {min}_{ℓ \in [k]} (\sum_{j \in [n]} [h_{ℓ} (j) = h_{ℓ} (i)] f_{j})$ . For any $i \in [n]$ it holds that

E [| {~ f}_{i} - f_{i} |] = Θ ⎛ ⎜ ⎜ ⎝ \frac{log (\frac{n}{B})}{B} ⎞ ⎟ ⎟ ⎠

Replacing $B$ by $B / k$ in Theorem 3.1 and using linearity of expectation we obtain the desired bound for Count-Min in the upper right hand side of Table 1. The natural assumption that $B \leq n / k$ simply says that the total number of buckets is upper bounded by the number of items.

To prove Theorem 3.1 we start with the following lemma which is a special case of the theorem.

Lemma 3.2.

Suppose that we are in the setting of Theorem 3.1 and further that⁴⁴4In particular we dispose with the assumption that $B \leq n / k$ . $n = B$ . Then

E [|_{i} - f_{i} |] = O (\frac{1}{n}) .

Proof.

It suffices to show the result when $k = 2$ since adding more hash functions and corresponding tables only decreases the value of $| {~ f}_{i} - f_{i} |$ . Define $Z_{ℓ} = \sum_{j \in [n] ∖ {i}} [h_{ℓ} (j) = h_{ℓ} (i)] f_{j}$ for $ℓ \in [2]$ and note that these variables are independent. For a given $t \geq 3 / n$ we wish to upper bound $Pr [Z_{ℓ} \geq t]$ . Let $s < t$ and note that if $Z_{ℓ} \geq t$ then either of the following two events must hold:

There exists a $j \in [n] ∖ {i}$ with $f_{j} > s$ and $h_{ℓ} (j) = h_{ℓ} (i)$ .
The set ${j \in [n] ∖ {i} : h_{ℓ} (j) = h_{ℓ} (i)}$ contains at least $t / s$ elements.

Union bounding we find that

Pr [Z_{ℓ} \geq t] \leq Pr [E_{1}] + Pr [E_{2}] \leq \frac{1}{n s} + (\frac{n}{t / s}) n^{- t / s} \leq \frac{1}{n s} + {(\frac{e s}{t})}^{t / s} .

Choosing $s = \frac{t}{log (t n)}$ , a simple calculation yields that $Pr [Z_{ℓ} \geq t] = O (\frac{log (t n)}{t n})$ . As $Z_{1}$ and $Z_{2}$ are independent, $Pr [Z \geq t] = O ({(\frac{log (t n)}{t n})}^{2})$ , so

E [Z] = \int_{0}^{\infty} Pr [Z \geq t] d t \leq \frac{3}{n} + O ⎛ ⎝ \int_{3 / n}^{\infty} {(\frac{log (t n)}{t n})}^{2} d t ⎞ ⎠ = O (\frac{1}{n}) .

∎

Before proving the full statement of Theorem 3.1 we recall Bennett’s inequality.

Theorem 3.3 (Bennett’s inequality [Ben62]).

Let $X_{1}, \dots, X_{n}$ be independent, mean zero random variables. Let $S = \sum_{i = 1}^{n} X_{i}$ , and $σ^{2}, M > 0$ be such that $V a r [S] \leq σ^{2}$ and $| X_{i} | \leq M$ for all $i \in [n]$ . For any $t \geq 0$ ,

Pr [S \geq t] \leq exp (- \frac{σ^{2}}{M^{2}} h (\frac{t M}{σ^{2}})),

where $h : R_{\geq 0} \to R_{\geq 0}$ is defined by $h (x) = (x + 1) log (x + 1) - x$ . The same tail bound holds on the probability $Pr [S \leq - t]$ .

Remark 3.4.

It is well known and easy to check that for $x \geq 0$ ,

\frac{1}{2} x log (x + 1) \leq h (x) \leq x log (x + 1) .

We will use these asymptotic bounds repeatedly in this paper.

Proof of Theorem 3.1.

We start out by proving the upper bound. Let $N_{1} = [B] ∖ {i}$ and $N_{2} = [n] ∖ ([B] \cup {i})$ . Let $b \in [k]$ be such that $\sum_{j \in N_{1}} f_{j} \cdot [h_{b} (j) = h_{b} (i)]$ is minimized. Note that $b$ is itself a random variable. We also define

	$Y_{1}$	$= \sum j \in N_{1} f_{j} \cdot [h_{b} (j) = h_{b} (i)],$
	$Y_{2}$	$= \sum j \in N_{2} f_{j} \cdot [h_{b} (j) = h_{b} (i)]$

Clearly $| {~ f}_{i} - f_{i} | \leq Y_{1} + Y_{2}$ . Using Lemma 3.2, we obtain that $E [Y_{1}] = O (\frac{1}{B})$ . For $Y_{2}$ we observe that

E [Y_{2} ∣ b] = \sum j \in N_{2} \frac{f_{j}}{B} = O ⎛ ⎜ ⎜ ⎝ \frac{log (\frac{n}{B})}{B} ⎞ ⎟ ⎟ ⎠ .

We conclude that

E [| {~ f}_{i} - f_{i} |] \leq E [Y_{1}] + E [Y_{2}] = E [Y_{1}] + E [E [Y_{2} ∣ b]] = O ⎛ ⎜ ⎜ ⎝ \frac{log (\frac{n}{B})}{B} ⎞ ⎟ ⎟ ⎠,

as desired.

Next we show the lower bound. For $j \in [n]$ and $ℓ \in [k]$ we define $X_{ℓ}^{(j)} = f_{j} \cdot ([h_{ℓ} (j) = h_{ℓ} (i)] - \frac{1}{B})$ . Note that the variables $(X_{ℓ}^{(j)})_{j \in [n]}$ are independent. We also define $S_{ℓ} = \sum_{j \in N_{2}} X_{ℓ}^{(j)}$ for $ℓ \in [k]$ . Observe that $| X_{ℓ}^{(j)} | \leq f_{j} \leq \frac{1}{B}$ for $j \geq B$ , $E [X_{ℓ}^{(j)}] = 0$ , and that

V a r [S_{ℓ}] = \sum j \in N_{2} f_{j}^{2} (\frac{1}{B} - \frac{1}{B^{2}}) \leq \frac{2}{B^{2}} .

Applying Bennett’s inequality with $σ^{2} = \frac{2}{B^{2}}$ and $M = 1 / B$ thus gives that

Pr [S_{ℓ} \leq - t] \leq exp (- 2 h (\frac{t B}{2})) .

Defining $W_{ℓ} = \sum_{j \in N_{2}} f_{j} \cdot [h_{ℓ} (j) = h_{ℓ} (i)]$ it holds that $E [W_{ℓ}] = Θ (\frac{log (\frac{n}{B})}{B})$ and $S_{ℓ} = W_{ℓ} - E [W_{ℓ}]$ , so putting $t = E [W_{ℓ}] / 2$ in the inequality above we obtain that

Pr [W_{ℓ} \leq E [W_{ℓ}] / 2] = Pr [S_{ℓ} \leq - E [W_{ℓ}] / 2] \leq exp (- 2 h (Ω (log \frac{n}{B}))) .

Appealing to Remark 3.4 and using that $B \leq n / k$ the above bound becomes

	$Pr [W_{ℓ} \leq E [W_{ℓ}] / 2]$	$\leq exp (- Ω (log \frac{n}{B} \cdot log (log \frac{n}{B} + 1)))$
		$= exp (- Ω (log k \cdot log (log k + 1))) = k^{- Ω (log (log k + 1))} .$		(3)

By the independence of the events $(W_{ℓ} > E [W_{ℓ}] / 2)_{ℓ \in [k]}$ we have that

Pr [| {~ f}_{i} - f_{i} | \geq \frac{E [W_{ℓ}]}{2}] \geq (1 - k^{- Ω (log (log k + 1))})^{k} = Ω (1),

and so $E [| {~ f}_{i} - f_{i} |] = Ω (E [W_{ℓ}]) = Ω (\frac{log (\frac{n}{B})}{B})$ , as desired. ∎

4 (Nearly) tight Bounds for Count-Sketch with Zipfians

In this section we proceed to analyze Count-Sketch for Zipfians either using a single or more hash functions. We start with two simple lemmas which for certain frequencies $(f_{i})_{i \in [n]}$ of the items in the stream can be used to obtain respectively good upper and lower bounds on $E [| {~ f}_{i} - f_{i} |]$ in Count-Sketch with a single hash function. We will use these two lemmas both in our analysis of standard and learned Count-Sketch for Zipfians.

Lemma 4.1.

Let $w = (w_{1}, \dots, w_{n}) \in R^{n}$ , $η_{1}, \dots, η_{n}$ independent Bernoulli variables taking value $1$ with probability $p$ , and $σ_{1}, \dots, σ_{n} \in {- 1, 1}$ independent Rademachers, i.e., $Pr [η_{i} = 1] = Pr [η_{i} = - 1] = 1 / 2$ . Let $S = \sum_{i = 1}^{n} w_{i} η_{i} σ_{i}$ . Then

E [| S |] = O (\sqrt{p} ∥ w ∥_{2}) .

Proof.

Using that $E [σ_{i} σ_{j}] = 0$ for $i \neq j$ and Jensen’s inequality

E [| S |]^{2} \leq E [S^{2}] = E [n \sum i = 1 w_{i}^{2} η_{i}] = p ∥ w ∥_{2}^{2},

from which the result follows. ∎

Lemma 4.2.

Suppose that we are in the setting of Lemma 4.1. Let $I \subset [n]$ and let $w_{I} \in R^{n}$ be defined by $(w_{I})_{i} = [i \in I] \cdot w_{i}$ . Then

E [| S |] \geq \frac{1}{2} p {(1 - p)}^{| I | - 1} ∥ w_{I} ∥_{1} .

Proof.

Let $J = [n] ∖ I$ , $S_{1} = \sum_{i \in I} w_{i} η_{i} σ_{i}$ , and $S_{2} = \sum_{i \in J} w_{i} η_{i} σ_{i}$ . Let $E$ denote the event that $S_{1}$ and $S_{2}$ have the same sign or $S_{2} = 0$ . Then $Pr [E] \geq 1 / 2$ by symmetry. For $i \in I$ we denote by $A_{i}$ the event that ${j \in I : η_{j} \neq 0} = {i}$ . Then $Pr [A_{i}] = p (1 - p)^{| I | - 1}$ and furthermore $A_{i}$ and $E$ are independent. If $A_{i} \cap E$ occurs, then $| S | \geq | w_{i} |$ and as the events $(A_{i} \cap E)_{i \in I}$ are disjoint it thus follows that

∎

4.1 One hash-function

We are now ready to commence our analysis of Count-Sketch for Zipfians. As in the discussion succeeding Theorem 3.1 the following theorem yields the desired result for a single hash function as presented in Table 1.

Theorem 4.3.

Suppose that $B \leq n$ and let $h : [n] \to [B]$ and $s : [n] \to {- 1, 1}$ be truly random hash functions. Define the random variable $_{i} = \sum_{j \in [n]} [h (j) = h (i)] s (j) f_{j}$ for $i \in [n]$ . Then

E [| {~ f}_{i} - s (i) f_{i} |] = O (\frac{log B}{B}) .

Proof.

Let $i \in [n]$ be fixed. We start by defining $N_{1} = [B] ∖ {i}$ and $N_{2} = [n] ∖ ([B] \cup {i})$ and note that

Using the triangle inequality

E [X_{1}] \leq \frac{1}{B} \sum j \in N_{1} f_{j} = O (\frac{log B}{B}) .

Also, by Lemma 4.1, $E [X_{2}] = O (\frac{1}{B})$ and combining the two bounds we obtain the desired upper bound.

For the lower bound we apply Lemma 4.2 with $I = N_{1}$ concluding that

∎

4.2 Multiple hash functions

Let $k \in N$ be odd. For a tuple $x = (x_{1}, \dots, x_{k}) \in R^{k}$ we denote by $m e d i a n x$ the median of the entries of $x$ . The following theorem immediately leads to the result on CS with $k \geq 3$ hash functions claimed in Table 1.

Theorem 4.4.

Let $k \geq 3$ be odd, $n \geq k B$ , and $h_{1}, \dots, h_{k} : [n] \to [B]$ and $s_{1}, \dots, s_{k} : [n] \to {- 1, 1}$ be truly random hash functions. Define the random variable $_{i} = {m e d i a n}_{ℓ \in [k]} (\sum_{j \in [n]} [h_{ℓ} (j) = h_{ℓ} (i)] s_{ℓ} (j) f_{j})$ for $i \in [n]$ . Assume that⁵⁵5This very mild assumption can probably be removed at the cost of a more technical proof. In our proof it can even be replaced by $k \leq B^{2 - ε}$ for any $ε = Ω (1)$ . $k \leq B$ . Then

E [|_{i} - s (i) f_{i} |] = Ω (\frac{1}{B \sqrt{k} log k}), and E [Z] = O (\frac{1}{B \sqrt{k}})

The assumption $n \geq k B$ simply says that the total number of buckets is upper bounded by the number of items. Again using linearity of expectation for the summation over $i \in [n]$ and replacing $B$ by $B / k$ we obtain the claimed upper and lower bounds of $\frac{\sqrt{k}}{B log k}$ and $\frac{\sqrt{k}}{B}$ respectively. We note that even if the bounds above are only tight up to a factor of $log k$ they still imply that it is asymptotically optimal to choose $k = O (1)$ , e.g. $k = 3$ . To settle the correct asymptotic growth is thus of merely theoretical interest.

Proof.

If $B$ (and hence $k$ ) is a constant the results follows easily from Lemma 4.1, so in what follows we may assume that $B$ is larger than a sufficiently large constant.

We first prove the upper bound. Define $N_{1} = [B] ∖ {i}$ and $N_{2} = [n] ∖ ([B] \cup {i})$ . Let for $ℓ \in [k]$ , $X_{1}^{(ℓ)} = \sum_{j \in N_{1}} [h_{ℓ} (j) = h_{ℓ} (i)] s_{ℓ} (j) f_{j}$ and $X_{2}^{(ℓ)} = \sum_{j \in N_{2}} [h_{ℓ} (j) = h_{ℓ} (i)] s_{ℓ} (j) f_{j}$ . Finally write $X^{(ℓ)} = X_{1}^{(ℓ)} + X_{2}^{(ℓ)}$ .

As the absolute error in Count-Sketch with one pair of hash functions $(h, s)$ is always upper bounded by the corresponding error in Count-Min with the single hash function $h$ , we can use the bound in the proof of Lemma 3.2 to conclude that

Pr [| X_{1}^{(ℓ)} | \geq t] = O (\frac{log (t B)}{t B}),

when $t \geq 3 / B$ . Also

V a r [X_{2}^{ℓ}] = (\frac{1}{B} - \frac{1}{B^{2}}) \sum j \in N_{2} f_{j}^{2} \leq \frac{2}{B^{2}},

so by Bennett’s inequality (with $M = 1 / B$ and $σ^{2} = 2 / B^{2}$ ) and Remark 3.4,

Pr [| X_{2}^{(ℓ)} | \geq t] \leq 2 exp (- 2 h (t B / 2)) \leq 2 exp (- \frac{1}{2} t B log (\frac{t B}{2} + 1)) = O (\frac{log (t B)}{t B}),

for $t \geq \frac{3}{B}$ . It follows that for $t \geq 3 / B$ ,

Pr [| X^{(ℓ)} | \geq 2 t] \leq Pr [(| X_{1}^{(ℓ)} | \geq t)] + Pr (| X_{2}^{(ℓ)} | \geq t)] = O (\frac{log (t B)}{t B}) .

Let $C$ be the implicit constant in the $O$ -notation above. If $|_{i} - s (i) f_{i} | \geq 2 t$ , at least half of the values $(| X^{(ℓ)} |)_{ℓ \in [k]}$ are at least $2 t$ . For $t \geq 3 / B$ it thus follows by a union bound that

Pr [|_{i} - s (i) f_{i} | \geq 2 t] \leq 2 (\frac{k}{⌈ k / 2 ⌉}) {(C \frac{log (t B)}{t B})}^{⌈ k / 2 ⌉} \leq 2 {(4 C \frac{}{log (t B)} t B)}^{⌈ k / 2 ⌉} .

(4)

If $α = O (1)$ is chosen sufficiently large it thus holds that

	$\int_{α / B}^{\infty} Pr [\|_{i} - s (i) f_{i} \| \geq t] d t$	$= 2 \int_{α / (2 B)}^{\infty} Pr [\|_{i} - s (i) f_{i} \| \geq 2 t] d t$
		$\leq \frac{4}{B} \int_{α / 2}^{\infty} {(4 C \frac{log (t)}{t})}^{⌈ k / 2 ⌉} d t$
		$\leq \frac{1}{B 2^{k}} \leq \frac{1}{B \sqrt{k}} .$

Here the first inequality uses eq. 4 and a change of variable. The second inequality uses that ${(4 C \frac{log t}{t})}^{⌈ k / 2 ⌉} \leq (C^{'} / t)^{2 k / 5}$ for some constant $C^{'}$ followed by a calculation of the integral. For our upper bound it therefore suffices to show that $\int_{0}^{α / B} Pr [|_{i} - s (i) f_{i} | \geq t] d t = O (\frac{1}{B \sqrt{k}})$ . For this we need the following claim:

Claim 4.5.

Let $I \subset R$ be the closed interval centered at the origin of length $2 t$ , i.e., $I = [- t, t]$ . Suppose that $\frac{1}{\sqrt{k} B} \leq t \leq \frac{1}{2 B}$ . For $ℓ \in [k]$ , $Pr [X^{(ℓ)} \in I] = Ω (t B)$ .

Before proving this claim we will first show how to use it to establish the desired result. For this let $\frac{1}{\sqrt{k} B} \leq t \leq \frac{1}{B}$ be fixed. If $|_{i} - s (i) f_{i} | \geq t$ , at least half of the values $(X^{(ℓ)})_{ℓ \in [k]}$ are at least $t$ or at most $- t$ . Let us focus on bounding the probability that at least half are at least $t$ , the other bound being symmetric giving an extra factor of $2$ in the probability bound. By symmetry and the claim, $Pr [X^{(ℓ)} \geq t] = \frac{1}{2} - Ω (t B)$ . For $ℓ \in [k]$ we define $Y_{ℓ} = [X^{(ℓ)} \geq t]$ , and we put $S = \sum_{ℓ \in [k]} Y_{ℓ}$ . Then $E [S] = k (\frac{1}{2} - Ω (t B))$ . If at least half of the values $(X^{(ℓ)})_{ℓ \in [k]}$ are at least $t$ then $S \geq k / 2$ . By Hoeffding’s inequality we can bound the probability of this event by

Pr [S \geq k / 2] = Pr [S - E [S] = Ω (k t B)] = exp (- Ω (k t^{2} B^{2})) .

It follows that $Pr [|_{i} - s (i) f_{i} | \geq t] \leq 2 exp (- Ω (k t^{2} B^{2}))$ . Thus

	$\int_{0}^{α / B} Pr [\|_{i} - s (i) f_{i} \| \geq t] d t$	$\leq \frac{1}{B \sqrt{k}} + \int_{\frac{1}{B \sqrt{k}}}^{\frac{1}{2 B}} 2 exp (- Ω (k t^{2} B^{2})) d t + \int_{\frac{1}{2 B}}^{α / B} 2 exp (- Ω (k)) d t$
		$= O (\frac{1}{B \sqrt{k}}) + \frac{1}{B \sqrt{k}} \int_{1}^{\infty} exp (- t^{2}) d t = O (\frac{1}{B \sqrt{k}}) .$

It thus suffices to prove the claim.

Proof of creftypecap 4.5.

We first show that with probability $Ω (1)$ , $X_{2}^{(ℓ)}$ lies in the interval $[1 / B, γ / B]$ for some constant $γ$ . To see this we note that by Lemma 4.1, $E [| X_{2}^{(ℓ)} |] = O (\frac{1}{B})$ , so it follows by Markov’s inequality that if $γ = O (1)$ is large enough, the probability that $| X_{2}^{(ℓ)} | \geq γ / B$ is at most $\frac{1}{200}$ . For a constant probability lower bound on $| X_{2}^{(ℓ)} |$ we write

X_{2}^{(ℓ)} = \sum j \in N_{2} \cap {B + 1, \dots, 2 B} [h_{ℓ} (j) = h_{ℓ} (i)] s_{ℓ} (j) f_{j} + \sum j \in N_{2} \cap {2 B + 1, \dots, n} [h_{ℓ} (j) = h_{ℓ} (i)] s_{ℓ} (j) f_{j} := S_{1} + S_{2} .

Condition on $S_{2}$ . If $B \geq 4$ the probability that there exist exactly two $j \in N_{2} \cap {B + 1, \dots, 2 B}$ with $h_{ℓ} (j) = h_{ℓ} (i)$ is at least

With probability $1 / 4$ the corresponding signs $s_{ℓ} (j)$ are both the same as that of $S_{2}$ . By independence of $s_{ℓ}$ and $h_{ℓ}$ the probability that this occurs is at least $\frac{1}{32 e}$ and if it does, $| X_{2}^{(ℓ)} | \geq 1 / B$ . Combining these two bounds it follows that $| X_{2}^{(ℓ)} | \in [1 / B, γ / B$

(Learned) Frequency Estimation Algorithms under Zipfian Distribution

Authors

Count-Min: Optimal Estimation and Tight Error Bounds using Empirical Error Distributions

Frequency Estimation with One-Sided Error

Buffered Count-Min Sketch on SSD: Theory and Experiments

One Table to Count Them All: Parallel Frequency Estimation on Single-Board Computers

SALSA: Self-Adjusting Lean Streaming Analytics

Effective and Sparse Count-Sketch via k-means clustering

A Bayesian nonparametric approach to count-min sketch under power-law data streams

This week in AI

1 Introduction

1.1 Our results

1.2 Related work

1.3 Our techniques

1.4 Structure of the paper

2 Preliminaries

Count-Min.

Count-Sketch.

Remark 2.1.

Estimation Error.

Remark 2.2.

Zipfian Distribution

Learning Augmented Sketching Algorithms for Frequency Estimation.

3 Tight Bounds for Count-Min with Zipfians

Theorem 3.1.

Lemma 3.2.

Proof.

Theorem 3.3 (Bennett’s inequality [Ben62]).

Remark 3.4.

Proof of Theorem 3.1.

4 (Nearly) tight Bounds for Count-Sketch with Zipfians

Lemma 4.1.

Proof.

Lemma 4.2.

Proof.

4.1 One hash-function

Theorem 4.3.

Proof.

4.2 Multiple hash functions

Theorem 4.4.

Proof.

Claim 4.5.

Proof of creftypecap 4.5.