Optimal streaming and tracking distinct elements with high probability
share
The distinct elements problem is one of the fundamental problems in streaming algorithms --- given a stream of integers in the range {1,...,n}, we wish to provide a (1+ε) approximation to the number of distinct elements in the input. After a long line of research optimal solution for this problem with constant probability of success, using O(1/ε^2+ n) bits of space, was given by Kane, Nelson and Woodruff in 2010. The standard approach used in order to achieve low failure probability δ, is to take a median of δ^-1 parallel repetitions of the original algorithm and report the median of computed answers. We show that such a multiplicative space blow-up is unnecessary: we provide an optimal algorithm using O(δ^-1/ε^2 + n) bits of space --- matching known lower bounds for this problem. That is, the δ^-1 factor does not multiply the n term. This settles completely the space complexity of the distinct elements problem with respect to all standard parameters. We consider also strong tracking (or continuous monitoring) variant of the distinct elements problem, where we want an algorithm which provides an approximation of the number of distinct elements seen so far, at all times of the stream. We show that this variant can be solved using O( n + δ^-1/ε^2 + n) bits of space, which we show to be optimal.
READ FULL TEXT