DeepAI AI Chat
Log In Sign Up

The adaptable buffer algorithm for high quantile estimation in non-stationary data streams

by   Ognjen Arandjelovic, et al.

The need to estimate a particular quantile of a distribution is an important problem which frequently arises in many computer vision and signal processing applications. For example, our work was motivated by the requirements of many semi-automatic surveillance analytics systems which detect abnormalities in close-circuit television (CCTV) footage using statistical models of low-level motion features. In this paper we specifically address the problem of estimating the running quantile of a data stream with non-stationary stochasticity when the memory for storing observations is limited. We make several major contributions: (i) we derive an important theoretical result which shows that the change in the quantile of a stream is constrained regardless of the stochastic properties of data, (ii) we describe a set of high-level design goals for an effective estimation algorithm that emerge as a consequence of our theoretical findings, (iii) we introduce a novel algorithm which implements the aforementioned design goals by retaining a sample of data values in a manner adaptive to changes in the distribution of data and progressively narrowing down its focus in the periods of quasi-stationary stochasticity, and (iv) we present a comprehensive evaluation of the proposed algorithm and compare it with the existing methods in the literature on both synthetic data sets and three large `real-world' streams acquired in the course of operation of an existing commercial surveillance system. Our findings convincingly demonstrate that the proposed method is highly successful and vastly outperforms the existing alternatives, especially when the target quantile is high valued and the available buffer capacity severely limited.


page 1

page 2

page 3

page 4


Viewpoint distortion compensation in practical surveillance systems

Our aim is to estimate the perspective-effected geometric distortion of ...

Hybrid quantile estimation for asymmetric power GARCH models

Asymmetric power GARCH models have been widely used to study the higher ...

An Embedded Model Estimator for Non-Stationary Random Functions using Multiple Secondary Variables

An algorithm for non-stationary spatial modelling using multiple seconda...

A Clustering-based Framework for Classifying Data Streams

The non-stationary nature of data streams strongly challenges traditiona...

Quantile sheet estimator with shape constraints

A quantile sheet is a global estimator for multiple quantile curves. A q...

Efficient Quantile Tracking Using an Oracle

For incremental quantile estimators the step size and possibly other tun...

SQUAD: Combining Sketching and Sampling Is Better than Either for Per-item Quantile Estimation

Stream monitoring is fundamental in many data stream applications, such ...