Quantile Tracking in Dynamically Varying Data Streams Using a Generalized Exponentially Weighted Average of Observations

01/15/2019
by   Hugo Lewi Hammer, et al.
0

The Exponentially Weighted Average (EWA) of observations is known to be state-of-art estimator for tracking expectations of dynamically varying data stream distributions. However, how to devise an EWA estimator to rather track quantiles of data stream distributions is not obvious. In this paper, we present a lightweight quantile estimator using a generalized form of the EWA. To the best of our knowledge, this work represents the first reported quantile estimator of this form in the literature. An appealing property of the estimator is that the update step size is adjusted online proportionally to the difference between current observation and the current quantile estimate. Thus, if the estimator is off-track compared to the data stream, large steps will be taken to promptly get the estimator back on-track. The convergence of the estimator to the true quantile is proven using the theory of stochastic learning. Extensive experimental results using both synthetic and real-life data show that our estimator clearly outperforms legacy state-of-the-art quantile tracking estimators and achieves faster adaptivity in dynamic environments. The quantile estimator was further tested on real-life data where the objective is efficient online control of indoor climate. We show that the estimator can be incorporated into a concept drift detector for efficiently decide when a machine learning model used to predict future indoor temperature should be retrained/updated.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/13/2019

Joint Tracking of Multiple Quantiles Through Conditional Quantiles

Estimation of quantiles is one of the most fundamental real-time analysi...
research
01/15/2019

Parameter Estimation in Abruptly Changing Dynamic Environments

Many real-life dynamical systems change abruptly followed by almost stat...
research
04/27/2020

Efficient Quantile Tracking Using an Oracle

For incremental quantile estimators the step size and possibly other tun...
research
01/08/2020

Estimating Tukey Depth Using Incremental Quantile Estimators

The concept of depth represents methods to measure how deep an arbitrary...
research
11/23/2021

Trimmed Harrell-Davis quantile estimator based on the highest density interval of the given width

Traditional quantile estimators that are based on one or two order stati...
research
12/20/2022

Probabilistic quantile factor analysis

This paper extends quantile factor analysis to a probabilistic variant t...
research
03/20/2019

Large-Scale Online Experimentation with Quantile Metrics

Online experimentation (or A/B testing) has been widely adopted in indus...

Please sign up or login with your details

Forgot password? Click here to reset