Streaming data preprocessing via online tensor recovery for large environmental sensor networks

by   Yue Hu, et al.

Measuring the built and natural environment at a fine-grained scale is now possible with low-cost urban environmental sensor networks. However, fine-grained city-scale data analysis is complicated by tedious data cleaning including removing outliers and imputing missing data. While many methods exist to automatically correct anomalies and impute missing entries, challenges still exist on data with large spatial-temporal scales and shifting patterns. To address these challenges, we propose an online robust tensor recovery (OLRTR) method to preprocess streaming high-dimensional urban environmental datasets. A small-sized dictionary that captures the underlying patterns of the data is computed and constantly updated with new data. OLRTR enables online recovery for large-scale sensor networks that provide continuous data streams, with a lower computational memory usage compared to offline batch counterparts. In addition, we formulate the objective function so that OLRTR can detect structured outliers, such as faulty readings over a long period of time. We validate OLRTR on a synthetically degraded National Oceanic and Atmospheric Administration temperature dataset, with a recovery error of 0.05, and apply it to the Array of Things city-scale sensor network in Chicago, IL, showing superior results compared with several established online and batch-based low rank decomposition methods.


Robust Tensor Recovery with Fiber Outliers for Traffic Events

Event detection is gaining increasing attention in smart cities research...

Variational Bayesian Inference for Robust Streaming Tensor Factorization and Completion

Streaming tensor factorization is a powerful tool for processing high-vo...

Bayesian Robust Tensor Ring Model for Incomplete Multiway Data

Low-rank tensor completion aims to recover missing entries from the obse...

Scalable and Robust Tensor Ring Decomposition for Large-scale Data

Tensor ring (TR) decomposition has recently received increased attention...

Urban Rhapsody: Large-scale exploration of urban soundscapes

Noise is one of the primary quality-of-life issues in urban environments...

Bayesian tensor learning for structural monitoring data imputation and response forecasting

There has been increased interest in missing sensor data imputation, whi...

Please sign up or login with your details

Forgot password? Click here to reset