A feature-based framework for detecting technical outliers in water-quality data from in situ sensors

02/17/2019
by   Priyanga Dilini Talagala, et al.
0

Outliers due to technical errors in water-quality data from in situ sensors can reduce data quality and have a direct impact on inference drawn from subsequent data analysis. However, outlier detection through manual monitoring is unfeasible given the volume and velocity of data the sensors produce. Here, we proposed an automated framework that provides early detection of outliers in water-quality data from in situ sensors caused by technical issues.The framework was used first to identify the data features that differentiate outlying instances from typical behaviours. Then statistical transformations were applied to make the outlying instances stand out in transformed data space. Unsupervised outlier scoring techniques were then applied to the transformed data space and an approach based on extreme value theory was used to calculate a threshold for each potential outlier. Using two data sets obtained from in situ sensors in rivers flowing into the Great Barrier Reef lagoon, Australia, we showed that the proposed framework successfully identified outliers involving abrupt changes in turbidity, conductivity and river level, including sudden spikes, sudden isolated drops and level shifts, while maintaining very low false detection rates. We implemented this framework in the open source R package oddwater.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 14

page 17

page 18

page 19

page 20

11/16/2020

EventDetectR – An Open-Source Event Detection System

EventDetectR: An efficient Event Detection System (EDS) capable of detec...
10/31/2018

A framework for automated anomaly detection in high frequency water-quality data from in situ sensors

River water-quality monitoring is increasingly conducted using automated...
03/21/2021

Homophily Outlier Detection in Non-IID Categorical Data

Most of existing outlier detection methods assume that the outlier facto...
12/21/2016

Detecting Unusual Input-Output Associations in Multivariate Conditional Data

Despite tremendous progress in outlier detection research in recent year...
05/15/2019

Automated detection of business-relevant outliers in e-commerce conversion rate

We evaluate how modern outlier detection methods perform in identifying ...
06/09/2021

Deep Clustering based Fair Outlier Detection

In this paper, we focus on the fairness issues regarding unsupervised ou...
11/02/2021

Informative Planning in the Presence of Outliers

Informative planning seeks a sequence of actions that guide the robot to...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.