Functional outlier detection for density-valued data with application to robustify distribution to distribution regression

10/02/2021
by   Xinyi Lei, et al.
0

Distributional data analysis, concerned with statistical analysis and modeling for data objects consisting of random probability density functions (PDFs) in the framework of functional data analysis (FDA), has received considerable interest in recent years. However, many important aspects remain unexplored, such as outlier detection and robustness. Existing functional outlier detection methods are mainly used for ordinary functional data and usually perform poorly when applied to PDFs. To fill this gap, this study focuses on PDF-valued outlier detection, as well as its application in robust distributional regression. Similar to ordinary functional data, detecting the shape outlier masked by the "curve net" formed by the bulk of the PDFs is the major challenge in PDF-outlier detection. To this end, we propose a tree-structured transformation system for feature extraction as well as converting the shape outliers to easily detectable magnitude outliers, relevant outlier detectors are designed for the specific transformed data. A multiple detection strategy is also proposed to account for detection uncertainties and to combine different detectors to form a more reliable detection tool. Moreover, we propose a distributional-regression-based approach for detecting the abnormal associations of PDF-valued two-tuples. As a specific application, the proposed outlier detection methods are applied to robustify a distribution-to-distribution regression method, and we develop a robust estimator for the regression operator by downweighting the detected outliers. The proposed methods are validated and evaluated by extensive simulation studies or real data applications. Relevant comparative studies demonstrate the superiority of the developed outlier detection method with other competitors in distributional outlier detection.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/16/2018

Functional Outlier Detection and Taxonomy by Sequential Transformations

Functional data analysis can be seriously impaired by abnormal observati...
research
06/11/2021

DORO: Distributional and Outlier Robust Optimization

Many machine learning tasks involve subpopulation shift where the testin...
research
07/02/2021

Depth-based Outlier Detection for Grouped Smart Meters: a Functional Data Analysis Toolbox

Smart metering infrastructures collect data almost continuously in the f...
research
06/18/2018

Kernel-based Outlier Detection using the Inverse Christoffel Function

Outlier detection methods have become increasingly relevant in recent ye...
research
04/22/2021

Conditional Selective Inference for Robust Regression and Outlier Detection using Piecewise-Linear Homotopy Continuation

In practical data analysis under noisy environment, it is common to firs...
research
09/14/2021

A geometric perspective on functional outlier detection

We consider functional outlier detection from a geometric perspective, s...
research
05/15/2019

Automated detection of business-relevant outliers in e-commerce conversion rate

We evaluate how modern outlier detection methods perform in identifying ...

Please sign up or login with your details

Forgot password? Click here to reset