A New Framework for Distance and Kernel-based Metrics in High Dimensions

09/30/2019
by   Shubhadeep Chakraborty, et al.
0

The paper presents new metrics to quantify and test for (i) the equality of distributions and (ii) the independence between two high-dimensional random vectors. We show that the energy distance based on the usual Euclidean distance cannot completely characterize the homogeneity of two high-dimensional distributions in the sense that it only detects the equality of means and the traces of covariance matrices in the high-dimensional setup. We propose a new class of metrics which inherits the desirable properties of the energy distance and maximum mean discrepancy/(generalized) distance covariance and the Hilbert-Schmidt Independence Criterion in the low-dimensional setting and is capable of detecting the homogeneity of/completely characterizing independence between the low-dimensional marginal distributions in the high dimensional setup. We further propose t-tests based on the new metrics to perform high-dimensional two-sample testing/independence testing and study their asymptotic behavior under both high dimension low sample size (HDLSS) and high dimension medium sample size (HDMSS) setups. The computational complexity of the t-tests only grows linearly with the dimension and thus is scalable to very high dimensional data. We demonstrate the superior power behavior of the proposed tests for homogeneity of distributions and independence via both simulated and real datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/19/2019

Interpoint Distance Based Two Sample Tests in High Dimension

In this paper, we study a class of two sample test statistics based on i...
research
02/08/2019

Distance-based and RKHS-based Dependence Metrics in High Dimension

In this paper, we study distance covariance, Hilbert-Schmidt covariance ...
research
06/06/2018

On high-dimensional modifications of some graph-based two-sample tests

Testing for the equality of two high-dimensional distributions is a chal...
research
01/04/2020

High-Dimensional Independence Testing and Maximum Marginal Correlation

A number of universally consistent dependence measures have been recentl...
research
06/18/2018

MultiFIT: A Multivariate Multiscale Framework for Independence Tests

We present a framework for testing independence between two random vecto...
research
11/28/2022

Double Data Piling for Heterogeneous Covariance Models

In this work, we characterize two data piling phenomenon for a high-dime...
research
12/16/2022

On High Dimensional Behaviour of Some Two-Sample Tests Based on Ball Divergence

In this article, we propose some two-sample tests based on ball divergen...

Please sign up or login with your details

Forgot password? Click here to reset