Log In Sign Up

Big Variates: Visualizing and identifying key variables in a multivariate world

by   S. J. Watts, et al.

Big Data involves both a large number of events but also many variables. This paper will concentrate on the challenge presented by the large number of variables in a Big Dataset. It will start with a brief review of exploratory data visualisation for large dimensional datasets and the use of parallel coordinates. This motivates the use of information theoretic ideas to understand multivariate data. Two key information-theoretic statistics (Similarity Index and Class Distance Indicator) will be described which are used to identify the key variables and then guide the user in a subsequent machine learning analysis. Key to the approach is a novel algorithm to histogram data which quantifies the information content of the data. The Class Distance Indicator also sets a limit on the classification performance of machine learning algorithms for the specific dataset.


page 13

page 14

page 15

page 16


Undecidability of Underfitting in Learning Algorithms

Using recent machine learning results that present an information-theore...

An Information-Theoretic Perspective on Overfitting and Underfitting

We present an information-theoretic framework for understanding overfitt...

A Frequency Scaling based Performance Indicator Framework for Big Data Systems

It is important for big data systems to identify their performance bottl...

Adaptive Sampling for Rapidly Matching Histograms

In exploratory data analysis, analysts often have a need to identify his...

Generalized Compression Dictionary Distance as Universal Similarity Measure

We present a new similarity measure based on information theoretic measu...

Multivariate Pointwise Information-Driven Data Sampling and Visualization

With increasing computing capabilities of modern supercomputers, the siz...