Tell Me Something I Don't Know: Randomization Strategies for Iterative Data Mining

by   Sami Hanhijärvi, et al.
Aalto University Foundation

There is a wide variety of data mining methods available, and it is generally useful in exploratory data analysis to use many different methods for the same dataset. This, however, leads to the problem of whether the results found by one method are a reflection of the phenomenon shown by the results of another method, or whether the results depict in some sense unrelated properties of the data. For example, using clustering can give indication of a clear cluster structure, and computing correlations between variables can show that there are many significant correlations in the data. However, it can be the case that the correlations are actually determined by the cluster structure. In this paper, we consider the problem of randomizing data so that previously discovered patterns or models are taken into account. The randomization methods can be used in iterative data mining. At each step in the data mining process, the randomization produces random samples from the set of data matrices satisfying the already discovered patterns or models. That is, given a data set and some statistics (e.g., cluster centers or co-occurrence counts) of the data, the randomization methods sample data sets having similar values of the given statistics as the original data set. We use Metropolis sampling based on local swaps to achieve this. We describe experiments on real data that demonstrate the usefulness of our approach. Our results indicate that in many cases, the results of, e.g., clustering actually imply the results of, say, frequent pattern discovery.


page 1

page 2

page 3

page 4


Multi-source Data Mining for e-Learning

Data mining is the task of discovering interesting, unexpected or valuab...

A Survey of Some Density Based Clustering Techniques

Density Based Clustering are a type of Clustering methods using in data ...

Explicit probabilistic models for databases and networks

Recent work in data mining and related areas has highlighted the importa...

Comparing Apples and Oranges: Measuring Differences between Exploratory Data Mining Results

Deciding whether the results of two different mining algorithms provide ...

Using Background Knowledge to Rank Itemsets

Assessing the quality of discovered results is an important open problem...

Learning what matters - Sampling interesting patterns

In the field of exploratory data mining, local structure in data can be ...

An Effective and Efficient Approach for Clusterability Evaluation

Clustering is an essential data mining tool that aims to discover inhere...

Please sign up or login with your details

Forgot password? Click here to reset