Specious rules: an efficient and effective unifying method for removing misleading and uninformative patterns in association rule mining

We present theoretical analysis and a suite of tests and procedures for addressing a broad class of redundant and misleading association rules we call specious rules. Specious dependencies, also known as spurious, apparent, or illusory associations, refer to a well-known phenomenon where marginal dependencies are merely products of interactions with other variables and disappear when conditioned on those variables. The most extreme example is Yule-Simpson's paradox where two variables present positive dependence in the marginal contingency table but negative in all partial tables defined by different levels of a confounding factor. It is accepted wisdom that in data of any nontrivial dimensionality it is infeasible to control for all of the exponentially many possible confounds of this nature. In this paper, we consider the problem of specious dependencies in the context of statistical association rule mining. We define specious rules and show they offer a unifying framework which covers many types of previously proposed redundant or misleading association rules. After theoretical analysis, we introduce practical algorithms for detecting and pruning out specious association rules efficiently under many key goodness measures, including mutual information and exact hypergeometric probabilities. We demonstrate that the procedure greatly reduces the number of associations discovered, providing an elegant and effective solution to the problem of association mining discovering large numbers of misleading and redundant rules.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/19/2018

Itemsets of interest for negative association rules

So far, most of association rule minings have considered about positive ...
research
06/20/2018

Searching of interesting itemsets for negative association rules

In this paper, we propose an algorithm of searching for both positive an...
research
03/19/2018

Mining User Behavioral Rules from Smartphone Data through Association Analysis

The increasing popularity of smart mobile phones and their powerful sens...
research
03/06/2008

New probabilistic interest measures for association rules

Mining association rules is an important technique for discovering meani...
research
01/17/2022

Patterns of near-crash events in a naturalistic driving dataset: applying rules mining

This study aims to explore the associations between near-crash events an...
research
09/12/2011

Nominal Association Vector and Matrix

When response variables are nominal and populations are cross-classified...
research
07/29/2019

Discovering Association with Copula Entropy

Discovering associations is of central importance in scientific practice...

Please sign up or login with your details

Forgot password? Click here to reset