
Renormalized Mutual Information for Extraction of Continuous Features
We derive a welldefined renormalized version of mutual information that...
read it

Allrelevant feature selection using multidimensional filters with exhaustive search
This paper describes a method for identification of the informative vari...
read it

Relevant feature extraction for statistical inference
We introduce an algorithm that learns correlations between two datasets,...
read it

A multiplicative masking method for preserving the skewness of the original microrecords
Masking methods for the safe dissemination of microdata consist of disto...
read it

Trimming the Independent Fat: Sufficient Statistics, Mutual Information, and Predictability from Effective Channel States
One of the most fundamental questions one can ask about a pair of random...
read it

3D Face Recognition with Sparse Spherical Representations
This paper addresses the problem of 3D face recognition using simultaneo...
read it

Constructing Complexityefficient Features in XCS with Treebased Rule Conditions
A major goal of machine learning is to create techniques that abstract a...
read it
Sufficient Dimensionality Reduction with Irrelevant Statistics
The problem of finding a reduced dimensionality representation of categorical variables while preserving their most relevant characteristics is fundamental for the analysis of complex data. Specifically, given a cooccurrence matrix of two variables, one often seeks a compact representation of one variable which preserves information about the other variable. We have recently introduced "Sufficient Dimensionality Reduction' [GT2003], a method that extracts continuous reduced dimensional features whose measurements (i.e., expectation values) capture maximal mutual information among the variables. However, such measurements often capture information that is irrelevant for a given task. Widely known examples are illumination conditions, which are irrelevant as features for face recognition, writing style which is irrelevant as a feature for content classification, and intonation which is irrelevant as a feature for speech recognition. Such irrelevance cannot be deduced apriori, since it depends on the details of the task, and is thus inherently ill defined in the purely unsupervised case. Separating relevant from irrelevant features can be achieved using additional side data that contains such irrelevant structures. This approach was taken in [CT2002], extending the information bottleneck method, which uses clustering to compress the data. Here we use this sideinformation framework to identify features whose measurements are maximally informative for the original data set, but carry as little information as possible on a side data set. In statistical terms this can be understood as extracting statistics which are maximally sufficient for the original dataset, while simultaneously maximally ancillary for the side dataset. We formulate this tradeoff as a constrained optimization problem and characterize its solutions. We then derive a gradient descent algorithm for this problem, which is based on the Generalized Iterative Scaling method for finding maximum entropy distributions. The method is demonstrated on synthetic data, as well as on real face recognition datasets, and is shown to outperform standard methods such as oriented PCA.
READ FULL TEXT
Comments
There are no comments yet.