Tree structured sparse coding on cubes

1 The construction on the cube

1.1 Setup

We are given $N$ data points in $B^{d} = {- 1, 1}^{d}$ written as the $d \times N$ binary matrix $X$ . Our goal is to decompose $X$ as a tree of subcubes and “subcube corrections”. A $q$ dimensional subcube $C = C_{c, I^{r}}$ of $B^{d}$ is determined by a point $c \in B^{d}$ , along with a set of $d - q$ restricted indices $I^{r} = r_{1}, . . ., r_{d - q}$ . The cube $C_{c, I^{r}}$ consists of the points $b \in B^{d}$ such that $b_{r_{i}} = c_{r_{i}}$ for all $r_{i} \in I^{r}$ , that is

C_{c, I^{r}} = {b \in B^{d} s.t. b_{r_{i}} = c_{r_{i}} \forall r_{i} \in I^{r}} .

The unrestricted indices $I^{u} = {1, . . ., d} ∖ I^{r}$ can take on either value.

1.2 The construction

Here I will describe a simple version of the construction where each node in the tree corresponds to a subcube of the same dimension $q$ , and a hard binary clustering is used at each stage. Suppose our tree has depth $l$ . Then the construction consists of

A tree structured clustering of $X$ into sets $X_{i j}$ at depth (scale) $i \in {1, . . ., l}$ such that

$⋃ j X_{i j} = X,$
and cluster representatives (that is $d - i q$ -dimensional subcubes)

$C_{c_{i j}, I_{i j}^{r}}$

such that the restricted sets have the property that if $i j$ is an ancestor of $i^{'} j^{'}$ ,

$I_{i j}^{r} \cap I_{i^{'} j^{'}}^{r} = I_{i j}^{r},$

and

$c_{i j} (s) = c_{i^{'} j^{'}} (s)$

for all $s \in I_{i j}^{r}$

Here each $c_{i j}$

is a vector in

B^{d}

; the complete set of

c_{i j}

roughly corresponds to

W

from before. However, note that each

c_{i j}

has precisely

d - i q

entries that actually matter; and moreover because of the nested equalities, the leaf nodes carry all the information on the branch. This is not to say that the tree structure is not important or not used- it is, as the leaf nodes have to share coordinates. However once the full construction is specified, the leaf representatives are all that is necessary to code a data point.

1.3 Algorithms

We can build the partitions and representatives starting from the root and descending down the tree as follows: first, find the best fit $d - q$ dimensional subcube for the whole data set. This is given by a coordinate-wise mode; the free coordinates are the ones with the largest average discrepancy from their modes. Remove the $q$ fixed coordinates from consideration. Cluster the reduced ( $d - q$ dimensional) data using $K$ means with $K = 2$ ; on each cluster find the best fit $(d - q) - q$ cube. Continue to the leaves.

1.3.1 Refinement

The terms $C_{c_{i j}, I_{i j}^{r}}$ and $X_{i j}$ can be updated with a Lloyd type alternation. With all of the $X_{i j}$ fixed, loop through each $C$ from the root of the tree finding the best subcubes at each scale for the current partition. Now update the partition so that each $x$ is sent to its best fit leaf cube.

1.3.2 Adaptive $q$ , $l$ , etc.

In [1], one of the important points is that many of the model parameters, including the $q$ , $l$ , and the number of clusters could be determined in a principled way. While it is possible that some of their analysis may carry over to this setting, it is not yet done. However, instead of fixing $q$ , we can fix a percentage of the energy to be kept at each level, and choose the number of free coordinates accordingly.

2 Experiments

We use a binarized the MNIST training data by thresholding to obtain

X

. Here

d = 28^{2}

and

N = 60000

. Replace

70 %

of the entries in

X

with noise sampled uniformly from

{- 1, 1}

, and train a tree structured cube dictionary with

q = 80

and depth

l = 9

. The subdivision scheme used to generate the multiscale clustering is

2

-means initialized via randomized farthest insertion [2]; this means we can cycle spin over the dictionaries [5], to get many different reconstructions to average over. In this experiment the reconstruction was preformed 50 times for the noise realization. The results are visualized below.

Figure 1: Results of denoising using the tree structured coding. The top left image is the first 64 binarized MNIST digits after replacing $70 %$ of the data matrix with uniform noise. The top right image is recovered, using a binary tree of depth $l = 9$ and $q = 90$ , and 100 cycle spins, thus the non-binary output, as the final result is the average of the random clustering initialization (of course with the same noise realization). The bottom left image is recovered using robust pca [4], for comparison. The bottom right is the true binary data.

References

[1] W. Allard, G. Chen, and M. Maggioni. Multiscale geometric methods for data sets II: Geometric multi-resolution analysis. to appear in Applied and Computational Harmonic Analysis.
[2] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, SODA ’07, pages 1027–1035, Philadelphia, PA, USA, 2007. Society for Industrial and Applied Mathematics.
[3] Richard G. Baraniuk, Volkan Cevher, Marco F. Duarte, and Chinmay Hegde. Model-Based Compressive Sensing. Dec 2009.
[4] Emmanuel J. Candès, Xiaodong Li, Yi Ma, and John Wright.
Robust principal component analysis?
J. ACM, 58(3):11, 2011.
[5] R. R. Coifman and D. L. Donoho. Translation-invariant de-noising. Technical report, Department of Statistics, 1995.
[6] G. David and S. Semmes. Singular integrals and rectifiable sets in $R^{n}$ : au-delà des graphes Lipschitziens. Astérisque, 193:1–145, 1991.
[7] Laurent Jacob, Guillaume Obozinski, and Jean-Philippe Vert. Group lasso with overlap and graph lasso. In
Proceedings of the 26th Annual International Conference on Machine Learning
, ICML ’09, pages 433–440, New York, NY, USA, 2009. ACM.
[8] R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for sparse hierarchical dictionary learning. In International Conference on Machine Learning (ICML), 2010.
[9] P. W. Jones. Rectifiable sets and the traveling salesman problem. Invent Math, 102(1):1–15, 1990.
[10] Seyoung Kim and Eric P. Xing. Tree-guided group lasso for multi-task regression with structured sparsity. In ICML, pages 543–550, 2010.
[11] Gilad Lerman. Quantifying curvelike structures of measures by using $L_{2}$ Jones quantities. Comm. Pure Appl. Math., 56(9):1294–1365, 2003.

Tree structured sparse coding on cubes

Authors

Fast approximations to structured sparse coding and applications to object classification

Learning Efficient Structured Sparse Models

Single-Component Privacy Guarantees in Helper Data Systems and Sparse Coding

Querying collections of tree-structured records in the presence of within-record referential constraints

Restoring STM images via Sparse Coding: noise and artifact removal

From a Modified Ambrosio-Tortorelli to a Randomized Part Hierarchy Tree

Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

1 The construction on the cube

1.1 Setup

1.2 The construction

1.3 Algorithms

1.3.1 Refinement

1.3.2 Adaptive $q$ , $l$ , etc.

2 Experiments

References

Tree structured sparse coding on cubes

Authors

Fast approximations to structured sparse coding and applications to object classification

Learning Efficient Structured Sparse Models

Single-Component Privacy Guarantees in Helper Data Systems and Sparse Coding

Querying collections of tree-structured records in the presence of within-record referential constraints

Restoring STM images via Sparse Coding: noise and artifact removal

From a Modified Ambrosio-Tortorelli to a Randomized Part Hierarchy Tree

Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

This week in AI

1 The construction on the cube

1.1 Setup

1.2 The construction

1.3 Algorithms

1.3.1 Refinement

1.3.2 Adaptive q, l, etc.

2 Experiments

References

1.3.2 Adaptive $q$ , $l$ , etc.