High-Dimensional Screening Using Multiple Grouping of Variables

08/09/2012 ∙ by Divyanshu Vats, et al. ∙ Rice University 0

Screening is the problem of finding a superset of the set of non-zero entries in an unknown p-dimensional vector β given n noisy observations. Naturally, we want this superset to be as small as possible. We propose a novel framework for screening, which we refer to as Multiple Grouping (MuG), that groups variables, performs variable selection over the groups, and repeats this process multiple number of times to estimate a sequence of sets that contains the non-zero entries in β. Screening is done by taking an intersection of all these estimated sets. The MuG framework can be used in conjunction with any group based variable selection algorithm. In the high-dimensional setting, where p >> n, we show that when MuG is used with the group Lasso estimator, screening can be consistently performed without using any tuning parameter. Our numerical simulations clearly show the merits of using the MuG framework in practice.



There are no comments yet.


page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Let be an unknown -dimensional sparse vector with non-zero entries. Let denote the support of , i.e., the location of the non-zero entries in . Let be a known -dimensional vector that captures information about using the linear model


where is a known design matrix and is measurement noise. Throughout this paper, we assume that , i.e., the number of measurements available is smaller than the ambient dimensionality of .

Equation (1) is well studied in the literature owing to its application in many real world problems. For example, in compressive sensing, it is of interest to measure a signal using only a few measurements with a suitable choice of the design matrix [1, 2]. Given gene expression data, where typically the number of observations is much smaller than the total number of genes , it is of interest to study the relationships between genes [3]. These relationships are captured by the non-zero entries of the vector . A similar problem of estimating relationships arises when modeling economic data [4].

In this paper, we study the problem of finding a superset of the support of so that the number of elements in is less than and as close to as possible. In the literature, this problem is often referred to as variable screening or simply screening. Algorithms for screening are useful in reducing the dimensionality of from to . This allows practitioners to focus subsequent analysis on a much smaller set rather than dealing with the large set of variables.

I-a Main Contributions

Multiple Grouping (MuG): We propose a general framework for screening that groups variables, performs variable selection over the groups, and repeats this process multiple number of times over different choices of the groupings. The final estimated superset is the intersection of the supports estimated over each grouping. We refer to our framework as Multiple Grouping (MuG). The main intuition behind MuG is that if a variable is selected in one iteration, it may not be selected in another iteration since the variable may be grouped with other variables that are all zero. Figure 1 illustrates MuG using a simple example.

Tuning-Free Screening: The MuG framework can be used in conjunction with any group based variable selection algorithm. We study the application of MuG with group Lasso [5], which uses a modification of the popular Lasso algorithm [6] to perform variable selection over groups. Using properties of the Lasso and the group Lasso, we show that when , MuG with group Lasso can perform screening without using a tuning parameter in such a way that . This property of MuG is extremely useful, for example, in analyzing gene expression data where it is typical for to be of the order of thousands and to be of the order of hundreds. Moreover, we identify conditions under which MuG is high-dimensional consistent so that as .

I-B Related Work

Several algorithms have been proposed in the literature for estimating the support of , see [6, 7, 8, 9, 10, 11, 12, 13] for some examples and [14] for a comprehensive review. The performance of all the known algorithms depend on the choice of a tuning parameter that controls the cardinality of the estimated support. To estimate the true support in the high-dimensional regime, where , tuning parameters, also referred to as regularization parameters, may be chosen using stability selection [15] or extended Bayesian information criterion [16]. However, both these methods depend on other parameters which are difficult to select in practice. The MuG framework can be used to eliminate a majority of the variables without using any tuning parameter and then standard model selection algorithms may be used over the remaining set variables.

It has been observed that when using cross-validation to select the tuning parameter, under appropriate conditions, the estimated support can be a superset of . However, the cardinality of the estimated support can be quite large in practice, making cross-validation based methods inappropriate for screening.

Reference [17] outlines a screening algorithm, referred to as sure independence screening (SIS), that111Assuming the columns of are normalized so that . thresholds to find the superset of . Extensions of SIS have been proposed in [18, 19, 20, 21]. The performance of SIS is sensitive to the choice of the threshold and an appropriate choice of the threshold depends on the unknown parameters of the model in (1). The main advantage of MuG over SIS is that, when , screening may be done without using a tuning parameter or a threshold. Moreover, our numerical simulations clearly show that the MuG framework can discard more variables when compared to SIS. However, we note that SIS is computationally fast, with time complexity , and can be used in conjunction with MuG to trade-off computational complexity and accuracy.

Recent works in [22, 23, 24, 25] have analyzed the solutions of the Lasso to derive rules for discarding variables when solving the Lasso for a particular tuning parameter. Our work differs from this work since we perform screening to find a superset of the true support . Regardless, when using the Lasso and the group Lasso with the MuG framework, the algorithms in [22, 23, 24, 25] can be used to improve the computational complexity of solving the Lasso and group Lasso problems.

Another approach to tuning-free screening is to use properties of variable selection algorithms such as the Lasso. It is known that the Lasso can select at most variables [26, 27]. When , this means that screening can be done using the Lasso by choosing a tuning parameter so that the cardinality of the estimated support is . In our proposed algorithm of using MuG with group Lasso (see Algorithm 2), we use the Lasso to do tuning-free screening and then use the group Lasso multiple number of times to further screen for variables. Using the property that the group Lasso can only select at most groups, we are able to perform tuning-free screening. Our numerical simulations clearly show the improvements of using MuG versus simply using Lasso for screening.

I-C Paper Organization

The rest of the paper is organized as follows.

  • Section II presents the MuG framework with respect to an abstract group based variable selection algorithm.

  • Section III shows how MuG can be used with the group Lasso estimator.

  • Section IV outlines conditions under which MuG leads to a screening algorithm that is high-dimensional consistent.

  • Section V presents numerical simulations that show the advantages of using MuG in practice and compares MuG to other screening algorithms.

  • Section VI discusses some extensions of MuG.

  • Section VII summarizes the paper.

Ii Overview of Multiple Grouping


Fig. 1: An illustration of the Multiple Grouping (MuG) framework. The true support is and the estimated superset is .

In this Section, we give an overview of the Multiple Grouping (MuG) framework when used in conjunction with an abstract variable selection algorithm. Let index defined in (1). Define a collection of partitions or groupings of :


We have assumed that each group has at least one element and at most elements, where is small when compared to and . Moreover, the groups in a grouping are chosen such that they are disjoint and all elements in are mapped to a group in . Let be a generic variable selection algorithm:


The set is an estimate of the true support . We assume that, under certain conditions, can select all groups such that . The multiple grouping (MuG) framework for variable selection is to apply the variable selection algorithm over multiple groupings to obtain a sequence of estimates . The final estimated superset of the support is the intersection of all the estimates. Algorithm 1 summarizes the MuG framework and Figure 1 illustrates MuG using .

Compute for using (4).
Return .
Algorithm 1 Multiple Grouping (MuG)

Typical applications of group based variable selection algorithms assume that it is known a priori which groups of variables in are non-zero or zero. Our setting is different since we assume that is sparse (and not necessarily group sparse) and group variables to estimate a superset of the true support. Since the groupings can be chosen arbitrarily, we repeat this process multiple number of times using different groupings and take an intersection over all the estimates to find the final superset of the true support.

Relation to other methods: By applying a variable selection algorithm multiple number of times using different groupings, we are introducing randomness into the variable selection algorithm. Past ways of introducing randomness have relied on subsampling [28, 15] or random perturbations of the design matrix [15]. Our approach of using multiple groupings is a new method for introducing randomness into algorithms for improved performance.

Choosing : The parameter controls the number of variables selected in each iteration of the MuG framework. We want to choose so that all variables in are included in each

with high probability while

is as small as possible. This will ensure that with high probability. One way of doing this is by carefully choosing using some model selection algorithm, such as cross-validation, stability selection, of information criterion based methods. However, this can be computationally challenging. An alternative approach is to assume an upper bound, say , for the unknown sparsity level and choose in each iteration of MuG to select groups. As shown in [27], the group Lasso can only select at most groups, so choosing when using the group Lasso allows for tuning-free screening. We discuss this algorithm in Section III.

Choosing : The parameter controls the number of groupings we form in the MuG framework. It is clear that decreases or remains the same as increases. However, we do not want to be too large since, with small probability, there may exist a grouping for which we may discard an element in . On the other hand, choosing to be too small may not result in significant reduction in dimensionality. We show that when using MuG with group Lasso, choosing such that is sufficient to ensure consistency of the screening algorithm. Thus, choosing of the order of is sufficient to ensure that screening can be performed with high probability. In practice, can be chosen depending on the available computational resources.

Choosing : We discuss two methods for choosing . The first method chooses by randomly partitioning the set of indices . The second method chooses using the estimates . Our numerical simulations compare both these methods and also discusses the trade-offs in choosing , i.e., the maximum size of the groups in .

Iii MuG Using Group Lasso

So far, we have given an overview of the Multiple Grouping (MuG) framework using an abstract variable selection algorithm. In this Section, we show how MuG can be used with the group Lasso estimator, which was proposed in [5] as an extension to the Lasso for variable selection and prediction given prior knowledge about groups of variables that are either zero or non-zero. Section III-A outlines the MuG framework using the group Lasso. Section III-B presents an algorithm for grouping variables and empirically evaluates the algorithm using a simple numerical example.

Iii-a Tuning-Free Screening

Let be a grouping defined in (2)-(3) with groups. The weighted norm, using the grouping , is defined as follows:


where . The group Lasso, first proposed in [5], solves the following optimization problem:


When , the group Lasso reduces to the classical Lasso estimator [6]. Let the support over the groups be such that


In other words, is the set of groups in selected by the group Lasso estimator. The following Lemma, proved in [27], characterizes the cardinality of .

Lemma 1 ([27])

For all , , where is the number of groups in the grouping .

  • Solve (10) using grouping and choose the tuning parameter such that222If variables can not be selected, the chosen will be . Similarly, for the group Lasso, if groups can not be selected, the chosen will be ..

  • Initialize .

  • For

    • Choose a grouping that satisfies (2)-(3) and .

    • Solve (10) using and choose s.t.

    • Let be the support of the group Lasso estimator and update : .

Algorithm 2 Tuning-Free Screening

Using Lemma 1, we see that the Lasso can select at most variables and the group Lasso can select at most groups of variables. When , we can easily perform screening by solving the Lasso to select at most variables.

Using MuG, we may further reduce the dimensionality of the problem. Algorithm 2 outlines the MuG framework when used in conjunction with the group Lasso estimator in (10). We first solve the Lasso by choosing a that selects at most variables. If variables can not be selected, we select the maximum number of variables the Lasso can select. Next, we solve the group Lasso for multiple different choices of the groupings in such a way that at most groups are selected. Again, if groups can not be selected, we choose the maximum number of groups possible. The final step is to take an intersection over all the supports to find an estimate . The algorithm is tuning-free since we specify exactly how the tuning parameters are chosen in the algorithm. We note that although Algorithm 2 depends on the parameters (number of iterations) and (maximum size of the groups), both these parameters can be easily chosen to allow screening. We refer to Section V for more details.

When using standard implementations of the Lasso and the group Lasso, it may not be computationally feasible for all solutions of the Lasso to have support of size less than or equal to . Thus, in practice, we apply the Lasso for multiple different values of and choose a for which the estimated support is the smallest above . A similar step is done for the group Lasso solution. If we apply group Lasso for different tuning parameters, the worst case computational complexity of Algorithm 2 is times the complexity of computing group Lasso solutions. In practice, once a suitable tuning parameter has been selected in one iteration, subsequent tuning parameters becomes easier to estimate. Moreover, different approximations to group Lasso, such as group LARS [5], may be used to speed up computations.

Iii-B Choosing the Groupings

Fig. 2: Illustration of the adaptive grouping algorithm where we either group one variable from with variables from or group variables in together.

This Section addresses the problem of choosing the groupings in the MuG framework. We consider two methods.

Random Groupings: Partition the index set randomly such that each group in has at most elements.

Adaptive Groupings: Let be the current estimate after using MuG times. Note that all variables in are estimated to be zero in . To construct the grouping , randomly group an element in with at most elements from . This allows for grouping an element from that is possibly zero with other elements that are estimated to be zero. Once all elements in have been grouped, randomly group the remaining elements in groups of size at most . Fig. 2 illustrates this adaptive construction.

To compare the performance of the two grouping algorithms, we consider a simple example. Consider the linear model in (1) with , , , and . Suppose all non-zero elements in have magnitude and each entry in

is sampled independently from a standard normal distribution. Note that the MuG framework is sensitive to the choice of the

groupings. A different choice of the groupings may result in a different output . To study the properties of , we fix and and apply Algorithm 2 times over different choices of the set of groupings for both random and adaptive groupings.

(a) Random Grouping
(b) Adaptive Grouping
Fig. 3: Histogram of when applying MuG times over different choices of the set of groupings. The parameters are , , , , , and . In (a), we choose the groupings by randomly partitioning the index set in groups of size . In (b), we adaptively choose the groupings in each iteration of the MuG framework as described in Fig. 2.

Fig. 3 shows the histogram of the cardinality of . It is clear that the adaptive groupings approach results in estimates that have lower cardinality than that of the random groupings approach. For all the instances ( each for random and adaptive groupings), contained the true support . We also note that had we not used the group Lasso step in Algorithm 2 and simply used the Lasso for screening, variables would have been selected. On the other hand, using the group Lasso discards on average about half the variables estimated by the Lasso. This shows the benefits of using the MuG framework as opposed to using Lasso for tuning-free screening. We refer to Section V for additional numerical simulations showing the advantages of MuG.

Iv High-Dimensional Consistency

In this Section, we find conditions under which Algorithm 2 is high-dimensional consistent so that as and . As it turns out, the conditions we require are nearly similar to those required for screening using the Lasso estimator. Recall the model in (1) where and are known, is unknown, is the number of non-zero entries in , and is the measurement noise. Consider the following conditions on and .

  1. .

  2. for all groupings satisfying (2)-(3), , and constants and .

  3. for some constant that depends on and .

  4. For , , where is the support estimated by group Lasso.

We assume that the parameter scales with so that as . Further, and are also allowed to scale with . Our main result for consistency of Algorithm 2 is stated as follows.

Theorem 1 (Consistency of Screening)

Under the Assumptions (A1)-(A4), and , where is a constant. Further, if , then as , where is computed using Algorithm 2 and the probability is with respect to the distribution of the noise .

See Appendix A.

Theorem 1 identifies a sufficient condition on the scaling of the parameter . In particular, we see that choosing , for , is sufficient for consistency of the screening algorithm. Since we assume , in practice we choose an appropriate so that . We now make some additional remarks regarding Theorem 1.

Remark 1 (Assumption (A1))

The normalization assumption (A1) is standard and can be easily achieved by scaling the matrix and correspondingly the vector .

Remark 2 (Assumption (A2))

Assumption (A2), which comes from [29]

, ensures restricted strong convexity (RSC) of the least squares loss function in (

10) so that the loss function is “not too flat” around the optimal solution [29]

. We note that alternative conditions based on restricted eigenvalues

[30, 14], which are similar to the RSC conditions, may also be assumed instead of (A2). As shown in [29], matrices whos rows are sampled from a multivariate normal distribution satisfy (A2) when given a sufficient number of observations .

Remark 3 (Assumption (A3))

Assumption (A3) is a standard condition that imposes a lower bound on , the minimum absolute value of the non-zero entries in . Informally, a small requires more observations for consistent estimation using Lasso and group Lasso. It is interesting to see how scales with the group size . If we do not use MuG and simply use the Lasso for screening, (A3) reduces333See Lemma 2 in Appendix A. to . Using MuG with group Lasso increases the lower bound on by . Thus, although the MuG framework may result in screening such that , this comes at the cost of requiring the minimum absolute value in to be slightly larger than that required when simply using the Lasso for screening ( in Algorithm 2). If , then the increase in the lower bound on is small and converges to as grows large. Thus, it is desirable to choose as small as possible. Another motivation for choosing to be small is so that the convergence rate for the probability of error, which is , is small as possible. We will numerically explore different choices of in Section V.

Remark 4 (Assumption (A4))

Assumption (A4) ensures that if a variable is selected for some tuning parameter , then is also selected for any tuning parameter less than . We need (A4) since we always choose the smallest possible tuning parameter when solving the group Lasso problem. We can easily relax (A4) so that . Furthermore, we can modify Algorithm 2 in the following way so that (A4) is no longer needed in Theorem 1 to prove consistency of the screening algorithm:

  • Find the regularization path of the group Lasso by solving (10) for multiple tuning parameters. Instead of assuming that is the support estimated so that group Lasso selects groups, let be the set of all variables selected in the regularization path.

Remark 5 (Discussion)

From Remarks 1-4, it is clear that (A1)-(A3) are standard assumptions required for proving consistency of the Lasso and the group Lasso estimators. Assumption (A1) can be easily achieved by scaling the columns of . Assumption (A4) is specific for MuG, but as discussed in Remark 4, it can easily be avoided using a minor modification of Algorithm 2. Thus, (A2) and (A3) are the main assumptions that determine the the success of Algorithm 2. As discussed in Remark 2, there are a wide class of matrices that satisfy (A2) when given an appropriate number of observations. Assumption (A3) is satisfied when the non-zero entries in have sufficiently large magnitude. We note that if (A2)-(A3) do not hold, then it is likely that Algorithm 2 will miss variables from the true support. As it turns out, since the performance of the Lasso also depends on (A2)-(A3), using the Lasso estimator for screening will also miss variables from the true support. The same is true for the sure independence screening (SIS) algorithm, which actually requires a stronger condition than (A2) for high-dimensional consistent screening [17]. In such cases, it is desirable to perform screening in such a way that contains as many variables from the true support as possible. Using numerical simulations in the next Section on matrices that do not satisfy (A2), we will see that MuG is able to retain more true variables when compared to the Lasso or the SIS algorithm. Finally, we recall that unlike screening algorithms based on the Lasso or the SIS, Algorithm 2 has the advantage of not requiring a tuning parameter.

V Numerical Simulations

In this Section, we provide extensive numerical simulations to show the advantages of using MuG in practice. We assume the linear model in (1) with and consider three different choices of the design matrix :

  • (IND) Each entry is sampled independently from . We let and or depending on the example considered.

  • (TOP) Each row in is sampled independently from , where is a covariance matrix that is Toeplitz such that , where we choose . We let and or depending on the example considered.

  • (RL) We use preprocessed data from [31], where and , such that each row of corresponds to gene expression values from genes relating to Lymph node status for understanding breast cancer treatment [32].

The matrices in (IND) and (TOP) satisfy the so called mutual incoherence property [33, 34, 35] such that exact support recovery is possible using the Lasso given sufficient number of observations. The matrix in (RL) does not satisfy mutual incoherence, which means that no matter how many observations, the support can not be estimated exactly using Lasso. We always normalize the columns of such that . For each design matrix, we randomly choose with a desired sparsity level and a desired to simulate the observations . We emphasize that although we choose the design matrix in (RL) from real data, the actual measurements are being simulated. This is common practice in the literature for testing the performance of sparse recovery algorithms on real design matrices [15]. We evaluate four possible screening algorithms:

  • MuG: This is our proposed algorithm outlined in Section III (see Algorithm 2) with the adaptive grouping statregy described in Section III-B.

  • SIS: This is the sure independence screening algorithm proposed in [17]. Given that the columns are normalized, the algorithm computes by thresholding such that . When comparing MuG and SIS, we choose the threshold so that the estimates from both SIS and MuG have the same cardinality.

  • LCV: This is cross-validated Lasso, where we select the Lasso tuning parameter using cross-validation. We randomly chose of the data for training and the rest for testing and applied Lasso on a grid of values and repeated this process times. The final chosen minimized the mean negative log-likelihood over the training data. It has been shown theoretically [34] and observed empirically that this method may be used to perform screening. In fact, algorithms such as the adaptive Lasso and the thresholded Lasso use LCV as the first stage in estimating and .

  • MuG+LCV: This computes the intersection of MuG and LCV. The main motivation behind using this method is that since both LCV and MuG result in screening and both the methods are different, the intersection of the results from both these methods can result in a that has lower cardinality.

(a) (IND),
(b) (IND),
(c) (IND),
(d) (TOP),
(e) (TOP),
(f) (TOP),
Fig. 4: Results when

is sampled from a Gaussian distribution. See Section 

V-A for more details.

We evaluate screening algorithms using (i) the fraction of variables in that are not in , which we denote by FPR, and (ii) the fraction of variables in that are not in , which we denote by FNR:


In general, we want both the FPR and FNR to be as small as possible. Section V-A discusses results on applying MuG for different choices of the number of groupings . Section V-B discusses results on how the MuG estimates depend on the group size and the parameter .

V-a Number of Groupings

Figures 4 and 5 show results on applying various screening algorithms when is generated as described by (IND), (TOP), and (RL) and . The -axis in all figures is the value of and the

-axis is either the FPR or FNR of a screening algorithm. The lines are mean values of either the FPR or FNR and the shaded regions are the standard deviation of the FPR. For MuG, we always show the FPR using error bars. The LCV method is independent of

, which is why it does not change with .


  1. We clearly see that as increases, the FPR decreases and the FNR either remains constant or increases at a very small rate. For cases when , the FPR decreases at a larger rate and the FNR increases at a small rate. This is because, when , MuG removes more variables in each iteration than when .

  2. We observe that MuG based algorithms perform better than simply using cross-validation (LCV) or using the sure independence screening (SIS) algorithm. The difference between MuG and SIS is more pronounced in cases where is much greater than , see for example Fig. 4(a) and Fig. 4(d).

  3. Combining LCV and MuG, which we refer to as MuG+LCV, leads to a much smaller FPR, while only increasing the FNR by a small amount. On the other hand, simply using LCV results in a much larger FPR. For example, in Fig. 4(a), LCV has a FPR of whereas MuG+LCV has an FPR of about . This shows that the MuG estimates are clearly very different from the Lasso estimates. We note that in the plots, we do not report the FNR values for LCV since this information can be extracted from the FNR plots for MuG+LCV and MuG.

  4. The difference between the performance of MuG and SIS is more pronounced in Fig. 5, where the matrix corresponds to real measurements of gene expression values. For example, in Fig. 5(a), MuG has an FNR of nearly and SIS has an FNR of nearly . This means that for the same cardinality of , the estimate of MuG contains nearly all the true variables, while SIS is only able to retain of the true variables. The reason for this significant difference in performance is due to high correlations between the columns of , in which case SIS is known to perform poorly [36]. These correlations occur because genes in the same pathway tend to produce correlated measurements [37].

(a) (RL),
(b) (RL),
Fig. 5: Results when is the matrix of gene expression values. See Figure 4 for the legend and Section V-A for more details.

V-B Size of the Groups and the Parameter

In this Section, we present numerical simulations to study the performance of MuG as the size of the groupings and the parameter change. Fig. 6(a) shows results for the (IND) example with , , , and . We applied MuG using different choices of ranging from to and chose . As increases, the mean FPR first decreases and then eventually increases. The mean FNR increases with . A similar trend is seen in Fig. 6(b), where . The only difference is that the number of observations are sufficient for screening, so the FNR is zero as ranges from to . Both these examples show that choosing to be large does not necessarily result in superior screening algorithms. In practice, we find that choosing results in good screening algorithms.

Fig. 6(c) shows results on applying MuG to (IND) and (RL) where we fix all the parameters and vary . Only one variable in is changed, so it is expected that this particular variable will be difficult to estimate when is small. This is indeed the case from the plot in Fig. 6(c).

(a) and
(b) and
Fig. 6: Performance of MuG as the size of the groupings and change. See Section V-B for more details.

Vi Extensions and Future Research

We presented the MuG framework in the context of the linear regression problem in (

1) with a sparsity constraint on . We now briefly discuss some extensions of MuG along with some future research directions.

Computational Complexity and Beyond Lasso: The main focus in this paper was to present the MuG framework for variable screening and analyze it’s statistical properties when used with the group Lasso estimator. Although we saw that using MuG with group Lasso resulted in tuning free screening, the potential disadvantage of using the group Lasso is that applying group Lasso multiple number of times may not be computationally feasible for large scale problems. In such cases, it may be useful to first apply the computationally fast SIS algorithm and then use MuG with group Lasso to further screen variables. Alternatively, we can also use other group based variable selection algorithms such as group LARS [5], group marginal regression [38], cluster representative Lasso (CRL) [39], block orthogonal matching pursuit [40], or block CoSaMP [41]. This will be a subject of future research work.

Structured Sparsity: For many problems, prior knowledge can be useful in constructing better estimates of . For example, if it is known that is group sparse, group based estimators, such as those in [5, 40, 41], can be used to estimate using less number of observations. In this case, MuG can be easily applied by forming groupings over the known groups. For applications in image processing [41, 42], it is natural to assume the variables have a tree-structured sparsity pattern which leads to forming a set of overlapping groups. Again, the MuG framework can be applied by forming the groupings over the overlapping groups and using algorithms in [43] for solving the overlapping group Lasso problem.

Graphical Model Selection:

A graphical model is a probability distribution defined on graphs. The nodes in the graph denote random variables and the edges in the graph denote statistical relationships amongst random variables

[44]. The graphical model selection problem is to estimate the unknown graph given observations drawn from a graphical model. One possible algorithm for estimating the graph is by solving a Lasso problem at each node in the graph to estimate the neighbors of each node [34]. Our proposed algorithm using MuG in Algorithm 2 can be used to estimate a superset of the true edges in the graph. There are many other algorithms in the literature for learning graphical models. One method, which is commonly referred to as the graphical Lasso [45] or gLasso, solves an -regularized maximum likelihood problem to estimate a graph. The MuG framework can be applied to gLasso by placing a group penalty on the inverse covariance. However, in this case, it is not clear if parameter-free screening can be done. An alternative method is to assume a conservative upper bound on the number of edges in the graph to perform screening. Our future work will explore this problem.

Exact Support Recovery: Our primary interest in this paper was screening, i.e., to estimate a superset of the true support. Exact support recovery can be easily achieved by applying known algorithms for variable selection once screening has been done. However, it is also of interest to study if exact support recovery or nearly exact support recovery can be achieved using the MuG framework. This may require assuming some upper bound on the support of and then applying MuG with this upper bound.

Vii Summary

In this paper, we presented a novel framework for screening, which we refer to as Multiple Grouping (MuG), that groups variables, performs variable selection over the groups, and repeats this process multiple number of times using different choices of the groupings. The final superset of the true variables is computed by taking an intersection over all the estimated sets over each grouping. The main advantage of MuG over other screening algorithms is that MuG can perform screening in the linear regression problem without using a tuning parameter. Theoretically, we proved consistency of the tuning-free screening algorithm and our numerical simulations showed the advantages of using MuG in practice. We also discussed some future research directions of using MuG in problems involving structured sparsity, graphical models, and exact support recovery.


We thank Brendan Ames, Jarvis Haupt, Nikhil Rao, Adam Rothman, and Vincent Tan for discussions and comments that improved the quality of the paper.

Appendix A Proof of Theorem 1

The following lemma establishes consistency of the group Lasso solutions.

Lemma 2 ([29])

Under (A1)-(A3), there exists a such that the solution to the group Lasso problem in (10) with the grouping satisfies


with probability at least , where is a constant.

Using Lemma 2, there exists a such that if and


Choosing as in (A3) ensures that (14) is satisfied. Under Assumption (A4), choosing any ensures that the support is contained in . Thus, we have

where is a constant, is the tuning parameter chosen in Algorithm 2, and we let . To complete the proof, we have


We use the union bound to get (16). Choosing such that ensures that as . Thus, given a set of groupings , we have established consistency of the MuG screening algorithm. If the groupings are chosen randomly, either using the random grouping or adaptive grouping approaches outlined in Section III-B, we will still get the same consistency result since the bound in (16) only depends on the maximum size of the group .


  • [1] E. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on Information Theory, vol. 52, no. 2, pp. 489–509, 2006.
  • [2] D. Donoho, “Compressed sensing,” IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.
  • [3] A. Wille, P. Zimmermann, E. Vranová, A. Fürholz, O. Laule, S. Bleuler, L. Hennig, A. Prelic, P. Von Rohr, L. Thiele et al., “Sparse graphical gaussian modeling of the isoprenoid gene network in arabidopsis thaliana,” Genome Biol, vol. 5, no. 11, p. R92, 2004.
  • [4] J. Fan, J. Lv, and L. Qi, “Sparse high dimensional models in economics,” Annual review of economics, vol. 3, p. 291, 2011.
  • [5] M. Yuan and Y. Lin, “Model selection and estimation in regression with grouped variables,” Journal of The Royal Statistical Society Series B, vol. 68, no. 1, pp. 49–67, 2006.
  • [6] R. Tibshirani, “Regression shrinkage and selection via the Lasso,” Journal of the Royal Statistical Society, Series B, vol. 58, pp. 267–288, 1996.
  • [7] J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Transactions on Information Theory, vol. 53, no. 12, pp. 4655–4666, 2007.
  • [8] H. Zou, “The adaptive Lasso and its oracle properties,” Journal of the American Statistical Association, vol. 101, pp. 1418–1429, December 2006.
  • [9] E. J. Candes, M. B. Wakin, and S. P. Boyd, “Enhancing sparsity by reweighted -minimization,” Journal of Fourier Analysis and Applications, vol. 14, no. 5-6, pp. 877–905, 2008.
  • [10] L. Wasserman and K. Roeder, “High dimensional variable selection,” Annals of statistics, vol. 37, no. 5A, p. 2178, 2009.
  • [11] S. A. van de Geer, P. Bühlmann, and S. Zhou, “The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso),” Electronic Journal of Statistics, vol. 5, pp. 688–749, 2011.
  • [12] T. Zhang, “Adaptive forward-backward greedy algorithm for learning sparse representations,” IEEE Transactions on Information Theory, vol. 57, no. 7, pp. 4689–4708, 2011.
  • [13] A. Belloni, V. Chernozhukov, and L. Wang, “Square-root Lasso: pivotal recovery of sparse signals via conic programming,” Biometrika, vol. 98, no. 4, pp. 791–806, 2011.
  • [14] P. Bühlmann and S. van de Geer,

    Statistics for High-Dimensional Data: Methods, Theory and Applications

    .   Springer-Verlag New York Inc, 2011.
  • [15] N. Meinshausen and P. Bühlmann, “Stability selection,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 72, no. 4, pp. 417–473, 2010.
  • [16] J. Chen and Z. Chen, “Extended Bayesian information criteria for model selection with large model spaces,” Biometrika, vol. 95, no. 3, pp. 759–771, 2008.
  • [17] J. Fan and J. Lv, “Sure independence screening for ultrahigh dimensional feature space,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 70, no. 5, pp. 849–911, 2008.
  • [18]

    J. Fan, R. Samworth, and Y. Wu, “Ultrahigh dimensional feature selection: beyond the linear model,”

    Journal of Machine Learning Research

    , vol. 10, pp. 2013–2038, 2009.
  • [19] J. Fan and R. Song, “Sure independence screening in generalized linear models with np-dimensionality,” Annals of Statistics, vol. 38, no. 6, pp. 3567–3604, 2010.
  • [20] J. Fan, Y. Feng, and R. Song, “Nonparametric independence screening in sparse ultra-high-dimensional additive models,” Journal of the American Statistical Association, vol. 106, no. 494, pp. 544–557, 2011.
  • [21] T. Ke, J. Jin, and J. Fan, “Covariance assisted screening and estimation,” Arxiv preprint arXiv:1205.4645, 2012.
  • [22] R. Tibshirani, J. Bien, J. Friedman, T. Hastie, N. Simon, J. Taylor, and R. Tibshirani, “Strong rules for discarding predictors in Lasso-type problems,” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2011.
  • [23] L. El Ghaoui, V. Viallon, and T. Rabbani, “Safe feature elimination for the Lasso,” Journal of Machine Learning Research. Submitted, 2011.
  • [24] Z. Xiang, H. Xu, and P. Ramadge, “Learning sparse representations of high dimensional data on large scale dictionaries,” in Advances in Neural Information Processing Systems (NIPS), 2011.
  • [25] Z. J. Xiang and P. J. Ramadge, “Fast Lasso screening tests based on correlations,” in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2012, pp. 2137–2140.
  • [26] M. Osborne, B. Presnell, and B. Turlach, “On the Lasso and its dual,” Journal of Computational and Graphical statistics, pp. 319–337, 2000.
  • [27] H. Liu and J. Zhang, “Estimation consistency of the group Lasso and its applications,” in

    Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS)

    , 2009.
  • [28] F. Bach, “Bolasso: model consistent Lasso estimation through the bootstrap,” in Proceedings of the International Conference on Machine Learning (ICML), 2008.
  • [29] S. Negahban, P. Ravikumar, M. Wainwright, and B. Yu, “A unified framework for high-dimensional analysis of -estimators with decomposable regularizers,” Statistical Science, vol. 27, no. 4, pp. 538–557, 2012.
  • [30] P. Bickel, Y. Ritov, and A. Tsybakov, “Simultaneous analysis of Lasso and Dantzig selector,” Annals of Statistics, vol. 37, no. 4, pp. 1705–1732, 2009.
  • [31] L. Li and K. Toh, “An inexact interior point method for -regularized sparse covariance selection,” Mathematical Programming Computation, vol. 2, no. 3, pp. 291–315, 2010.
  • [32] J. Pittman, E. Huang, H. Dressman, C. Horng, S. Cheng, M. Tsou, C. Chen, A. Bild, E. Iversen, A. Huang et al., “Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes,” Proceedings of the National Academy of Sciences of the United States of America, vol. 101, no. 22, p. 8431, 2004.
  • [33] P. Zhao and B. Yu, “On model selection consistency of Lasso,” Journal of Machine Learning Research, vol. 7, pp. 2541–2563, 2006.
  • [34] N. Meinshausen and P. Bühlmann, “High-dimensional graphs and variable selection with the Lasso,” Annals of statistics, vol. 34, no. 3, p. 1436, 2006.
  • [35] M. J. Wainwright, “Sharp thresholds for noisy and high-dimensional recovery of sparsity using -constrained quadratic programming (Lasso),” IEEE Transactions on Information Theory, vol. 55, no. 5, May 2009.
  • [36] C. Genovese, J. Jin, L. Wasserman, and Z. Yao, “A comparison of the Lasso and marginal regression,” Journal of Machine Learning Research, vol. 13, pp. 2107–2143, 2012.
  • [37] M. Segal, K. Dahlquist, and B. Conklin, “Regression approaches for microarray data analysis,” Journal of Computational Biology, vol. 10, no. 6, pp. 961–980, 2003.
  • [38] W. U. Bajwa and D. G. Mixon, “Group model selection using marginal correlations: The good, the bad and the ugly,” in 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2012, pp. 494–501.
  • [39] P. Bühlmann, P. Rütimann, S. van de Geer, and C.-H. Zhang, “Correlated variables in regression: Clustering and sparse estimation,” Journal of Statistical Planning and Inference, vol. 143, no. 11, pp. 1835–1858, 2013.
  • [40] Y. Eldar, P. Kuppinger, and H. Bolcskei, “Block-sparse signals: Uncertainty relations and efficient recovery,” IEEE Transactions on Signal Processing, vol. 58, no. 6, pp. 3042–3054, 2010.
  • [41] R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde, “Model-based compressive sensing,” IEEE Transactions on Information Theory, vol. 56, no. 4, pp. 1982–2001, 2010.
  • [42] N. S. Rao, R. D. Nowak, S. J. Wright, and N. G. Kingsbury, “Convex approaches to model wavelet sparsity patterns,” in IEEE International Conference on Image Processing (ICIP), 2011, pp. 1917–1920.
  • [43] L. Jacob, G. Obozinski, and J. Vert, “Group Lasso with overlap and graph Lasso,” in Proceedings of the 26th Annual International Conference on Machine Learning (ICML), 2009, pp. 433–440.
  • [44] S. L. Lauritzen, Graphical Models.   Oxford University Press, USA, 1996.
  • [45] O. Banerjee, L. El Ghaoui, and A. d’Aspremont, “Model selection through sparse maximum likelihood estimation for multivariate Gaussian or binary data,” Journal of Machine Learning Research, vol. 9, pp. 485–516, 2008.