1 Introduction
Feature Selection is one of the main contemporary problems in Machine Learning and has been approached from many directions. One modern approach to feature selection in linear models consists in minimizing an regularized empirical risk. This particular risk encourages the model to have a good balance between a low classification error and high sparsity (where only a few features are used for classification). As the regularized problem is combinatorial, many approaches such as the LASSO [1] try to address the combinatorial problem by using more practical norms such as . These approaches have been developed with two main goals in mind: restricting the number of features for improving classification speed, and limiting the used features to the most useful to prevent overfitting. These classical approaches to sparsity aim at finding a sparse representation of the features space that is global to the entire dataset.
We propose a new approach to sparsity where the goal is to limit the number of features per datapoint, thus datumwise sparse classification (DWSC). This means that our approach allows the choice of features used for classification to vary relative to each datapoint; data points that are easy to classify can be inferred on without looking at very many features, and more difficult datapoints can be classified using more features. The underlying motivation is that, while classical approaches balance between accuracy and sparsity at the dataset level, our approach optimizes this balance at the individual datum level, thus resulting in equivalent accuracy at higher overall sparsity. This kind of sparsity is interesting for several reasons: First, simpler explanations are always to be preferred as per Occam’s Razor. Second, in the knowledge extraction process, such datumwise sparsity is able to provide unique information about the underlying structure of the data space. Typically, if a dataset is organized onto two different subspaces, the datumwise sparsity principle will allows the model to automatically choose to classify using only the features of one or another of the subspace.
DWSC considers feature selection and classification as a single sequential decision process. The classifier iteratively chooses which features to use for classifying each particular datum. In this sequential decision process, datumwise sparsity is obtained by introducing a penalizing reward when the agent chooses to incorporate an additional feature into the decision process. The model is learned using an algorithm inspired by Reinforcement Learning
[2].The contributions of the paper are threefold: (i.) We propose a new approach where classification is seen as a sequential process where one has to choose which features to use depending on the input being inferred upon. (ii.) This new approach results in a model that obtains good performance in terms of classification while maximizing datumwise sparsity, i.e. the mean number of features used for classifying the whole dataset. It also naturally handles multiclass classification problems, solving them by using as few features as possible for all classes combined. (iii.) We perform a series of experiments on 14 different corpora and compare the model with those obtained by the LARS [3], and a regularized SVM, thus providing a qualitative study of the behaviour of our algorithm.
The paper is organized as follow: First, we define the notion of datumwise sparse classifiers and explain the interest of such models in Section 2. We then describe our sequential approach to classification and detail the learning algorithm and the complexity of such an algorithm in Section 3. We describe how this approach can be extended to multiclass classification in Section 4. We detail experiments on 14 datasets, and also give a qualitative analysis of the behaviour of this model in Section 6. The related work is given in Section 7.
2 DatumWise Sparse Classifiers
We consider the problem of supervised multiclass classification^{1}^{1}1Note that this includes the binary supervised classification problem as a special case. where one wants to learn a classification function to associate one category
to a vector
, where , being the dimension of the input vectors. is the set of parameters learned from a training set composed of input/output pairs . These parameters are commonly found by minimizing the empirical risk defined by:(1) 
where is the loss associated to a prediction error.
This empirical risk minimization problem does not consider any prior assumption or constraint concerning the form of the solution and can result in overfitting models. Moreover, when facing a very large number of features, obtained solutions usually need to perform computations on all the features for classifying any input, thus negatively impacting the model’s classification speed. We propose a different risk minimization problem where we add a penalization term that encourages the obtained classifier to classify using on average as few features as possible. In comparison to classical or regularized approaches where the goal is to constraint the number of features used at the dataset level, our approach performs sparsity at the datum level, allowing the classifier to use different features when classifying different inputs. This results in a datumwise sparse classifier that, when possible, only uses a few features for classifying easy inputs, and more features for classifying difficult or ambiguous ones.
We consider a different type of classifier function that, in addition to predicting a label given an input , also provides information about which features have been used for classification. Let us denote . We define a datumwise classification function of parameters as:
where is the predicted output and is a dimensional vector , where implies that feature has been taken into consideration for computing label on datum . By convention, we denote the predicted label as and the corresponding vector as . Thus, if , feature has been used for classifying into category .
This definition of datawise classifiers has two main advantages: First, as we will see in the next section, because can explain its use of features with , we can add constraints on the features used for classification. This allows us to encourage datumwise sparsity which we define below. Second, while this is not the main focus of our article, analysis of gives a qualitative explanation of how the classification decision has been made, which we study in Section 6. Note that the way we define datumwise classification is an extension to the usual definition of a classifier.
2.1 DatumWise Sparsity
Datumwise sparsity is obtained by adding a penalization term to the empirical loss defined in equation (1) that limits the average number of features used for classifying:
(2) 
The term is the norm ^{2}^{2}2The ’norm’ is not a proper norm, but we will refer to it as the norm in this paper, as is common in the sparsity community. of , i.e. the number of features selected for classifying , that is, the number of elements in equal to 1. In the general case, the minimization of this new risk results in a classifier that on average selects only a few features for classifying, but may use a different set of features w.r.t to the input being classified. We consider this to be the crux of the DWSC model: the classifier takes each datum into consideration differently during the inference process.
(left)  (right) 

Note that the optimization of the loss defined in equation (2
) is a combinatorial problem that cannot be easily solved. In the next section of this paper, we propose an original way to deal with this problem, based on a Markov Decision Process.
3 DatumWise Sparse Sequential Classification
We consider a Markov Decision Problem (MDP, [4])^{3}^{3}3The MDP is deterministic in our case. to classify an input . At the beginning, we have no information about , that is, we have no attribute/feature values. Then, at each step, we can choose to acquire a particular feature of , or to classify . The act of classifying in the category ends an “episode” of the sequential process. The classification process is a deterministic process defined by:

A set of states , where state corresponds to the state where the agent is currently classifying datum and has selected features specified by . The number of currently selected features is thus .

A set of actions where denotes the set of possible actions in state . We consider two types of actions:

is the set of feature selection actions such that, for corresponds to choosing feature . Action corresponds to a vector with only the element equal to 1, i.e. . Note that the set of possible feature selection actions on state , denoted , is equal to the subset of currently unselected features, i.e. .

is the set of classification actions , that correspond to assigning a label to the current datum. Classification actions stop the sequential decision process.


A transition function defined only for feature selection actions (since classification actions are terminal):
where is an updated version of such that .
3.0.1 Policy
We define a parameterized policy , which, for each state , returns the best action as defined by a scoring function :
The policy decides which action to take by applying the scoring function to every action possible from state and greedily taking the highest scoring action. The scoring function reflects the overall quality of taking action in state , which corresponds to the total reward obtained by taking action in and thereafter following policy ^{4}^{4}4This corresponds to the classical Qfunction in Reinforcement Learning.:
Here corresponds to the reward obtained at step while having started in state and followed the policy with parameterization for steps. Taking the sum of these rewards gives us the total reward from state until the end of the episode. Since the policy is deterministic, we may refer to a parameterized policy using simply . Note that the optimal parameterization obtained after learning (see Sec. 3.3) is the parameterization that maximizes the expected reward in all stateaction pairs of the process.
In practice, the initial state of such a process for an input corresponds to an empty vector where no feature has been selected. The policy sequentially picks, one by one, a set of features pertinent to the classification task, and then chooses to classify once enough features have been considered.
3.0.2 Reward
The reward function reflects the immediate quality of taking action in state relative to the problem at hand. We define a reward function over the training set : which reflects how good of a decision taking action on state for input is relative to our classification task. This reward is defined as follows^{5}^{5}5Note that we can add to the reward at the end of the episode, and give a constant intermediate reward of . These two approaches are interchangeable.:

If corresponds to a feature selection action, then the reward is .

If corresponds to a classification action i.e. , we have:
In practice, we set to avoid situations where classifying incorrectly is a better decision than choosing multiple features.
3.1 Reward Maximization and Loss Minimization
As explained in section 2, our ultimate goal is to find the parameterization that minimizes the datumwise empirical loss defined in equation (2). The training process for the MDP described above is the maximization of a reward function. Let us therefore show that maximizing the reward function is equivalent to minimizing the datumwise empirical loss.
where is the action taken at time by the policy for the training example .
Such an equivalence between risk minimization and reward maximization shows that the optimal classifier corresponds to the optimal policy in the MDP defined previously. This equivalence allows us to use classical MDP resolution algorithms in order to find the best classifier. We detail the learning procedure in Section 3.3.
3.2 Inference and Approximated Decision Processes
Due to the infinite number of possible inputs , the number of states is also infinite. Moreover, the reward function is only known for the values of that are in the training set and cannot be computed for any other input. For these two reasons, it is not possible to compute the score function for all stateaction pairs in a tabular manner, and this function has to be approximated.
The scoring function that underlies the policy is approximated with a linear model^{6}^{6}6
Although nonlinear models such as neural networks may be used, we have chosen to restrict ourselves to a linear model to be able to properly compare performance with that of other stateoftheart linear sparse models.
:and the policy defined by such a function consists in taking in state the action that maximizes the scoring function i.e .
Due to their infiniteness, the stateaction pairs are represented in a feature space. We note the featurized representation of the stateaction pair. Many definitions may be used for this feature representation, but we propose a simple projection: we restrict the representation of to only the selected features. Let be the restriction of according to :
To be able to differentiate between an attribute of that is not yet known, and an attribute that is simply equal to 0, we must keep the information present in . Let be the intermediate representation that corresponds to the concatenation of with . Now we simply need to keep the information present in in a manner that allows each action to be easily distinguished by a linear classifier. To do this we use the blockvector trick [5] which consists in projecting into a higher dimensional space such that the position of inside the global vector is dependent on action :
In , the block is at position where is the index of action in the set of all the possible actions. Thus, is offset by an amount dependent on the action .
3.3 Learning
The goal of the learning phase is to find an optimal policy parameterization which maximizes the expected reward, thus minimizing the datumwise regularized loss defined in (2). As explained in Section 3.2, we cannot exhaustively explore the state space during training, and therefore we use a MonteCarlo approach to sample example states from the learning space. We use the Approximate Policy Iteration (API) algorithm with rollouts [6]. Sampling stateaction pairs according to a previous policy , API consists in iteratively learning a better policy by way of the Bellman equation. The API With Rollouts algorithm is composed of three main steps that are iteratively repeated:

The algorithm begins by sampling a set of random states: the
vector is sampled from a uniform distribution in the training set, and
is also sampled using a uniform binomial distribution.

For each state in the sampled state, the policy is used to compute the expected reward of choosing each possible action from that state. We now have a feature vector for each stateaction pair in the sampled set, and the corresponding expected reward denoted .

The parameters
of the new policy are then computed using classical linear regression on the set of states —
— and corresponding expected rewards —— obtained previously. The generalizing capacity of the classifier gives an estimated score to stateaction pairs even if we have never visited them.
After a certain number of iterations, the parameterized policy converges to a final policy which is used for inference.
4 Preventing Overfitting in the Sequential Model
In section 3.0.1, we explain the process by which, at each step, we either choose a new feature or classify the current datum. This process is at the core of DWSC but can suffer from overfitting if the number of features is larger than the number of training examples. In such a case, DWSC would tend to learn to select the more specific features for each training example. In classical regularization models that are not datumwise, the classifier must use the same set of features for classifying any data and thus overly specific features are not chosen because they usually appear in only a few training examples.
We propose a very simple variant of the general model that allows us to avoid overfitting. We still allow DWSC to choose how many features to use before classifying an input , but we constrain it to choose the features in the same order for all the inputs. For that, we constrain the score of the feature selection actions to depend only on the vector of the state . An example of the effect of such a constraint is presented in Fig. 2. This constraint is handled in the following manner:
(3) 
where implies that the score is computed using only the values of and — is ignored. This corresponds to having two different types of stateaction feature vectors depending on the type of action:
(4) 
Example  Features Selected  Example  Features Selected  

2  3  :  2  3  
1  4  2  3  :  2  3  1  4  
3  :  2  3  
2  3  1  :  2  3  1  
Unconstrained Model  Constrained Model 
Although this constraint forces DWSC to choose the features in the same order, it will still automatically learn the best order in which to choose the features, and when to stop adding features and classify. However, it will avoid choosing very different features sets for classifying different inputs (the first features chosen will be common to all the inputs being classified) and thus avoid the overfitting problem.
5 Complexity Analysis
Learning Complexity:
As explained in section 3.3, the learning method is based on Reinforcement Learning with Rollouts. Such an approach is expensive in term of computations because it needs — at each iteration of the algorithm — to simulate trajectories in the decision process, and then to learn the scoring function based on these trajectories. Without giving the details of the computation, the complexity of each iteration is , where is the number of states used for rollouts (which in practice is proportional to the number of training examples), is the number of features and is the number of possible categories. This implies a learning method which is quadratic w.r.t. the number of features; the proposed approach is not able to deal with problems with thousands of possible features. Breaking this complexity is an active research perspective with some leads.
Inference Complexity:
Inference on an input consists in sequentially choosing features, and then classifying . At step , one has to perform linear computations in order to choose the best action, where is the number of possible actions when features have already been acquired. The inference complexity is thus , where is the mean number of features chosen by the system before classifying. In fact, due to the shape of the function presented in Section 3.2 and the linear nature of , the score of the actions can be efficiently incrementally computed at each step of the process by just adding the contribution of the newly added feature. The complexity is thus reduced to . Moreover, the constrained model which results in ordering the features, has a lower complexity of because in that case, the model does not have to choose between the different remaining features, and has only the choice to classify or get the next feature w.r.t. to the learned order.
If the learning complexity of our model is higher than baseline global linear methods, the inference speed is very close for the unconstrained model, and equivalent for the constrained one. In practice, most of the baseline methods choose a subset of variables in a couple seconds to a couple minutes, whereas our method takes from a dozen minutes to an hour, depending on the number of features and categories. In practice inference is indeed of the same speed, which is in our opinion the important factor.
6 Experiments
Name  Number of Features  Number of examples  Number of Classes  Task 

Australian  14  690  2  Binary 
Breast Cancer  10  683  2  Binary 
Diabetes  8  768  2  Binary 
German Numer  24  1,000  2  Binary 
Heart  13  270  2  Binary 
Ionosphere  34  351  2  Binary 
Liver Disorders  6  345  2  Binary 
Sonar  60  208  2  Binary 
Splice  60  1,000  2  Binary 
Svm Guide 3  21  1,284  2  Binary 
Segment  19  2,310  7  Multiclass 
Vehicle  18  846  4  Multiclass 
Vowel  10  1,000  11  Multiclass 
Wine  13  178  3  Multiclass 
Experiments were run on 14 different datasets obtained from the LibSVM Website^{7}^{7}7http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. Ten of these datasets correspond to a binary classification task, four to a multiclass problem. The datasets are described in Table 1. For each dataset, we randomly sampled different training sets by taking from to of the examples as training examples, with the remaining examples being kept for testing. We performed experiments with three different models: L1SVM was used as a baseline linear model with regularization^{8}^{8}8Using LIBLINEAR [7].. LARS was used to obtain the optimal solution of the LASSO problem for all values of the regularization coefficient at once^{9}^{9}9We use the implementation from the authors of the LARS, available in R.. DatumWise Sequential Model (DWSM) was tested with the two versions presented above: (i) DWSMUn is the original unconstrained model and (ii) DWSMCon is the constrained model for preventing overfitting.
For the evaluation, we used a classical accuracy measure which corresponds to on the test set of each dataset. We perform 3 training/testing set splits of a given dataset to obtain averaged figures. The sparsity has been measured as the proportion of features not used for SVM and LARS in binary classification, and the mean proportion of features not used to classify testing examples in DWSM. For multiclass problems where one LARS/SVM model is learned for each category, the sparsity is the proportion of features that have not been used in any of the models.
For the sequential experiments, the number of rollout states (step 1 of the learning algorithm) has been set to 2,000 and the number of policy iterations has been fixed to 10. Note that experiments with more rollout states and/or more iterations give similar results. Experiments were made using an alpha mixture policy with to ensure the stability of the learning process. We tested the different models with different values of which controls the sparsity. Note that even with a value, contrary to the baseline models, the DWSM model does not use all of the features for classification.
Corpus  Train Size  Sparsity = 0.8  Sparsity = 0.6  Sparsity = 0.4  

DWSMUn  DWSMCon  SVM L1  DWSMUn  DWSMCon  SVM L1  DWSMUn  DWSMCon  SVM L1  
australian  0.05  85.31  85.13  85.52  84.63  84.83  85.42  84.01  84.32  85.15 
0.1  85.75  86.16  85.51  86.00  86.32  85.51  86.13  85.70  86.34  
0.25  85.39  86.16  85.33  86.76  86.99  85.33  86.56  86.49  86.10  
0.5  83.58  84.13  83.47  84.33  84.19  83.47  84.85  84.28  83.70  
breastcancer  0.05  87.73  88.50  88.17  96.58  96.49  94.93  96.57  96.68  96.62 
0.1  89.25  89.16  88.19  94.70  95.13  91.91  95.42  95.63  92.88  
0.25  91.16  88.48  92.64  97.06  96.76  94.38  97.09  97.11  96.71  
0.5  82.65  82.60  90.26  96.92  95.98  93.41  96.50  97.01  94.84  
diabetes  0.05  68.61  69.10  65.85  71.30  71.36  67.46  72.15  0.00  71.07 
0.1  70.92  69.14  65.71  72.52  71.62  64.97  73.26  72.81  70.23  
0.25  68.39  68.58  64.88  71.83  72.42  65.88  74.80  74.96  75.09  
0.5  70.65  69.69  62.27  72.67  70.90  67.30  73.82  73.62  72.14  
german.numer  0.05  70.58  70.47  67.36  70.74  70.39  69.95  69.99  70.28  69.73 
0.1  69.82  69.62  69.10  70.81  70.39  71.52  71.79  0.00  72.85  
0.25  72.25  72.00  65.98  72.67  73.26  72.89  73.10  0.00  74.11  
0.5  70.03  70.62  69.72  71.50  72.37  71.97  72.96  74.05  72.68  
heart  0.05  48.33  48.17  45.42  51.17  50.67  65.73  0.00  0.00  68.24 
0.1  75.50  74.27  73.42  76.61  75.78  73.76  77.60  77.49  74.94  
0.25  76.17  78.50  76.26  81.31  81.70  83.33  82.24  83.00  83.64  
0.5  70.34  68.87  69.20  77.15  80.34  78.83  80.48  80.40  80.58  
ionosphere  0.05  69.52  71.36  73.55  73.44  73.02  72.23  74.77  75.16  72.59 
0.1  71.58  71.09  71.84  75.12  74.63  75.89  74.97  74.93  74.49  
0.25  79.65  80.29  75.94  85.18  85.44  81.58  85.58  85.69  82.78  
0.5  77.31  78.40  71.15  82.94  82.68  78.18  84.96  84.16  79.64  
liverdisorders  0.05  60.40  59.37  57.01  60.07  61.25  57.01  60.29  64.27  57.74 
0.1  56.70  56.24  55.41  55.85  55.98  56.43  56.69  55.00  55.86  
0.25  56.69  56.14  54.18  58.07  57.02  55.10  58.69  57.97  56.93  
0.5  58.93  59.55  60.84  60.10  58.81  60.96  59.33  60.84  61.33  
sonar  0.05  57.59  59.95  64.14  68.50  66.49  65.15  69.45  70.48  61.24 
0.1  61.69  64.40  64.12  68.68  73.93  64.12  74.25  75.20  63.53  
0.25  67.32  64.74  67.52  73.52  70.63  74.52  75.22  73.36  72.82  
0.5  68.19  64.71  65.77  72.18  69.76  69.37  73.73  71.60  65.77  
splice  0.05  67.23  68.41  67.82  70.14  68.66  65.93  70.51  69.89  64.47 
0.1  66.90  66.87  61.46  70.35  67.99  62.63  71.05  70.07  61.62  
0.25  73.87  73.89  70.49  74.81  75.30  72.28  75.60  76.64  71.74  
0.5  72.86  76.79  72.78  74.98  77.88  70.36  77.09  0.00  69.35  
svmguide3  0.05  77.15  77.13  77.17  77.32  77.25  78.25  77.48  77.37  78.21 
0.1  77.31  77.28  76.59  77.94  78.11  78.95  78.58  78.94  78.37  
0.25  76.67  76.56  75.96  77.44  77.14  77.40  78.21  77.72  77.91  
0.5  77.71  77.78  76.87  78.55  78.63  78.15  79.38  79.47  78.37 
) using different training sizes. The accuracy has been linearly interpolated from curves like the ones given in Figure
3.6.1 Results
For each corpus and each training size, we have computed sparsity/accuracy curves showing the performance of the different models w.r.t. to the sparsity of the solution. Only two representative curves are given in Figure 3. To summarize the performances over all the datasets, we give the accuracy of the different models for three levels of sparsity in tables 2 and 3. Due to a lack of space, these tables do not present the LARS’ performance, which are equivalent to the performances of the SVM. Note that in order to obtain the accuracy for a given level of sparsity, we have computed a linear interpolation on the different curves obtained for each corpus and each training size. This linear interpolation allows us to compare the baseline sparsity methods — that choose a fixed number of features — with the average number of features chosen by DWSC. This compares the average amount of information considered by each classifier. We believe this approach still provides a good appreciation of the algorithm’s capacities.
Corpus  Train Size  Sparsity = 0.8  Sparsity = 0.6  Sparsity = 0.4  

DWSMUn  DWSMCon  L1SVM  DWSMUn  DWSMCon  L1SVM  DWSMUn  DWSMCon  L1SVM  
segment  0.1  42.06  41.23  35.31  53.87  53.02  45.49  54.83  56.57  56.98 
0.2  40.76  40.17  40.48  55.70  56.34  45.97  57.42  59.10  53.24  
0.5  43.29  0.00  37.17  54.09  0.00  45.15  56.43  0.00  50.52  
0.75  43.78  41.13  38.22  55.10  53.60  44.80  56.54  56.99  47.00  
vehicle  0.1  34.23  37.52  43.36  43.50  45.34  50.25  47.21  0.00  56.54 
0.2  38.32  39.27  53.04  45.84  45.68  53.36  48.68  47.91  52.83  
0.5  39.74  39.51  42.95  46.64  47.57  50.30  0.00  48.40  51.99  
0.75  40.32  40.37  41.04  49.96  49.31  53.68  51.86  51.53  53.77  
vowel  0.1  18.03  19.27  9.83  24.17  22.82  16.24  25.28  25.80  18.38 
0.2  0.00  15.27  14.71  20.17  15.93  0.00  22.59  15.93  
0.5  18.98  17.81  9.57  24.56  25.33  17.73  28.45  27.31  23.76  
0.75  19.85  19.49  14.41  28.01  31.45  24.58  32.09  32.74  26.69  
wine  0.1  70.22  70.66  73.58  76.42  77.87  89.38  78.66  76.67  91.36 
0.2  71.52  72.68  80.34  78.27  79.11  92.12  78.76  77.72  94.16  
0.5  72.99  74.41  74.40  79.43  80.60  86.90  82.15  79.50  91.38  
0.75  76.21  75.04  72.00  80.18  81.84  94.00  83.23  80.93  96.00 
Table 2 shows that, for a sparsity level of 80%, the DWSMUn and the DWSMCon models outperform the baseline SVM classifier. This is particularly true for 7 of the 10 datasets while the results are more ambiguous on the three others datasets: breast, ionosphere and sonar. For a sparsity of 40%, similar results are obtained. Depending on the corpus and the training size, different configurations are observed. Some datasets can be easily classified using only a few features, such as australian for example. In that case, our approach gives similar results in comparison to methods (see Figure 3–left). For some other datasets, our method clearly outperforms baseline methods (Figure 3–right). On the splice dataset, our model is better than the best (nonsparse) SVM using only less than 20% of the features on average. This is due to the fact that our sequential process, which solves a different classification problem, is more appropriate for some particular datasets, particularly when the distribution of the data is split up amongst distinct subspaces.
In this case, our model is able to choose more appropriate features for each input.
When using small training sets with some datasets — sonar or ionosphere — where overfitting is observed (accuracy decreases with more features used), the DWSMCon seems to be a better choice than the unconstrained version and thus is a version of the algorithm that is wellsuited when the number of learning examples is small.
Concerning the multiclass problems, similar effects can be observed (see Table 3). The model seems particularly interesting when the number of categories is high, as in segment and vowel. This is due to the fact that the average sparsity is optimized by the sequential model for the multiclass problem while SVM and LARS, which need to learn one model for each category, perform separate sparsity optimizations for each class.
Figure 4 gives some qualitative results. First, from the left histogram, one can see that some features are used in 100% of the decisions. This illustrates the ability of the model to detect important features that must be used for decision. Note that many of these features are also used by the SVM and the LARS models. The sparsity gain in comparison to the baseline model is obtained through the features 1 and 9 that are only used in about 20% of decisions. From the right histogram, one can see that the DWSM model mainly classifies using 1, 2, 3 or 10 features, showing that the model is able to adapt its behaviour to the difficulty of classifying a particular input. This is confirmed by the green and violet histograms that show that for incorrect decisions (i.e. very difficult inputs) the classifier almost always acquires all the features before classifying. These difficult inputs seem to have been identified, but the set of features is not sufficient for a good understanding. This behaviour opens appealing research directions concerning the acquisition and creation of new features (see Section 8).
7 Related Work
Feature selection comes in three main flavors [8]: wrapper, filter, or embedded approaches. Wrapper approaches involve searching the feature space for an optimal subset of features that maximize classifier performance. The feature selection step wraps around the classifier, using the classifier as a blackbox evaluator of the selected feature subset. Searching the entire feature space is very quickly intractable and therefore various approaches have been proposed to restrict the search (see [9, 10]). The advantage of the wrapper approaches is that the feature subset decision can take into consideration feature interdependencies and avoid redundant features, however the problem remains of the exponential size of the search space. Filter approaches rank the features by some scoring function independent of their effect on the associated classifier. Since the choice of features is not influenced by classifier performance, filter approaches rely purely on the adequacy of their scoring functions. Filtering methods are susceptible to not discriminating redundant features, and missing feature interdependencies (since each feature is scored individually). Filter approaches are however easier to compute and more statistically stable relative to changes in the dataset. Embedded approaches include feature selection as part of the learning machine. These include algorithms solving the LASSO problem [1], and other linear models involving a regularizer based on a sparsity inducing norm (norms [11]
, group LASSO, …). Kernel machines provide a mixture of feature selection and construction as part of the classification problem. Decision trees are also considered embedded approaches although they are also similar to filter approaches in their use of heuristic scores for tree construction. The main critique of embedded approaches is twofold: they are susceptible to include redundant features, and not all the techniques described are easily applied to multiclass problems.In brief, both filtering and embedded approaches have their drawbacks in terms of their ability to select the best subset of features, whereas wrapper methods have their main drawback in the intractability of searching the entire feature space. Furthermore, all existing methods perform feature selection based on the whole training set, the same set of features being used to represent any data.
Our sequential decision problem defines both feature selection and classification tasks. In this sense, our approach resembles an embedded approach. In practice, however, the final classifier for each single datapoint remains a separate entity, a sort of blackbox classifying machine upon which performance is evaluated. Additionally, the learning algorithm is free to navigate over the entire combinatorial feature space. In this sense our approach resembles a wrapper method.
There has been some work using similar formalisms [12], but with different goals and lacking in experimental results. Sequential decision approaches have been used for costsensitive classification with similar models [13]. There have also been applications of Reinforcement Learning to optimize anytime classification [14]. We have previously looked at using Reinforcement Learning for finding a stopping point in feature quantity during text classification [15].
Finally, in some sense, DWSC has some similarity with decision trees as each new datapoint that is labeled is following a different path in the feature space. However, the underlying mechanism is quite different both in term of inference procedure and learning criterion. There has been some work in using RL for generating decision trees [16], but that approach is still tied to decision tree construction heuristics and the end product remains a decision tree.
8 Conclusion
In this article we introduced the concept of datumwise classification, where we learn both a classifier and a sparse representation of the data that is adaptive to each new datum being classified. We took an approach to sparsity that considers the combinatorial space of features, and proposed a sequential algorithm inspired by Reinforcement Learning to solve this problem. We showed that finding an optimal policy for our Reinforcement Learning problem is equivalent to minimizing the regularized loss of our classification problem. Additionally we showed that our model works naturally on multiclass problems, and is easily extended to avoid overfitting on datasets where the number of features is larger than the number of examples. Experimental results on 14 datasets showed that our approach is indeed able to increase sparsity while maintaining equivalent classification accuracy.
References
 [1] R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society. Series B, no. January 1994, 1994.
 [2] R. Sutton and A. Barto, Reinforcement Learning. MIT Press, 1998.
 [3] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Leastangle regression,” Annals of statistics, vol. 32, no. 2, pp. 407–499, 2004.
 [4] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, 1994.
 [5] S. HarPeled, D. Roth, and D. Zimak, “Constraint classification: A new approach to multiclass classification,” Algorithmic Learning Theory, pp. 1 – 11, 2002.
 [6] M. G. Lagoudakis and R. Parr, “Reinforcement learning as classification: Leveraging modern classifiers,” ICML, 2003.
 [7] R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin, “LIBLINEAR: A library for large linear classification,” JMLR, vol. 9, pp. 1871–1874, 2008.
 [8] I. Guyon and A. Elisseefi, “An Introduction to Variable and Feature Selection,” Journal of Machine Learning Research, vol. 3, no. 78, pp. 1157–1182, Oct. 2003.

[9]
S. Girgin and P. Preux, “Feature discovery in reinforcement leanring using genetic programming,” in
Proc. EuroGP, ser. Lecture Notes in Artificial Intelligence (LNCS), vol. 4971. Springer, Mar. 2008, pp. 218–229.
 [10] R. Gaudel and M. Sebag, “Feature Selection as a OnePlayer Game,” ICML, 2010.
 [11] Z. Xu, H. Zhang, Y. Wang, X. Chang, and Y. Liang, “L1/2 regularization,” SCIENCE CHINA Information Sciences, vol. 53, no. 6, pp. 1159–1169, 2010.
 [12] E. Ertin, “Reinforcement learning and design of nonparametric sequential decision networks,” in Proceedings of SPIE. Spie, 2002, pp. 40–47.
 [13] S. Ji and L. Carin, “Costsensitive feature acquisition and classification,” Pattern Recognition, vol. 40, no. 5, pp. 1474–1485, May 2007.
 [14] B. Póczos, Y. AbbasiYadkori, C. Szepesvári, R. Greiner, and N. Sturtevant, “Learning when to stop thinking and do something!” ICML ’09, pp. 1–8, 2009.
 [15] G. DulacArnold, L. Denoyer, and P. Gallinari, “Text Classification: A Sequential Reading Approach,” ECIR, pp. 411–423, 2011.
 [16] M. Preda, “Adaptive building of decision trees by reinforcement learning,” in Proceedings of the 7th WSEAS, 2007, pp. 34–39.