Accelerating Cross-Validation in Multinomial Logistic Regression with ℓ_1-Regularization

by   Tomoyuki Obuchi, et al.

We develop an approximate formula for evaluating a cross-validation estimator of predictive likelihood for multinomial logistic regression regularized by an ℓ_1-norm. This allows us to avoid repeated optimizations required for literally conducting cross-validation; hence, the computational time can be significantly reduced. The formula is derived through a perturbative approach employing the largeness of the data size and the model dimensionality. Its usefulness is demonstrated on simulated data and the ISOLET dataset from the UCI machine learning repository.



There are no comments yet.


page 1

page 2

page 3

page 4


Optimizing Approximate Leave-one-out Cross-validation to Tune Hyperparameters

For a large class of regularized models, leave-one-out cross-validation ...

Approximate cross-validation formula for Bayesian linear regression

Cross-validation (CV) is a technique for evaluating the ability of stati...

Subsampling Bias and The Best-Discrepancy Systematic Cross Validation

Statistical machine learning models should be evaluated and validated be...

Computing AIC for black-box models using Generalised Degrees of Freedom: a comparison with cross-validation

Generalised Degrees of Freedom (GDF), as defined by Ye (1998 JASA 93:120...

The Early Roots of Statistical Learning in the Psychometric Literature: A review and two new results

Machine and Statistical learning techniques become more and more importa...

How is Machine Learning Useful for Macroeconomic Forecasting?

We move beyond "Is Machine Learning Useful for Macroeconomic Forecasting...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Multinomial classification is a ubiquitous task. There are several ways to treat this task, such as the naive Bayesian methods, neural networks, decision trees, and hierarchical classification schemes 

(Trevor et al., 2009). Among them, in this paper, we focus on multinomial logistic regression (MLR), which is simple but powerful enough to be used in many present day applications.

Let us denote each feature vector by

and its class by , where denotes the index of given data. The MLR uses a linear structural model with parameters and computes a class- bias as an overlap:


A probability such that the feature vector

belongs to the class is computed through a softmax function as:


These define the MLR.

The maximum likelihood estimation is usually employed to train the MLR, though the learning result tends to be inefficient when the data size is not sufficiently larger than the model dimensionality or noises in relevant levels are present. A common technique to overcome this difficulty is to introduce a penalty or regularization. In this paper, we use an

-regularization, which induces a sparse classifier as a learning result and is accepted to be effective. Given

data points , the -regularized estimator is defined by the following optimization problem:


where we denote the negative log-likelihood as and define a regularized cost function or Hamiltonian .

The introduction of regularization causes another problem of model selection or hyper-parameter estimation with respect to . A versatile framework providing a reasonable estimate is cross-validation (CV), but it has a disadvantage in terms of the computational cost. The literal CV requires repeated optimizations which can be a serious computational burden when the data size and the model dimensionality are large. The purpose of this paper is to resolve this problem by inventing an efficient approximation of CV.

Our technique is based on a perturbative expansion employing the largeness of the data size and the model dimensionality. Similar techniques were also developed for the Bayesian learning of simple perceptron and committee machine 

(Opper and Winther, 1996, 1997)

, for Gaussian process and support vector machine 

(Opper and Winther, 2000a, b; Vapnik and Chapelle, 2000)

, for linear regression with the

-regularization (Obuchi and Kabashima, 2016; Rad and Maleki, 2018; Wang et al., 2018) and with the two-dimensional total variation (Obuchi et al., 2017). Actually, this perturbative approach is fairly general and can be applied to a wide class of generalized linear models with simple convex regularizations. For example in the present MLR case, it is easy to extend our result to the case where both the - and -regularizations exist (elastic net, Zou and Hastie, 2005), which is used in a common implementation (Friedman et al., 2010). The derivation of our approximate formula below is, however, conducted on the case of the -regularization only, for simplicity. The extension to the elastic net case is stated after the derivation.

The rest of the paper is organized as follows. In sec. 2, we state our formulation and how to derive the approximate formula. In sec. 3, we compare our approximation result with that of the literally conducted CV on simulated data and on the ISOLET dataset from UCI machine learning repository (Lichman, 2013). The accuracy and the computational time of our approximate formula are reported in comparison with the literal CV. The limitation of a simplified version of the approximation is also examined. The last section is devoted to the conclusion.

2 Formulation

In the maximum likelihood estimation framework, it is natural to employ a predictive likelihood as a criterion for model selection (Bjornstad, 1990; Ando and Tsay, 2010). We require a good estimator of the predictive likelihood, and the CV provides a simple realization of it. Particularly in this paper, we consider an estimator based on the leave-one-out (LOO) CV. The LOO solution is described by


Denoting the overlap of with the LOO solution as , as well as that with the full solution , we can define the LOO estimator (LOOE) of the predictive negative log-likelihood as:


In the following, the predictive negative log-likelihood is simply called prediction error. The minimum of the LOOE determines the optimal value of though its evaluation requires us to solve eq. (6) times, which is computationally demanding.

2.0.1 Notations

Here, we fix the notations for a better flow of the derivation shown below. By summarizing the class index, we introduce a vector notation of the overlap as and an extended vector representation of the weight vectors as . The th component of can thus be decomposed into two parts as where denotes the class index and represents the component index of the feature vector. Namely we write . Correspondingly, we leverage a matrix to define a repetition representation of the feature vector : Each component is defined as:


This yields simple and convenient relations:


Further, the class- probability of th data at the full solution is denoted by:


These notations express the gradient and the Hessian of at the full solution as:




In addition, we denote the cost function Hessians at the respective solutions as:


Finally, we introduce the symbol representing the index set of the active components of and . Given , we denote the active components of a vector by the subscript as . A similar notation is used for any matrix and the symbol is assumed to represent all of the components in the corresponding dimension.

2.1 Approximate formula

For a simple derivation, it is important to consider that the -dependence of appears only in the overlap . Hence, it is sufficient to provide the relation between and in order to derive the approximate formula.

A crucial assumption to derive the formula is that the active set is “common” between the full and LOO solutions, and ; namely . Although this assumption is literally not true, we numerically confirmed that this approximately holds. In other words, the change of the active set is small enough compared to the size of the active set itself when considering the LOO operation when and are large. Moreover, in a related problem of an -regularized linear regression, the so-called LASSO, it has been shown that the contribution of the active set change vanishes in a limit keeping  (Obuchi and Kabashima, 2016). It is expected that the same holds in the present problem. Hence, we adopt this assumption in the following definition. Note that this idea of the active set constantness can be found in preceding analyses of support vector machine (Opper and Winther, 2000b; Vapnik and Chapelle, 2000).

Once the active set is assumed to be known and unchanged by the LOO operation, it is easy to determine the active components of the full and LOO solutions and . The vanishing condition of the gradient of the cost function is the determining equation:


The difference between the gradients is only , and hence the difference between and is expected to be small. Denoting the difference as and expanding eq. (19) with respect to up to the first order, we obtain an equation determining :


Inserting this and eq. (12) into the definition and multiplying from left, we obtain:


This equation implies that the matrix inversion operation is necessary for each , which still requires a significant computational cost. To avoid this, we employ an approximation and the Woodbury matrix inversion formula in conjunction with eqs. (13,16,17). The result is:


Inserting this into eq. (21) and simplifying several factors, we obtain:




Now, all of the variables on the righthand side of eq. (24) can be computed from the full solution only, which enables us to estimate the LOOE by leveraging a one-time optimization using all of the data , while avoiding repeated optimizations.

We should mention the computational cost of this approximation: it is mainly scaled as . The first two terms come from the construction of and , and the last one is derived from the inverse of . If is proportional to the feature dimensionality , this computational cost is of the third order with respect to the system dimensionality and . This is admittedly not cheap and the computational cost for the -fold literal CV with a moderate value of becomes smaller than that for our approximation in a large dimensionality limit. We, however, stress that there actually exists a wide range of and values in which our approximation outperforms the literal CV in terms of the computational time, as later demonstrated in sec. 3. Moreover, for treating much larger systems, we invent a further simplified approximation based on the above approximate formula. The computational cost of this simplified version is scaled only linearly with respect to the system parameters and . Its derivation is in sec. 2.2 and the precision comparison to the original approximation is in sec. 3.

Another sensitive issue is present in computing . Occasionally the cost function Hessian

has zero eigenvalues and is not invertible. We handle this problem in the next subsection.

2.1.1 Handling zero modes

In the MLR, there is an intrinsic symmetry such that the model is invariant under the addition of any constant vector to the weight vectors of all classes:


In this sense, the weight vectors defining the same model are “degenerated” and our MLR is singular. For finite , this is not harmful because the regularization term resolves this singularity and selects an optimal one with the smallest value of among the degenerated vectors. However, this does not mean that the associated Hessian is non-singular. The regularization term does not provide any direct contribution to the Hessian and as a result, the Hessian tends to have some zero modes. This prevents taking the inverse Hessian in eq. (25). How can we overcome this?

One possibility is to fix the weights of one certain class at constant values when solving the optimization problem (4). This is termed “gauge fixing” in physics, and one convenient gauge in the present problem will be the zero gauge in which the weights in a chosen class are fixed at zeros. This is actually found in some earlier implementations (Krishnapuram et al., 2005; Schmidt, 2010) and is preferable for our approximate formula because it removes the harmful zero modes of the Hessian from the beginning. However, some other implementations which are currently well accepted do not employ such gauge fixing (Friedman et al., 2010), and moreover even with gauge fixing very small eigenvalues sometimes accidentally emerge in the Hessian. Hence, for user convenience, we require another way of avoiding this problem.

Another possibility is to remove the zero modes by hand. By construction, the zero modes are associated to the model invariance. This implies that those zero modes are irrelevant and may be removed. In fact, we are only interested in the perturbations which truly change the model, and the modes which maintain the model unchanged are unnecessary. According to this consideration, we replace in eq. (25) with the zero-mode-removed inverse Hessian . The computation of is straightforward: we perform the eigenvalue decomposition of and obtain the eigenvalues

and eigenvectors

, which allows us to represent


where denotes the index set of the modes with finite eigenvalues. Then, is defined as:


Finally, we replace by in eq. (25), and obtain:


By using this instead of eq. (25), the problem caused by the zero modes can be avoided.

2.1.2 Extension to the mixed regularization case

Let us briefly state how we can generalize the present result to the case of the mixed regularizations of the - and -terms (elastic net, Zou and Hastie, 2005). The problem to be solved can be defined as follows:


where denotes the norm. Following the derivation in sec. 2.1, we realize that the derivation is essentially the same, and the difference only appears in the cost function Hessian:



is the identity matrix of size

. As a result, we can compute the LOO solution by leveraging the same equation as eq. (24) by replacing the definition of , eq. (25), with:


Thanks to the term, the zero mode removal is not needed since the eigenvalues are lifted up by .

2.1.3 Binomial case

The binomial case is particularly interesting in several applications and thus we write down the specific formula for this case.

In the binomial case, it is fairly common to express the class as a binary

and to use the following logit function:




If we identify in this case as in the two-class MLR case, this is nothing but the two-class MLR with a zero gauge . Hence, there is no harmful zero mode in the Hessian and we can straightforwardly apply our approximate formula. The explicit form in this case is:


where and


and is the active set of the full solution, as before.

Note that this approximation can be easily generalized to arbitrary differentiable output functions by replacing the logit function . Readers are thus encouraged to implement approximate CVs in a variety of different problems.

2.2 Further simplified approximation

As mentioned above, the computational cost of our approximation is and should be reduced for treating larger systems. For this, we derive a further simplified approximation based on the invented approximate formula above. We call this a self-averaging (SA) approximation according to physics terminology.

The basic idea for simplifying our approximate formula is to assume that correlations between and are sufficiently weak. The meaning of “correlation” is not evident here, but as seen in sec. A the Hessian can be connected to a (rescaled) covariance between and

in a statistical mechanical formulation introducing a probability distribution of

. Our weak correlation assumption requires that the correlation between different feature components is negligibly small; , where are the class indices and are the feature component indices defined thus far, and is the rescaling factor. In this way, the Hessian is assumed to be expressed in a rather restricted form:


Namely, the SA Hessian is allowed to take finite values if and only if the two indices share the same feature vector component. The dependence on the data index is also assumed to be negligible, implying that strong heterogeneity among feature vectors is assumed to be absent.

To proceed with the computation, we require a closed equation to determine the matrix for . Its derivation is rather technical and is deferred to sec. A. The result is:


where and is the set of active class variables at the feature component ; the other components of related to inactive variables are zeros. The SA approximation of , , is defined by:


Using the solution of eqs. (41,42), the approximate formula is now simply expressed as:


Note that there is no factor like in contrast to eq. (24), because we directly approximate in eq. (21).

When solving eqs. (41,42), the inverse at the right-hand side of eq. (41) becomes occasionally ill-defined again due to the presence of zero modes. In such cases, we should remove the zero modes as eq. (28). Putting and performing the eigenvalue decomposition, we define its zero-mode-removed inverse as:


where is the index set of the modes with finite eigenvalues. This requires a computational cost at a maximum. Leveraging this approach, a naive way to solve eqs. (41,42) is a recursive substitution. If this converges in a constant time, irrespectively of the system parameters and , then the computational cost of the SA approximation is scaled as . This is linear in the feature dimensionality and the data size and hence, its advantage is significant.

2.3 Summary of procedures

Here, we summarize the two versions of the approximation derived thus far as algorithmic procedures. We call the first version, based on eq. (24), the approximate CV or ACV, and call the second one, using eq. (43), the self-averaging approximate CV or SAACV. The procedures of ACV and SAACV are given in Alg. 1 and Alg. 2, respectively; they are written for the case of the mixed regularization (30).

1:procedure ACV()
2:     Compute the active set from
3:     Compute by eqs. (1,9), (14) and (15)
5:     if  is large enough then
7:     else
8:         Compute by eq. (28)
9:     end if
10:     for  do
13:     end for
14:     Compute from by eq. (8)
15:     return
16:end procedure
Algorithm 1 Approximate CV of the MLR
1:procedure SAACV()
2:     Compute the active sets from
3:     Compute by eqs. (1,9), (14) and (15)
4:      Start initialization
5:     for  do
6:         ,
7:         ,
8:     end for
9:      End initialization
10:     while  do Compute by recursion
14:         for  do
15:              if  is large enough
17:              else
18:                  Compute by eq. (44) from
19:              end if then
22:         end for
25:     end while
26:     for  do
28:     end for
29:     Compute from by eq. (8)
30:     return
31:end procedure
Algorithm 2 Self-averaging approximate CV of the MLR

Comments are added for specifying the time consuming parts in the entire procedures. In Alg. 2, we describe an actual implementation for solving by recursion, which is not fully specified in sec. 2.2. The symbol denotes the Frobenius norm and we set the threshold judging the convergence as in typical situations. We also set as the threshold judging if is large or not.

3 Numerical experiments

In this section, we examine the precision and actual computational time of ACV and SAACV in numerical experiments. Both simulated and actual datasets (from UCI machine learning repository, Lichman, 2013) are used.

For examination, we compute the errors also by literally conducting -fold CV with some s, and compare it to the result of our approximate formula. In principle, we should compare our approximate result with that of the LOO CV () because our formula approximates it. However for large , the literal LOO CV requires huge computational burdens despite that the result is empirically not much different from that of the -hold CV with moderate s. Hence in some of the following experiments with large , we use the -hold CV instead of the LOO CV. Further, to directly check the approximation accuracy, we also compute the normalized error difference defined as


where denotes the literal CV estimator of the prediction error while is the approximated LOOE. Moreover, as a reference, we compute the negative log-likelihood of the full solution as:


and call it the training error, hereafter. The training error is expected to be a monotonic increasing function with respect to , while the prediction one is supposed to be non-monotonic.

In all of the experiments, we used a single CPU of Intel(R) Xeon(R) E5-2630 v3 2.4GHz. To solve the optimization problems in eqs. (4,6), we employed Glmnet (Friedman et al., 2010) which is implemented as a MEX subroutine in MATLAB®. The two approximations were implemented as raw codes in MATLAB. This is not the most optimized approach, because as seen in Algs. 1,2 our approximate formula uses a number of for and while loops which are slow in MATLAB, and hence the comparison is not necessarily fair. However, even in this comparison there is a significant difference in the computational time between the literal CV and our approximations, as shown below.

In Glmnet, the corresponding optimization problem is parameterized as follows:


In the following experiments, we present the results based on this parameterization. We basically prefer in which the term is absent, because the main contribution of the present paper is to overcome technical difficulties stemming from the term. However, Glmnet or its employing algorithm occasionally loses its stability in some uncontrolled manner without the term. Hence, in the following experiments we adaptively choose the value of .111When employing our distributed codes implementing the approximate formula (Obuchi, 2017; Takahashi and Obuchi, 2017) in conjunction with Glmnet, the parameters and are read as and .

A sensitive point which should be noted is the convergence problem of the algorithm for solving the present optimization problem. In Glmnet, a specialized version of coordinate descent methods is employed, and it requires a threshold to judge the algorithm convergence. Unless explicitly mentioned, we set this as being tighter than the default value. This is necessary since we treat problems of rather large sizes. A looser choice for rather strongly affects the literal CV result, while it does not change the full solution or the training error as much. As a result, our approximations employing only the full solution are rather robust against the choice of compared to the literal CV. This is also demonstrated below.

3.1 On simulated dataset

Let us start by testing with the simulated data. Suppose each “true” feature vector is independently identically drawn (i.i.d.) from the following Bernoulli-Gaussian prior:



denotes a Gaussian distribution whose mean and variance are

and , respectively. The resultant feature vector becomes -sparse and its norm becomes on average. Then, we choose a class from uniformly and randomly, and generate an observed feature vector by leveraging the following linear process:


where is an observation noise each component of which is i.i.d. from a Gaussian .

For convenience, we introduce the ratio of the data size to the feature dimensionality, , and now obtain five parameters characterizing the experimental setup. It is rather heavy to obtain the dependence of all parameters and below, and hence we mainly focus on the dependence on , , and . Other parameters are set to be and .

3.1.1 Result

Let us summarize the result on simulated data.

Fig. 1 shows the plots of the prediction and training errors against for at and .

Figure 1: (Upper) Log-log plots of the errors against for several values of the class number . Other parameters are fixed at , , and . The approximation results are consistent with the literal LOO CV results, except at small s, which is presumably due to a numerical instability occurring in the literal CV at small s. Here, . (Lower) The normalized error difference (45) plotted against . The parameters of each panel are them of the corresponding upper one. The horizontal dotted lines denote and drawn for comparison. For small s the difference is not negligible, but the literal CV itself is not stable in that region and hence the error difference is not reliable.

This demonstrates that both approximations provide consistent results with the literal LOO CV, except at small s. This inconsistency at small s is considered to be due to a numerical instability occurring in the literal CV. Actually, for small s, we have observed that certain small changes in the data induce large differences in the literal CV result. This example demonstrates that our approximations provide robust curves even in such situations. Note that as grows the number of estimated parameters increases while the data size is fixed, meaning that the problem becomes more and more underdetermined with the growth of . Hence, Fig. 1 demonstrates that the developed approximations work irrespectively of how much the problem is underdetermined.

Fig. 2 exhibits the -dependence of the errors and the approximation results for and .

Figure 2: (Upper) Log-log plots of the errors against for several noise strengths. Other parameters are fixed at , , and . The approximation results are consistent with the literal LOO CV, irrespectively of the noise strength. The convergence threshold is set to be for the case . Here, . (Lower) The normalized error difference (45) plotted against . The parameters of each panel are them of the corresponding upper one. In the whole region the difference is negligibly small.

For the very weak noise case (, left), the difference between the predictive and training errors is negligible and hence all four curves are not discriminable. For the moderate (, middle) and large (, right) noise cases, the training curve is very different from the predictive ones. The approximation curves are again consistent with the literal LOO one.

Fig. 3 demonstrates how the approximation accuracy changes as the system size grows.

Figure 3: Log-log plots of the errors against for several values of feature dimensionality . Other parameters are fixed at , , and . Here, .

For small sizes , a discriminable difference exists between the results of the approximations and the literal LOO CV, as well as the difference between the results of the two approximations. This is expected, because our derivation relies on the largeness of and . For large systems , the difference among the two approximations and the literal CV is much smaller. Considering this example in conjunction with the middle panel of Fig. 1, we can recognize that our approximate formula becomes fairly precise for in this parameter set. The normalized error difference corresponding to Fig. 3 is shown in Fig. 4.

Figure 4: The plot of the normalized error difference corresponding to Fig. 3. The difference tends to be smaller as the system size increases.

We can observe that the difference tends to be smaller as the system size increases, which is expected because the perturbation employed in our approximate formula is justified in the large limit.

Finally, let us consider the actual computational time to evaluate and the approximate LOOEs, and observe its system size dependence. The left panel of Fig. 5 provides the plot of the actual computational time against the system size. Here, the number of examined points of to obtain a solution path is different from size to size, and hence the plotted time is given as the whole computational time to obtain the solution path divided by the number of s points.

Figure 5: (Left) Actual computational time spent to find the solution of eq. (4) and that for ACV and SAACV, plotted against the feature dimensionality in a double logarithmic scale. Note that the computational time for the -fold CV is about times larger than that for finding the solution of eq. (4), represented by the green asterisks. Parameters are fixed at , , and . Here, . (Right) The errors are obtained for the two convergence thresholds and . Error bars are omitted for visibility. For the tighter case , the minimum value of in the examined range is larger than that of the case , though the systematic difference with the results of the literal LOO CV is already clear. The training errors of these two different , represented by black circles and left-pointing triangles, are completely overlapping. The system parameters are , , , and . Here, .

The left panel of Fig. 5 clearly displays the advantage and disadvantage of the developed approximations. For small sizes, the computational time for optimization to obtain is shorter than the time to compute the approximate LOOEs, and hence the literal CV is better. However, for larger systems, the optimization cost increases rapidly and for the approximate CV is better. For