 # Variable projection without smoothness

The variable projection technique solves structured optimization problems by completely minimizing over a subset of the variables while iterating over the remaining variables. Over the last 30 years, the technique has been widely used, with empirical and theoretical results demonstrating both greater efficacy and greater stability compared to competing approaches. Classic examples have exploited closed form projections and smoothness of the objective function. We apply the idea in broader settings, where the projection subproblems can be nonsmooth, and can only be solved inexactly by iterative methods. We illustrate the technique on sparse deconvolution and robust machine learning applications. Open source code for nonsmooth variable projection is available through github

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

This paper revolves around the Variable Projection technique in nonsmooth optimization. One of the prominent early references on the topic 

concentrates on separable least-squares problems, having numerous applications in chemistry, mechanical systems, neural networks, and telecommunications; see the surveys of

 and , and references therein. Setting the stage, consider a separable least-squares problem

 minx,θf(x,θ):=∥y−Φ(x)θ∥22, (1)

where the matrix-valued map is smooth in the parameter . One may formally eliminate the variable by rewriting the problem:

 minx ˜f(x) where ˜f(x):=minθf(x,θ). (2)

Though is implicitly defined, it can be explicitly evaluated, since is a convex quadratic in . The variable projection technique for this class of problems, going back to , then aims to solve the original problem (1) by instead running a nonlinear optimization solver on . The authors of  showed that when the Gauss-Newton method for (1) converges superlinearly, so do Gauss-Newton variants for (2); moreover, empirical evidence suggests that the latter schemes outperform the former.

The underlying principle is much broader than the class of nonlinear least squares problems. For example, the authors of [5, 4] consider the class of problems

 minx,θ f(x,θ), (3)

where is a

-smooth function, the vector

models the primary set of parameters, while

is a secondary set of nuisance parameters (such as variance, tuning, or regularization). Applying the variable projection technique requires approximate evaluation of the function

along with its first (and potentially second) derivatives; such formulas are readily available under mild conditions .

There is significant practical interest in nonsmooth functions , generalizing (3). Nonsmooth regularization in both and occurs often in high dimensional and signal processing settings, as well as in PDE constrained optimization, and machine learning problems. In many cases, problem-specific projection-based algorithms have been developed. Our goal is to provide a unifying framework for a wide range of problems, illustrating its current uses in a range of fields, and to explore new applications.

The outline of the paper is as follows. In Section 2, we present important applications of nonsmooth variable projection algorithms. Throughout, we emphasize the subtleties involved, and in several cases propose novel techniques. In Section 3, we justify some technical claims made in Section 2, using basic techniques of variational analysis. Section 4 presents promising numerical results with applications to exponential fitting, sparse deconvolution, and robust learning formulations. Conclusions complete the paper.

## 2 Nonsmooth Variable Projection in Applications

As alluded to in the introduction, the variable projection technique has been widely used in different application domains. In this section, we describe a number of notable examples: PDE-constrained optimization, exponential data fitting, robust inference, and multiple kernel learning. We emphasize that despite the variety of these applications, the underlying algorithmic techniques all fall within the same paradigm. However, there are subtle modifications that may be required to make the scheme work, which we emphasize when appropriate. As a warm-up for the general technique considered, we begin with ODE or PDE constrained problems.

### 2.1 PDE-constrained optimization

Optimization problems with ODE and PDE constraints occur often when modeling physical systems; see e.g. optimal control [10, 22] and inverse problems in geophysics [13, 6] astrophysics  and medical imaging . Setting notation, consider the problem

 minx,u g(x,u)s.t.c(x,u)=0, (4)

where is the state variable (a vector field), the system is an ODE or PDE with parameters of interest , and is an objective that we aim to minimize (e.g. data misfit). Here we assume that is lower bounded and convex. Upon discretization, becomes a high-dimensional vector. For the current exposition, we will assume that the PDE is linear. Abusing notation slightly, we assume that the system (4) is already discretized, and is in the form

 minx,u g(x,u)s.t.H(x)u=q, (5)

where

is a square invertible matrix depending

-smoothly on , the function is -smooth and the margin function is convex for each .

In large-scale applications, it is impractical to store and update the entire vector . Instead, adjoint state methods compute on the fly and optimize in alone. Observe that the problem is equivalent to minimizing the smooth function and it is well-known [13, 23] that is given by

 ∇˜f(x)=∇xg(x,¯u)+∇⟨¯v,H(⋅)¯u⟩(x),

where satisfy the equations

 (6)

A variational viewpoint elucidates the formula. Indeed, basic convex duality (e.g. [7, Corollary 3.3.11]) implies that can equivalently be written as

 ˜f(x)=supvminu g(x,u)+⟨v,H(x)u−q⟩. (7)

Now observe that equation (6) simply says that is a saddle point of (7) while the description of is the partial gradient with respect to of the inner function in (7) evaluated at .

##### Penalty methods for PDE constrained optimization

In some applications, it may be desirable to relax the constraint and formulate a penalized problem introduced in 

 minx,u g(u,x)+λ⋅p(H(x)u−q),

where is a penalty parameter and is strongly convex, smooth penalty function (e.g. ). The variable projection technique is immediate here. An easy application of [Theorem 10.58] shows that the function

 ˜g(x):=minu g(u,x)+λ⋅p(H(x)u−q) (8)

is differentiable and its derivative is simply the partial derivative of the inner function in evaluated at , where achieves the minimum. For an extended explanation, see Section 3. Notice that in case of the least squares objective , only one equation solve is needed to compute and hence the derivate . This is in contrast to the adjoint method where another equation solve is needed to compute the adjoint variable . For promising numerical results, see . Hence the variable projection technique, even in this rudimentary setting, can yield surprising computational benefits.

### 2.2 Exponential data-fitting

In this section, we present the general class of exponential data-fitting problems – one of the prime applications of variable projection . The general formulation of these problems starts with a signal model of the form

 yi=n∑j=1ajexp(−φij(θ)),

where are unknown weights, are the measurements, and are given functions that depend on an unknown parameter . Some examples of this class are given in table 1.

Introducing the matrix with entries , we express the noisy measurements as

 y=Φ(θ)a+ϵ.

A natural approach is then to formulate a nonlinear least squares problem,

 mina,θ ∥Φ(θ)a−y∥22,

which is readily solved using the classic variable projection algorithm. In practice, however, we may not know the number of terms, , to include. We can of course over-parametrize the problem, choosing a rather large value for

, but this may lead to unreliable estimates. By including a sparsity prior on

we force a sparser solution. We arrive at the formulation:

 mina,θ ∥Φ(θ)a−y∥22+λ∥a∥1.

Note also that we can replace the data misfit term to account for the statistics of the noise. Additional constraints on may be added as well [19, 33]. The variable projection approach allows us to efficiently solve this problem by projecting out using any suitable method to solve LASSO problems (we use IPsolve ) and using a non-linear optimization method to find the optimal . Indeed, we will see in Section 3 that provided has full column rank for all , the function

 ˜f(θ)=minaf(θ,a):=∥Φ(θ)a−y∥22+λ∥a∥1, (9)

is differentiable with gradient where achieves the minimum. If does not have full column rank, we can add a quadratic regularizer in and obtain an analogous result. We note that numerical optimization in both variables , without variable projection, performs poorly in comparison to the reduced approach; see section 4 and figure 2.

### 2.3 Trimmed Robust Formulations in Machine Learning

Data contamination is a significant challenge for a range of inference problems. An alternative to robust penalties is the trimmed approach [15, 28], where one solves an inference problem by minimizing over least residuals. Suppose that our inference problem can be written as

 minx∑igi(x)+r(x),

where represent the error or the negative log-likelihood of the ’th component, while is any regularizer. The trimmed approach aims to only use

“best” components, treating the rest as “outliers” to be excluded. However, correct classification of these residuals can only be done once

is known. A natural approach is to use the smallest residuals at each iteration of a numerical scheme. Recently, such a scheme was used to develop robust graphical models , using a sparsifying regularizer .

Consider the reformulation

 minx (minw∈Δk∩B∞∑iwigi(x))+r(x), (10)

where is a scaled simplex, and is the infinity norm unit ball. Variable projection naively seems directly applicable, since the function simply selects the smallest ’s. However, it is easy to see that this function is nonsmooth in the simplest of cases; see Fig. 1. Figure 1: g1(x)=(x−1)2 and g2(x)=(x+1)2. The resulting function min{g1,g2} is nonsmooth at the origin (black solid curve). The effect of the smoothing formulation (11) shown in dashed blue for β=1, and dash-dot red for β=0.5.

Instead, we propose a regularized formulation:

 minx (minw∈Δk∑iwigi(x)+β2∥w∥2)+r(x). (11)

We then seek to solve the problem , where

 ˜gβ(x) :=minw∈Δk∩B∞∑iwigi(x)+β2∥w∥2 =β2dist2Δk∩B∞(−β−1g(x))−(2β)−1∥g(x)∥2

Here we use the notation . A standard computation shows that is indeed differentiable with gradient , where is the nearest point to in the set of interest . Projection onto this set has been recently considered and implemented . Hence standard methods are directly applicable to solve (11), including prox-gradient for non-smooth , provided

is Lipschitz continuous. In the numerical section of this paper, we illustrate the approach on a logistic regression model with data contamination. In section

3, we observe that the regularization technique just described fits into a broader paradigm.

This example illustrates the importance of a theoretical framework underlying algorithmic development. While (10) is not differentiable, a small modification (11) is, and shows good performance in applications.

### 2.4 Multiple Kernel Learning

Kernel methods are a powerful technique in classification and prediction [32, 34]. In such problems we are given a set of samples and corresponding labels

and the goal is to classify new samples. To do so, we search for a function

, with a given map from to a specified (possibly infinite-dimensional) function class (Reproducing Kernel Hilbert Space [29, 31]), such that . We can then use to classify new samples. This problem can be formalized as follows

 minv∈H,b∈R,ξ∈Rm+ 12∥v∥2H+Cm∑i=1ξi s.t. yi(⟨v,Φ(xi)⟩+b)≥1−ξifor i=1,2,…,m,

The dual to this problem is always a finite dimensional Quadratic Program:

 minα∈[0,C]m12∥α∥2K−1Tαs.t.m∑i=1αiyi=0, (12)

where is the Kernel matrix given by . Once the dual is solved, is recovered via .

The choice of kernel is an art-form; there are many options available, and different kernels perform better on different problems. To develop a disciplined approach in this setting, the multiple kernel learning framework has been proposed. In this framework, we suppose that we have a choice of kernels functions with corresponding kernels . We can now consider the linear combination

 K(w)=M∑i=1wiKi,

where are some weights. A natural question is to find the best weighted combination; such an approach has been proposed in . Specifically, requiring to be in the unit simplex yields the problem

 minw∈Δ1ˆf(w):={minα∈[0,C]m12∥α∥2K(w)−1Tαs.t.m∑i=1αiyi=0}.

This problem was solved in  by using variable projection, with the outer problem solved by prox-gradient. Notice that each iteration requires a complete solve of a Quadratic Program.

Our novel approach switches which variable is projected out. Analogously to the trimmed regression case, a naive approach is given below:

 minα∈[0,C]m˜f(α)−1Tαs.t.m∑i=1αiyi=0, (13)

where

 ˜f(α)=minw∈Δ112∥α∥2K(w)=minw∈Δ112∑iwi⟨Kiα,α⟩ (14)

As in trimmed regression, this approach does not work, as can be nonsmooth. We use the same smoothing modification, considering a modified function

 ˜fβ(α) =minw∈Δ112∑iwi⟨Kiα,α⟩+β2∥w∥2 (15) =β2dist2Δk(−β−1g(α))−(2β)−1∥g(α)∥2

where . Now, is smooth, with , where is the nearest point to in the unit simplex. This projection has also been studied and implemented . With the smoothing extension, standard approaches can be applied to solve (13) with the modified . In our numerical experiments, demonstrate the efficiency of this novel formulation.

## 3 Theory

In this section, we discuss some of the claimed derivative formulas in the aforementioned applications. To this end, recall that a function is differentiable at if there exists a vector , called the gradient and denoted , satisfying

 limy→¯xf(y)−f(¯x)−⟨v,y−¯x⟩∥y−¯x∥=0.

On the other hand, a function that is differentiable at is strictly differentiable at if it satisfies the slightly stronger property

 limz,x→¯xf(z)−f(x)−⟨∇f(¯x),z−x⟩∥z−x∥=0.

The following fundamental result [Theorem 10.58] is key for establishing derivative formulas for in (8) and for in (9). Strict differentiability, in particular, implies local Lipschitz continuity of the function, in contrast to conventional differentiability [25, Theorem 9.18].

###### Theorem 1 (Derivative of the projected function).

Consider a function satisfying the following properties

1. is continuous;

2. is level-bounded in locally uniformly in , meaning that for any and any compact set , the union of sublevel sets is bounded.

3. gradient exists for all and depends continuously on .

Define now the projected function and the minimizing set:

 ˜f(x):=infzf(x,z) and P(x):=\operatornamewithlimitsargminzf(x,z).

Then is strictly differentiable at any point for which the set is a singleton, and in this case equality holds.

Let us see how this theorem applies. Consider the function in the formulation (8). Clearly the inner function satisfies properties 1 and 3 of Theorem 1. Property 2 follows quickly from strong convexity of and the assumptions that is lower bounded and is invertible. Then the set is clearly a singleton, and the claimed derivative formula follows (even if is nonsmooth).

Next, consider the function in (9). Again, properties 1 and 3 of Theorem 1 hold. To justify property 2, we must assume that has full column rank for all . Then property 2 follows immediately. Since the objective is strongly convex in , the set is a a singleton and the derivative formula follows.

Next note that there are important limitations to Theorem 1. We already saw that the naive projected function in equation (10) can easily be nonsmooth; see figure 1. The same occurs for the naive formulation (14). The difficulty is that the parameter that is projected out varies over a constrained set, and hence properties 1 and 3 decisively fail.

As we saw in sections 2.3 and 2.4, we can circumvent these issues by smoothing the projected problem while still preserving the structure of the solution set (feasibility, sparsity, etc.) This technique seems promising much more generally for problems of the form

 minx,z g(z,x):=h(z,g(x))+r(x),

where , , and satisfy appropriate conditions. Such a smoothing technique in nonsmooth optimization appears to be new, and will be the subject of further investigation.

## 4 Numerical Illustrations

#### Direction-of-arrival estimation (DOA)

A classical problem in array-based signal processing is the estimation of the direction-of-arrival. In a typical setup, incoming signals from multiple sources (e.g., electromagnetic or acoustic waves) are recorded by an array of receivers. The goal is to identify the directions from which various contributions originated. This problem falls in the class of exponential data-fitting, as an incoming plane wave can be described by a complex exponential. The signal model is

 y=Φ(θ)a+ϵ,

where denotes the signal, is a matrix with elements , are the amplitudes of the individual components; are the directions and are the locations of the receivers.

The classical way of estimating the parameters is the MUSIC algorithm [30, 8]. This algorithm proceeds to estimate the DOA as follows. First, we treat both and

as random variables and define the signal correlation matrix

 Σθ=Ea{Φ(θ)aa∗Φ(θ)∗},

which is an matrix with rank . We let be the matrix whose columns span the null-space of . The MUSIC pseudo-spectrum is now defined as

 f(θ)=1∥P∗s(θ)∥2,

where . If is nearly in the range of , its projection onto the null-space is nearly zero, thus causing a large peak in the pseudo-spectrum. The components in the signal can then be detected by taking the largest peaks in the pseudo-spectrum.

In practice, we do not have access to the signal correlation. Instead, we construct the sample average

 Σy≈1NN∑i=1yiy∗i.

If the is i.i.d. Gaussian with variance and independent of , we have

 Σy=Σθ+σ2I,

in which case we construct from the singular vectors of

with singular values

. Proper identification of the null-space requires an estimate of the noise level.

##### MUSIC vs. Nonsmooth DOA (full and reduced)

An example with , and is shown in figure 2. Note that the observed data contains arrivals from only 3 distinct directions. We solved the LASSO problems using an interior point method (IPsolve ) while using a Quasi-Newton method to solve the outer optimization problem. We compare this variable projection approach to solving the full optimization problem in jointly using a non-smooth Quasi-Newton method . Figure 2: (a) observed and fitted data, (b) true and estimated direction-of-arrival and corresponding amplitudes, as well as the MUSIC spectrum, and (c) convergence of the full and reduced approach.

The observed and fitted data are shown in figure 2 (a) and the true and estimated DOAs and amplitudes as well as the MUSIC spectrum are shown in figure 2 (b). The MUSIC spectrum clearly picks up the strongest modes but fails to recover the weaker mode. Both the full and variable projections approaches correctly estimate both the directions and amplitudes of the 3 arrivals. The full approach, however, fails to converge as can be seen in 2 (c).

We repeated the above experiment for 100 different initial guesses and summarized the results in terms of the number of iterations required to reach the desired tolerance of and optimality (relative norm of the gradient) in figure 3. We see that the reduced approach almost always reaches the desired tolerance and requires less iterations to do so. In contrast, the full approach fails to reach the desired tolerance in roughly 35/100 cases. Figure 3: Results of reduced versus full-space optimization for 100 different initial guesses, with a required tolerance of 10−6. (a) shows the relative tolerance upon convergence (or failure) of the optimization, (b) shows the number of iterations required by both methods. We see that the reduced approach almost always reaches the desired tolerance and requires less iterations than the full approach.
##### Blind sparse deconvolution: full vs. reduced

In this example, we discuss a parametrized blind deconvolution problem. We are given a noisy image which we want to decompose into a number of parametrized basis functions. Such problems occur frequently in astrophysics , image deblurring [20, 2] and seismology [26, 17].

In this case we we let and use the interior point method IPsolve  to solve the LASSO problems with positivity constraints. The outer optimization over is done using a Quasi-Newton method. We compare the VP approach to a full optimization over the -space. The results are shown in figure 4. The ground truth image consists of four blobs; the initial guess for is obtained by picking the 50 largest peaks in the noisy image. The reconstructions obtained by the reduced and full approaches look similar, but we see that the full approach has trouble getting rid of the superfluous points. The convergence plots, shown in figure 5, show that the full approach does not fit the data as well as the reduced approach and converges very slowly. Figure 4: (a) ground truth image, (b) noisy image and initial guess for c, (c) results using the reduced approach, and (d) results using the full approach. Figure 5: Convergence of the full and reduced approaches in terms of the misfit (a) and the optimality (b).
##### Robust logistic regression using variable projection

Logistic regression is a popular alternative to support vector machines that seeks to find the best hyperplane to separate two classes of points. It can be formulated as an optimization problem as follows:

 minθf(θ):=∑ilog(1+exp(−yixTiθ))+R(θ), (16)

where represents the th feature, is a class label, describes the hyperplane we seek, and is a regularizer; in this example, we use a small multiple of as the regularizer. In this section, we consider a classification problem where a portion of the data has been contaminated. We pick features in , and we replace 10% (200 features) with random noise, with entries on average ten times larger than those of actual features. The smoothed trimmed approach for this problem is given by

 minθ˜f(θ)=minw∈Δk∩B∞{∑iwilog(1+exp(−yixTiθ))+β2∥w∥2}+R(θ), (17)

as described in section 2. We then compare three scenarios:

1. Data has not been contaminated, standard logistic regression used

2. Data has been contaminated, standard logistic regression used

3. Data has been contaminated, robust logistic regression (17) used with .

Note that the algorithm doesn’t use information about the exact number of contaminated features. Rather, it assumes that at least half of the data is good. The results were nearly identical without using regularization . We used for the smoothing.

##### Fast MKL using variable projection

In this example, we show how variable projection can be used for Multiple Kernel Learning. We solve the smoothed dual problem (15) using SQP with approximate Hessian . We consider two parametrized classes of kernel functions, polynomial and Gaussian:

 kpolp(x,x′)=(1+xTx′)p,
 kGausss(x,x′)=exp(−∥x−x′∥22/a2).

As an illustrative example, we consider classifying points in to the left and to the right of an elliptic curve. In figure 6, we show the results for various kernels. In table 3, we show the number of iterations required for each kernel. Figure 6: Classification of points in the plane using different kernels. The red and blue stars indicate the training set, the solid line denotes the elliptic curve used to separate the points and the dotted line shows the classification resulting from the optimization procedure.

In figure 7 we show the result using a total of 12 kernels; 5 polynomial kernels with and 7 Gaussian kernels with parameters . Our approach quickly hones in on the Gaussian kernel with and takes a total of 107 iterations. Figure 7: Classification of points in the plane using the new multiple kernel learning approach.

## 5 Conclusions

Variable projection has been successfully used in a variety of contexts; the popularity of the approach is largely due to its superior numerical performance when compared to joint optimization schemes. In this paper, we considered a range of nonsmooth applications, illustrating the use of variable projection for sparse deconvolution and direction of arrivals estimation. We showed that differentiability of the projected function can be understood using basic variational analysis, and that this approach has limitations and can fail for interesting cases. In particular, for robust formulations and multiple kernel learning, we showed that the projected function can fail to be differentiable. To circumvent this difficulty, we proposed a novel smoothing technique for the inner problem that preserves feasibility and structure of the solution set, while guaranteeing differentiability of the projected function. Numerical examples in all cases showed that variable projection schemes, when appropriately applied, are highly competitive for a wide range of nonsmooth applications.

Acknowledgment.

Research of A. Aravkin was partially supported by the Washington Research Foundation Data Science Professorship. Research of D. Drusvyatskiy was partially supported by the AFOSR YIP award FA9550-15-1-0237. The research of T. van Leeuwen was in part financially supported by the Netherlands Organisation of Scientific Research (NWO) as part of research programme 613.009.032.

## References

•  G. S. Abdoulaev, K. Ren, and A. H. Hielscher. Optical tomography as a PDE-constrained optimization problem. Inverse Problems, 21(5):1507–1530, oct 2005.
•  M. S. C. Almeida and M. a. T. Figueiredo. Parameter estimation for blind and non-blind deblurring using residual whiteness measures. IEEE Transactions on Image Processing, 22(7):2751–2763, 2013.
•  A. Y. Aravkin, J. V. Burke, and G. Pillonetto. Sparse/robust estimation and kalman smoothing with nonsmooth log-concave densities: Modeling, computation, and theory. Journal of Machine Learning Research, 14:2689–2728, 2013.
•  A. Y. Aravkin and T. van Leeuwen. Estimating nuisance parameters in inverse problems. Inverse Problems, 28(11):115016, nov 2012.
•  B. Bell and J. Burke. Algorithmic differentiation of implicit functions and optimal values. In C. H. Bischof, H. M. Bücker, P. D. Hovland, U. Naumann, and J. Utke, editors, Advances in Automatic Differentiation, pages 67–77. Springer, 2008.
•  G. Biros and O. Ghattas. Inexactness issues in the Lagrange-Newton-Krylov-Schur method for PDE-constrained optimization. In Large-Scale PDE-Constrained Optimization, pages 93–114. Springer, 2003.
•  J. Borwein and A. Lewis. Convex analysis and nonlinear optimization. CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC, 3. Springer, New York, second edition, 2006. Theory and examples.
•  M. Cheney. The linear sampling method and the MUSIC algorithm. Inverse Problems, 17(4):591–595, 2001.
•  L. Condat. Fast Projection onto the Simplex and the l1 Ball. Technical report, 2014.
•  J. Dennis, M. Heinkenschloss, and L. Vicente. Trust-region interior-point SQP algorithms for a class of nonlinear programming problems. SIAM Journal on Control and Optimization, 36(5):1750–1794, 1998.
•  G. Golub and V. Pereyra. The differentiation of pseudo-inverses and nonlinear least squares which variables separate. SIAM J. Numer. Anal., 10(2):413–432, 1973.
•  G. Golub and V. Pereyra. Separable nonlinear least squares: the variable projection method and its applications. Inverse Problems, 19(2):R1, 2003.
•  E. Haber, U. M. Ascher, and D. Oldenburg. On optimization techniques for solving nonlinear inverse problems. Inverse Problems, 16(5):1263–1280, oct 2000.
•  S. M. Hanasoge. Full waveform inversion of solar interior flows. The Astrophysical Journal, 797(1):23, nov 2014.
•  R. Koenker and G. Bassett Jr.

Regression quantiles.

Econometrica: journal of the Econometric Society, pages 33–50, 1978.
•  A. S. Lewis and M. L. Overton. Nonsmooth optimization via quasi-Newton methods. Mathematical Programming, 141(1-2):135–163, oct 2013.
•  T. T. Y. Lin and F. J. Herrmann. Robust estimation of primaries by sparse inversion via one-norm minimization. Geophysics, 78(3):R133–R150, 2013.
•  M. Osborne. Separable least squares, variable projection, and the Gauss-Newton algorithm. Electronic Transactions on Numerical Analysis, 28(2):1–15, 2007.
•  D. P. O’Leary and B. W. Rust. Variable projection for nonlinear least squares problems. Computational Optimization and Applications, 54(3):579–593, apr 2013.
•  S. U. Park, N. Dobigeon, and A. O. Hero. Semi-blind sparse image reconstruction with application to MRFM. IEEE Transactions on Image Processing, 21(9):3838–3849, 2012.
•  V. Pereyra and G. Scherer, editors. Exponential Data Fitting and its Applications. Bentham Science and Science Publishers, mar 2012.
•  P. Philip.

Optimal Control and Partial Differential Equations.

Technical report, 2009.
•  R.-E. Plessix. A review of the adjoint-state method for computing the gradient of a functional with geophysical applications. Geophysical Journal International, 167(2):495–503, 2006.
•  A. Rakotomamonjy, F. Bach, S. Canu, and Y. Grandvalet. More efficiency in multiple kernel learning. In Proceedings of the 24th international conference on Machine learning, pages 775–782. ACM, 2007.
•  R. Rockafellar and R. Wets. Variational Analysis, volume 317. Springer, 1998.
•  A. A. Royer, M. G. Bostock, and E. Haber. Blind deconvolution of seismograms regularized via minimum support. Inverse Problems, 28(12):125010, dec 2012.
•  A. Ruhe and P. Wedin. Algorithms for separable nonlinear least squares problems. SIAM Rev., 22(3):318–337, 1980.
•  D. Ruppert and R. J. Carroll. Trimmed least squares estimation in the linear model. Journal of the American Statistical Association, 75(372):828–838, 1980.
•  S. Saitoh. Theory of reproducing kernels and its applications, volume 189 of Pitman Research Notes in Mathematics Series. Longman Scientific & Technical, Harlow; copublished in the United States with John Wiley & Sons, Inc., New York, 1988.
•  R. Schmidt. Multiple emitter location and signal parameter estimation. IEEE Transactions on Antennas and Propagation, 34(3):276–280, mar 1986.
•  B. Schölkopf, R. Herbrich, and A. Smola. A generalized representer theorem. In Computational learning theory (Amsterdam, 2001), volume 2111 of Lecture Notes in Comput. Sci., pages 416–426. Springer, Berlin, 2001.
•  B. Schölkopf and A. J. Smola. Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press, 2002.
•  P. Shearer and A. C. Gilbert. A generalization of variable elimination for separable inverse problems beyond least squares. Inverse Problems, 29(4):045003, Apr. 2013.
•  A. J. Smola and B. Schölkopf. A tutorial on support vector regression. Statistics and computing, 14(3):199–222, 2004.
•  T. van Leeuwen and F. J. Herrmann. A penalty method for PDE-constrained optimization in inverse problems. Technical Report 1, jan 2016.
•  W. Wang and M. A. Carreira-Perpinán. Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. arXiv preprint arXiv:1309.1541, 2013.
•  E. Yang and A. C. Lozano. Robust gaussian graphical modeling with the trimmed graphical lasso. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2584–2592. Curran Associates, Inc., 2015.