Discriminative Optimization: Theory and Applications to Computer Vision Problems

07/13/2017 ∙ by Jayakorn Vongkulbhisal, et al. ∙ Carnegie Mellon University 0

Many computer vision problems are formulated as the optimization of a cost function. This approach faces two main challenges: (i) designing a cost function with a local optimum at an acceptable solution, and (ii) developing an efficient numerical method to search for one (or multiple) of these local optima. While designing such functions is feasible in the noiseless case, the stability and location of local optima are mostly unknown under noise, occlusion, or missing data. In practice, this can result in undesirable local optima or not having a local optimum in the expected place. On the other hand, numerical optimization algorithms in high-dimensional spaces are typically local and often rely on expensive first or second order information to guide the search. To overcome these limitations, this paper proposes Discriminative Optimization (DO), a method that learns search directions from data without the need of a cost function. Specifically, DO explicitly learns a sequence of updates in the search space that leads to stationary points that correspond to desired solutions. We provide a formal analysis of DO and illustrate its benefits in the problem of 3D point cloud registration, camera pose estimation, and image denoising. We show that DO performed comparably or outperformed state-of-the-art algorithms in terms of accuracy, robustness to perturbations, and computational efficiency.



There are no comments yet.


page 10

page 13

page 21

page 22

page 23

page 24

page 25

page 26

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Mathematical optimization plays an important role for solving many computer vision problems. For instance, optical flow, camera calibration, homography estimation, and structure from motion are computer vision problems solved as optimization. Formulating computer vision problems as optimization problems faces two main challenges: (i) Designing a cost function that has a local optimum that corresponds to a suitable solution. (ii) Selecting an efficient and accurate algorithm for searching the parameter space. Conventionally, these two steps have been treated independently, leading to different cost functions and search algorithms. However, in the presence of noise, missing data, or inaccuracies of the model, this conventional approach can lead to undesirable local optima or even not having an optimum in the correct solution.

Consider Fig. 1a-top which illustrates a 2D alignment problem in a case of noiseless data. A good cost function for this problem should have a global optimum when the two shapes overlap. Fig. 1b-top illustrates the level sets of the cost function for the Iterative Closest Point (ICP) algorithm [1] in the case of complete and noiseless data. Observe that there is a well-defined optimum and that it coincides with the ground truth. Given a cost function, the next step is to find a suitable algorithm that, given an initial configuration (green square), finds a local optimum. For this particular initialization, the ICP algorithm will converge to the ground truth (red diamond in Fig. 1b-top), and Fig. 1d-top shows the convergence region for ICP in green. However, in realistic scenarios with the presence of perturbations in the data, there is no guarantee that there will be a good local optimum in the expected solution, while the number of local optima can be large. and Fig. 1b-bottom show the level set representation for the ICP cost function in the case of corrupted data. We can see that the shape of cost function has changed dramatically: there are more local optima, and they do not necessarily correspond to the ground truth (red diamond). In this case, the ICP algorithm with an initialization in the green square will converge to a wrong optimum. It is important to observe that the cost function is only designed to have an optimum at the correct solution in the ideal case, but little is known about the behavior of this cost function in the surroundings of the optimum and how it will change with noise.

Figure 1: 2D point alignment using ICP and DO. (a) Data. (b) Level sets of the cost function for ICP. We used the optimal matching at each parameter value to compute the cost. (c) Inferred level sets for the proposed DO. The level sets is approximately reconstructed using the surface reconstruction algorithm [2] from the update directions of DO. (d) The update direction of DO. (e) Regions of convergence for ICP and DO. See text for detailed description (best seen in color).

To address the aforementioned problems, this paper proposes Discriminative Optimization (DO). DO exploits the fact that we often know from the training data where the solutions should be, whereas traditional approaches formulate optimization problems based on an ideal model. Rather than following a descent direction of a cost function, DO directly learns a sequence of update directions leading to a stationary point. These points are placed “by design" in the desired solutions from training data. This approach has three main advantages. First, since DO’s directions are learned from training data, they take into account the perturbations in the neighborhood of the ground truth, resulting in more robustness and a larger convergence region, as can be seen in Fig. 1d. Second, because DO does not optimize any explicit function (e.g., registration error), it is less sensitive to model misfit and more robust to different types of perturbations. Fig. 1c illustrates the contour level inferred from the update directions learned by DO. It can be seen that the curve levels have a local optimum on the ground truth and fewer local optima than ICP in Fig. 1b. Fig. 1e shows that the convergence regions of DO change little despite the perturbations, and always include the regions of ICP. Third, to compute update directions, traditional approaches require the cost function to be differentiable or continuous, whereas DO’s directions can always be computed. We also provide a proof of DO’s convergence in the training set. We named our approach DO to reflect the idea of learning to find a stationary point directly rather than that of optimizing a “generative” cost function.

In this work, we study the properties of DO and its relationship to mathematical programming. Based on this relationship, we propose a framework for designing features where the update directions can be interpreted as the gradient direction of an unknown cost function. We show that our approach can handle both ordered (e.g., image pixels) and unordered (e.g., set of feature matches) data types. We provide a synthetic experiment which confirms our interpretation. We apply DO to the problems of point cloud registration, camera pose estimation, and image denoising, and show that DO can obtain state-of-the-art results.

2 Related Works

2.1 Optimization in computer vision

Many problems in computer vision involve solving inverse problems, that is to estimate a set of parameters that satisfies , where models the phenomena of interest. For example, in camera resection problem [3], can represent the camera parameters and the geometric projective error. Optimization-based framework tackles these problems by searching for the parameters that optimize a certain cost function, which is typically designed to penalize the deviation of from . Since selecting a cost function is a vital step for solving problems, there exists a large amount of research on the robustness properties of different penalty functions [4, 5, 6]. Instead of using a fixed cost function, many approaches use continuation methods to deform the cost function as the optimization is solved [7, 8]. On the other hand, many computer vision problems are ill-posed [9]

, and require some forms of regularization. This further leads to a combination of different penalty functions and requires setting the value of hyperparameters, which make the issue even more complicated.

Due to a large variety of design choices, it is not trivial to identify a suitable cost function for solving a problem. Instead of manually design a cost function, several works proposed to use machine learning techniques to learn a cost function from available training data. For example, kernel SVM 

[10], boosting [11], metric learning [12], and nonlinear regressors [13]

have been used to learn a cost function for image-based tracking, alignment, and pose estimation. Once a cost function is learned, the optimal parameters are solved using search algorithms such as descent methods or particle swarm optimization. However, a downside of these approaches is that they require the form of the cost function to be imposed, e.g., 

[12] requires to cost to be quadratic, thereby restricting the class of problems that they can solve.

2.2 Learning search direction

Instead of using search directions from a cost function, recent works proposed to use learning techniques to directly compute such directions. This is done by learning a sequence of regressors that maps a feature vector to an update vector that points to the desired parameters. We refer to these algorithms as supervised sequential update (SSU). The concept of SSUs is similar to gradient boosting (GB) 

[14, 15], which uses weak learners to iteratively update parameter estimates. However, they differ in that GB performs update using a fixed feature vector, while SSUs also update the feature vector, which allows more information to be incorporated as the parameter is updated. Here, we provide a brief review of SSUs.

Cascade pose regression [16] trains a sequence of random ferns for image-based object pose estimation. The paper also shows that the training error decreases exponentially under weak learner assumptions. [17] learns a sequence of boosted regressors that minimizes error in parameter space. Supervised descent method (SDM) [18, 19] learns a sequence of linear maps as the averaged Jacobian matrices for minimizing nonlinear least-squares functions in the feature space. They also provided conditions for the error to strictly decrease in each iteration. More recent works include learning both Jacobian and Hessian matrices [20]; running Gauss-Newton algorithm after SSU [21]; using different maps in different regions of the parameter space [22]

; and using recurrent neural network as the sequence of maps while also learns the feature 

[23]. While the mentioned works learn the regressors in a sequential manner, Sequence of Learned Linear Predictor [24] first learns a set of linear maps then selects a subset to form a sequence of maps.

We observe that most SSUs focus on image-based tracking and pose estimation. This is because the feature for these problems is rather obvious: they use intensity-based features such as intensity difference, SIFT, HOG, etc. Extending SSUs to other problems require designing new features. Our previous work [25] proposes a new feature and extends SSU to the problem of point cloud registration. Still, it is not straightforward to design features for other applications. In this work, we propose DO as a simple extension of previous SSUs. We study the its properties, propose a general framework for designing features, and apply DO to other computer vision problems, such as point cloud registration, camera pose estimation, and image denoising.

Recently, deep learning has received tremendous interest for its success in various tasks in computer vision and natural language processing 

[26]. Notably, some works use deep learning to solve optimization [27, 28], but they differ from DO since they still need a cost function to be defined. On the other hand, deep learning has been used for some applications similar to those in this paper, e.g., camera pose estimation [29], homography estimation [30], and image denoising [31]. However, DO differs from these works in that DO can combine the mathematical model of each problem with the available training data, while deep learning approaches are purely data-driven. The model-driven side of DO allows it to be interpreted and analyzed more easily, and significantly reduces the amount of data and computational power required for training.

3 Discriminative Optimization (DO)

In this section, we provide the motivation and describe the Discriminative Optimization (DO) framework.

3.1 Motivation from fixed point iteration

DO aims to learn a sequence of update maps (SUM) to update an initial parameter vector to a stationary point. The idea of DO is based on the fixed point iteration of the form


where is the parameter at step , and is the update vector. Eq. (1) is iterated until vanishes, i.e., until a stationary point is reached. An example of fixed point iteration for solving optimization is the gradient descent algorithm [32]. Let be a differentiable cost function. The gradient descent algorithm for minimizing is expressed as


where is a step size. One can see that the scaled gradient is used as in (1), and it is known that the gradient vanishes when a stationary point is reached.

In contrast to gradient descent where the updates are derived from a cost function, DO learns the updates from the training data. The major advantages are that no cost function is explicitly designed and the neighborhoods around the solutions of the perturbed data are taken into account when the maps are learned.

3.2 DO framework

DO uses an update rule in the form of (1). The update vector is computed by mapping the output of a function with a sequence of matrices111

Here, we used linear maps due to their simplicity and computational efficiency. However, other non-linear regression functions can be used in a straightforward manner.

. Here, is a function that encodes a representation of the data (e.g., extracts features from an image at position ). Given an initial parameter , DO iteratively updates , using:


until convergence to a stationary point. The sequence of matrices learned from training data forms a sequence of update maps (SUM).

3.2.1 Learning a SUM

Suppose we are given a training set as a set of triplets , where is the initial parameter for the problem instance (e.g., the image), is the ground truth parameter (e.g., position of the object on the image), and extract features from the problem instance. The goal of DO is to learn a sequence of update maps that updates to . To learn the maps, we minimize the least-square error:


where is the norm. After we learn a map , we update each using (3), then proceed to learn the next map. This process is repeated until some terminating conditions, such as until the error does not decrease much or a maximum number of iterations is reached. To see why (4) learns stationary points, we can see that for with , (4) will force to be close to zero, thereby inducing a stationary point around

. In practice, we use ridge regression to learn the maps to prevent overfitting:


where is the Frobenius norm, and is a hyperparameter. The pseudocode for training a SUM is shown in Alg. 1.

3.2.2 Solving a new problem instance

To solve a new problem instance with an unseen function and an initialization , we update with the obtained SUM using (3) until a stationary point is reached. However, in practice, the number of maps is finite, say maps. We observed in many cases that the update at the iteration is still large, which means the stationary point is still not reached, and that is far from the true solution. For example, in the registration task, the rotation between initial orientation and the solution might be so large that we cannot obtain the solution within a fixed number of update iterations. To overcome this problem, we keep updating using the map until the update is small or the maximum number of iterations is reached. This approach makes DO different from previous works in Sec. 2.2, where the updates are only performed up to the number of maps. Alg. 2 shows the pseudocode for updating the parameters.

1:  for  to  do
2:     Compute with (5).
3:     for  to  do
4:        Update .
5:     end for
6:  end for
Algorithm 1 Training a sequence of update maps (SUM)
1:  Set
2:  for  to  do
3:     Update
4:  end for
5:  Set .
6:  while  and  do
7:     Update
8:     Update
9:  end while
Algorithm 2 Searching for a stationary point

4 Theoretical Analysis of DO

In this section, we analyze the theoretical properties of DO. Specifically, we discuss the conditions for the convergence of the training error, and the relation between DO and mathematical optimization.

4.1 Convergence of training error

Here, we show that under a weak assumption on , we can learn a SUM that updates to , i.e., the training error converges to zero. First, we define the monotonicity at a point condition:

Definition 1.

(Monotonicity at a point) A function is

(i) monotone at if


for all ,

(ii) strictly monotone at if


for all and the equality holds only at ,

(iii) strongly monotone at if


for some and all .

It can be seen that if is strongly monotone at then is strictly monotone at , and if is strictly monotone at then is monotone at . With the above definition, we obtain the following result:

Theorem 1.

(Convergence of SUM’s training error) Given a training set , if there exists a linear map where is strictly monotone at for all , and if there exists an where , then the update rule:


with obtained from (4), guarantees that the training error strictly decreases in each iteration:


Moreover, if is strongly monotone at , and if there exist such that for all , then the training error converges to zero. If then the error converges to zero linearly.

The proof of Thm. 1 is provided in the appendix. In words, Thm. 1 says that if each instance is similar in the sense that each is strictly monotone at , then sequentially learning the optimal maps with (4) guarantees that the training error strictly reduces in each iteration. If is strongly monotone at and upperbounded then the error converges to zero. Note that is not required to be differentiable or continuous. Xiong and De la Torre [19] also presents a convergence result for a similar update rule, but it shows the strict reduction of error of a single function under a single ideal map. It also requires an additional condition called ‘Lipschitz at a point,’ This condition is necessary for bounding the norm of the map, otherwise the update can be too large, preventing the reduction in error. In contrast, Thm. 1 explains the convergence of multiple functions under the same SUM learned from the data, where the each learned map can be different from the ideal map . To ensure reduction of error, Thm. 1 also does not require the ‘Lipschitz at a point’ the norms of the maps are adjusted based on the training data. Meanwhile, to ensure convergence to zero, Thm. 1 requires an upperbound which can be thought of as a relaxed version of ‘Lipschitz at a point’ (note that does not need to be ). These weaker assumptions have an important implication as it allows robust discontinuous features, such as HOG in [19], to be used as . Finally, we wish to point out that Thm. 1 guarantees the reduction in the average error, not the error of each instance .

4.2 Relation to mathematical programming

In this section, we explore the relation between DO and mathematical programming. Specifically, we show that monotonicity-at-a-point is a generalization of monotonicity and pseudomonotonocity, which are the properties of the gradient of convex and pseudoconvex functions [33, 34]. Understanding this relation leads to a framework for designing in Sec. 5. We begin this section by providing definitions and propositions relating generalized convexity and monotonicity, then we provide our result in the end.

A pseudoconvex function is defined as follows.

Definition 2.

(Pseudoconvexity [34]) A differentiable function is

(i) pseudoconvex if for any distinct points ,


(ii) strictly pseudoconvex if for any distinct points ,


(iii) strongly pseudoconvex if there exists such that for any distinct points ,


Fig. 2d shows examples of pseudoconvex functions. In essence, pseudoconvex functions are differentiable functions where the sublevel sets are convex and all stationary points are global minima. Pseudoconvex functions generalize convex functions: all differentiable convex functions are pseudoconvex. Pseudoconvex functions are used as penalty functions for their stronger robustness than convex ones [6, 8, 35]. Next, we introduce pseudomonotonicity.

Definition 3.

(Pseudomonotonicity [34]) A function is

(i) pseudomonotone if for any distinct points ,


(ii) strictly pseudomonotone if for any distinct points ,


(iii) strongly pseudomonotone if there exists such that for any distinct points ,


It can also be shown that monotone (resp., strictly, strongly) functions are pseudomonotone (resp., strictly, strongly) [34]. The following propositions provides a relation between the gradients of pseudoconvex functions and pseudomonotonicity.

Proposition 1.

(Convexity and monotonicity [34]) A differentiable function is pseudoconvex (resp., strictly, strongly) if and only if its gradient is pseudomonotone (resp., strictly, strongly).

Next, we provide our result on the relation between monotonicity-at-a-point and pseudomonotonicity.

Proposition 2.

(Pseudomonotonicity and monotonicity at a point) If a function is pseudomonotone (resp., strictly, strongly) and , then is monotone (resp., strictly, strongly) at .

The converse of the proposition is not true. For example, is strictly monotone at , but not strictly pseudomonotone (counterexample at and ). Prop. 2 shows that monotonicity-at-a-point is a generalization of pseudomonotonicity, implying that the conditions in Thm. 1 are weaker than the conditions for the gradient maps of pseudoconvex and convex functions.

5 Designing

The function which provides information about each problem instance is crucial for solving a problem. In this section, we describe a framework to design for solving a class of problem based on our analysis in Sec. 4. We are motivated by the observation that many problems in computer vision aim to find such that , where models the problem of interest (see Sec. 2.1). To solve such problem, one may formulate an optimization problem of the form


where is a penalty function, e.g., sum of squares, norm, etc. If is differentiable, then we can use gradient descent to find a minimum and returns it as the solution. The choice of has a strong impact on the solution in terms of robustness to different perturbations, and it is not straightforward to select that will account for perturbations in real data. The following framework is based on the concept of using training data to learn the update directions that mimic gradient descent of an unknown , thereby bypassing the manual selection of .

5.1 from the gradient of an unknown penalty function

For simplicity, we assume is differentiable, but the following approach also applies when it does not. Let us observe its derivative:


where we express as to reduce notation clutter. We can see that the form of affects only the last term in the RHS of (18), while the Jacobian does not depend on it. Since different ’s are robust to different perturbations, this last term determines the robustness of the solution. Here, we will use DO to learn this term from a set of training data.

In order to do so, we need to express (18) as . First, we rewrite (18) as the update vector , where we replace the derivative of with a generic function :


where is row of , and is element of . We then rewrite (20) as the following convolution:


where is the Dirac delta function. It can be seen that (21) is equivalent to (18), while being linear in . This allows us to learn using linear least squares. To do so, we will express (21) in the form of . For simplicity, we will look at the element of :


Eq. 24 expresses as an inner product between and over and , where


The following results discusses the convergence of training data when in (26) is used.

Proposition 3.

(Convergence of the training error with an unknown penalty function) Given a training set , where and differentiable, if there exists a function such that for each , is differentiable strictly pseudoconvex with the minimum at , then the training error of DO with from (26) strictly decreases in each iteration. Alternatively, if is differentiable strongly pseudoconvex with Lipschitz continuous gradient, then the training error of DO converges to zero.

Under similar conditions, we can also show the same convergence results for that is nondifferentiable strictly and strongly convex functions. Roughly speaking, Prop. 3 says that if there exists a penalty function such that for each the global minimum of (17) is at with no other local minima, then using (26) allows us to learn for DO. Note that we do not need to explicitly know what such penalty function is. Thus, we can say that using (26) is equivalent to learning a surrogate of the gradient of an unknown cost function. This illustrates the potential of DO as a tool for solving a broad class of problems where the penalty function is unknown.

5.2 Computing

Eq. (26) expresses as a function. To compute in practice, we need to express it as a vector. To do so, we will convert in (26) into a discrete grid, then vectorize it. Specifically, we first discretize into a -dimensional grid with bins in each dimension, where a bin evaluates to if is discretized to that bin, and for all other bins. Let us denote this grid as , and let be a function where returns the index that discretizes to. We can express the vectorized as the following Kronecker product of standard bases:


With this discretization, we can express in (26) in a discrete form as


By concatenating over and , we obtain the final form of as


where denotes vector concatenation. The dimension of is . We show how to apply (29) to applications in Sec. 6. Note that the above approach is one way of designing to use with SUM. It is possible to use different form of (e.g., see Sec. 6.2), or replace with a nonlinear map.

6 Experiments

In this section, we first provide an intuition into DO with an analytical example, then we apply DO to three computer vision problems: 3D point cloud registration, camera pose estimation, and image denoising. All experiments were performed in MATLAB on a single thread on an Intel i7-4790 3.60GHz computer with 16GB memory.

6.1 Optimization with unknown 1D cost functions

In this experiment, we demonstrate DO’s potential in solving 1D problems without an explicit cost function. Specifically, given a set of number , we are interested in finding the solution of the problem


A typical approach to solve this problem is to solve the optimization


for some function . The form of depends on the assumption on the distribution of , e.g., the maximum likelihood estimation for i.i.d. Gaussian would use . If the an explicit form of is known, then one can compute in closed form (e.g., is squared value or absolute value) or with an iterative algorithm. However, using a that mismatches with the underlying distribution of could lead to an optimal, but incorrect, solution . Here, we will use DO to solve for from a set of training data.

For this problem, we defined 6 ’s as follows:


The first 3 ’s are convex, where is a nonsmooth function; is a combination of different powers; is an asymmetric function (i.e., ). The latter 3 ’s are pseudoconvex, where has exponents smaller than ; while and are inverted Gaussian function with different widths. Pseudoconvex functions are typically used as robust penalty functions[35]

because they penalize outliers less than convex functions. Recall that sum of pseudoconvex functions may not be pseudoconvex, and can have multiple local minima. The graphs of the functions and the gradient

222Here, we abuse the word gradient to include subdifferential for nonsmooth convex functions and generalized subdifferential for nonconvex functions [36]. are shown in Fig. 2a,b,d,e. We call the problem in (31) that uses as .

Figure 2: Learning to solve unknown cost functions. (a-c) show three convex functions, their gradients, and the learned for each function. (d-f) show similar figures for pseudoconvex functions. (g) shows training error in each step . (h) shows the squared norm of the maps . (i) shows the first map of each function.

We generate the training data for as where ; is the initial estimate; and is the global minimizer of with the data . To find the minimizers, we use fminunc for convex functions, and grid search with the step size of for nonconvex functions. We trained the SUMs using the in Sec. 5, which in this case is simply


We use as the range of , and discretize it into bins. Let us denote the maps that learn from as . To illustrate the training error, we train up to 15 maps for each , but for test we set the number of maps to the last map that reduce the training RMSE more than . During test, we set and .

Fig. 2c,f show the scaled maps for each . We can see that the maps resemble the gradient of their respective functions, suggesting that DO can learn the gradients from training data without explicit access to the cost functions. The reason that learns the gradient is because stationary points need to satisfy . It should be noted that the first maps for all in Fig. 2i are different from their maps. This is because the first maps try to move as close to as possible and thus disregard the placement of the stationary point. The training errors in Fig. 2g show that convex functions are easier to learn than nonconvex ones. This is because nonconvex functions may have multiple local minima, which means there may not exist an ideal map where all training data are monotone at their solutions, thus may get stuck at a wrong stationary point. Fig. 2h shows that the map have decreasing norms, which represents reducing step sizes as the estimates approach the solutions.

We also perform an experiment on unseen sets of data, where we compare the global minimizer of of each test data against the solution from fminunc (quasi-Newton) of all and the solution of . Table I show the MAE over 1000 test sets. We can see that DO can approximate the solution better than using incorrect cost functions. An interesting point to note is that DO seems to be able to solve nonconvex problems better than fminunc, suggesting DO can avoid some local minima and more often terminate closer to the global minimum.

We summarize this section in 4 points. (i) We show that DO can learn to mimic gradient of unknown penalty functions. (ii) A very important point to note is that a single training data can have multiple ground truths, and DO will learn to find the solution based on the ground truths provided during the training. Thus, it is unreasonable to use DO that, say, trained with the mean as ground truth and hope to get the median as a result. (iii) A practical implication of this demonstration is that if we optimize a wrong cost function then we may obtain a bad optimum as solution, and it can be more beneficial to obtain training data and learn to solve for the solution directly. (iv) We show that for nonconvex problems, DO has the potential to skip local minima and arrive at a better solution than that of fminunc.

.0000 .0675 .1535 .0419 .0707 .2044 .0137
.0675 .0000 .1445 .1080 .1078 .2628 .0145
.1535 .1445 .0000 .1743 .1657 .2900 .0086
.0493 .1009 .1682 .0457 .0929 .1977 .0325
.0707 .1078 .1657 .0823 .0000 .1736 .0117
.2098 .2515 .2791 .1905 .2022 .1161 .0698
Table I: MAE for solving unknown cost functions. Best results in underline bold, and second best in bold.

6.2 3D point cloud registration

In this section, we perform experiments on the task of 3D point cloud registration. The problem can be stated as follows: Let be a matrix containing 3D coordinates of one shape (‘model’) and for the second shape (‘scene’), find the rotation and translation that registers to . Here, we briefly describe our parametrization and experiments. For more details, please see [25].

6.2.1 DO parametrization and training

We use Lie Algebra [37] to parametrize

, which represents rotation and translation, because it provides a linear space with the same dimensions as the degrees of freedom of our parameters. For

, we design it as a histogram that indicates the weights of scene points on the ‘front’ and the ‘back’ sides of each model point based on its normal vector. Let be a normal vector of the model point computed from its neighbors; be a function that applies rigid transformation with parameter to vector ; be the set of scene points on the ‘front’ of ; and contains the remaining scene points. We define as:


where normalizes to sum to , and controls the width of the function. can be precomputed (see [25]).

Given a model shape , we first normalized the data to lie in , and generated the scene models as training data by uniformly sampling with replacement 400 to 700 points from . Then, we applied the following perturbations: (i) Rotation and translation: We randomly rotated the model within 85 degrees, and added a random translation in . These transformations were used as the ground truth , with as the initialization. (ii) Noise and outliers:

Gaussian noise with standard deviation

was added to the sample. Then we added two types of outliers: sparse outliers (random 0 to 300 points within ); and structured outliers (a Gaussian ball of 0 to 200 points with the standard deviation of 0.1 to 0.25). Structured outliers is used to mimic other dense object in the scene. (iii) Incomplete shape: We used this perturbation to simulate self occlusion and occlusion by other objects. This was done by uniformly sampling a 3D unit vector , then projecting all sample points to , and removed the points with the top 40% to 80% of the projected values. For all experiments, we generated 30000 training samples, and trained a total of maps for SUM with in (5) and in (39) and (40), and set the maximum number of iterations to 1000.

6.2.2 Baselines and evaluation metrics

We compared DO with two point-based approaches (ICP [1] and IRLS [5]) and two density-based approaches (CPD [38] and GMMReg [39]). The codes for all methods were downloaded from the authors’ websites, except for ICP where we used MATLAB’s implementation. For IRLS, the Huber cost function was used.

We used the registration success rate and the computation time as performance metrics. We considered a registration to be successful when the mean error between the registered model points and the corresponding model points at the ground truth orientation was less than of the model’s largest dimension.

6.2.3 Synthetic data

We performed synthetic experiments using the Stanford Bunny model [40] (see Fig. 3). We used MATLAB’s pcdownsample to select 472 points from 36k points as the model . We evaluated the performance of the algorithms by varying five types of perturbations: (i) the number of scene points ranges from 100~4000 [default = 200~600]; (ii) the standard deviation of the noise ranges between 0~0.1 [default = 0]; (iii) the initial angle from 0 to 180 degrees [default = 0~60]; (iv) the number of outliers from 0~600 [default = 0]; and (v) the ratio of incomplete scene shape from 0~0.7 [default = 0]. While we perturbed one variable, the values of the other variables were set to the default values. Note that the scene points were sampled from the original 36k points, not from . All generated scenes included random translation within . A total of 50 rounds were run for each variable setting. Training time for DO was 236 seconds (incl. training data generation and precomputing features).

Examples of test data and the results are shown in Fig. 3. ICP required low computation time for all cases, but it had low success rates because it tends to get trapped in the local minimum closest to its initialization. CPD generally performed well except when number of outliers was high, and it required a high computation time. IRLS was faster than CPD, but it did not perform well with incomplete targets. GMMReg had the widest basin of convergence but did not perform well with incomplete targets, and it required long computation time for the annealing steps. For DO, its computation time was much lower than those of the baselines. Notice that DO required higher computation time for larger initial angles since more iterations were required to reach a stationary point. In terms of the success rate, we can see that DO outperformed the baselines in almost all test scenarios. This result was achievable because DO does not rely on any specific cost functions, which generally are modelled to handle a few types of perturbations. On the other hand, DO learns to cope with the perturbations from training data, allowing it to be significantly more robust than other approaches.

Figure 3: Results of 3D registration with synthetic data under different perturbations. (Left) Examples of scene points with different perturbations. (Middle) Success rate. (Right) Computation time.
Figure 4: Results of 3D registration with range scan data. (a) example 3D model (‘chef’). (b) Example of a 3D scene. We include surface rendering for visualization purpose. (c) Results of the experiment. (d) shows an example of registration steps of DO. The model was initialized 60 degrees from the ground truth orientation with parts of the model intersecting other objects. In addition, the target object is under 70% occlusion, making this a very challenging case. However, as iteration progresses, DO is able to successfully register the model.

6.2.4 Range-scan data

In this section, we performed 3D registration experiment on the UWA dataset [41]. This dataset contains 50 cluttered scenes with 5 objects taken with the Minolta Vivid 910 scanner in various configurations. All objects are heavily occluded (60% to 90%). We used this dataset to test our algorithm under unseen test samples and structured outliers, as opposed to sparse outliers in the previous section. The dataset includes 188 ground truth poses for four objects. We performed the test using all the four objects on all 50 scenes. From the original model, 300 points were sampled by pcdownsample to use as (Fig. 4a). We also downsampled each scene to 1000 points (Fig. 4b). We initialized the model from 0 to 75 degrees from the ground truth orientation with random translation within . We ran 50 initializations for each parameter setting, resulting in a total of rounds for each data point. Here, we set the inlier ratio of ICP to as an estimate for self-occlusion. Average training time for DO was 260 seconds for each object model.

The results and examples for the registration with DO are shown in Fig. 4c and Fig. 4d, respectively. IRLS, CPR, and GMMReg has very low success in almost every scene. This was because structured outliers caused many regions to have high density, creating false optima for CPD and GMMReg which are density-based approaches, and also for IRLS which is less sensitive to local minima than ICP. When initialized close to the solution, ICP could register fast and provided some correct results because it typically terminated at the nearest–and correct–local minimum. On the other hand, DO provided a significant improvement over ICP, while maintaining low computation time. We emphasize that DO was trained with synthetic examples of a single object and it had never seen other objects from the scenes. This experiment shows that we can train DO with synthetic data, and apply it to register objects in real challenging scenes.

Figure 5: Result for object tracking in 3D point cloud. (a) shows the 3D models of the kettle and the hat. (b) shows tracking results of DO and ICP in (top) 3D point clouds with the scene points in blue, and (bottom) as reprojection on RGB image. Each column shows the same frame.

6.2.5 Application to 3D object tracking

In this section, we explore the use of DO for 3D object tracking in 3D point clouds. We used Microsoft Kinect to capture RGBD videos at 20fps, then reconstruct 3D scenes from the depth images. We used two reconstructed shapes, a kettle and a hat, as the target objects. These two shapes present several challenges besides self occlusion : the kettle has a smooth surface with few features, while the hat is flat, making it hard to capture from some views. We recorded the objects moving through different orientations, occlusions, etc. The depth images were subsampled to reduce computation load. To perform tracking, we manually initialized the first frame, while subsequent frames were initialized using the pose in the previous frames. Here, we only compared DO against ICP because IRLS gave similar results to those of ICP but could not track rotation well, while CPD and GMMReg failed to handle structured outliers in the scene (similar to Sec. 6.2.4). Fig. 5b shows examples of the results. It can be seen that DO can robustly track and estimate the pose of the objects accurately even under heavy occlusion and structured outliers, while ICP tended to get stuck with other objects. The average computation time for DO was 40ms per frame. This shows that DO can be used as a robust real-time object tracker in 3D point cloud.

Failure case: We found DO failed to track the target object when the object was occluded at an extremely high rate, and when the object moved too fast. When this happened, DO would either track another nearby object or simply stay at the same position as in the previous frame.

6.3 Camera Pose Estimation

The goal of camera pose estimation is to estimate the relative pose between a given 3D and 2D correspondence set. Given where is 2D image coordinate and is the corresponding 3D coordinate of feature , we are interested in estimating the rotation matrix and translation vector , such that

where tilde denotes homogeneous coordinate, is a known intrinsic matrix, and denotes equivalence up to scale. General approaches for camera pose estimation involve solving nonlinear problems [42, 43, 44, 45]. Most of existing approaches assume that there are no outlier matches in the correspondence set. When outliers are present, they rely on RANSAC [46] to select the inliers. One approach that does not rely on RANSAC is REPPnP [47]. It finds the camera pose by solving for the robust nullspace of a matrix which represents algebraic projection error. In this section, we will use DO to find a set of inliers, then postprocess the inliers to obtain the camera pose. We show that our algorithm is more robust than REPPnP while being faster than RANSAC-based approaches.

Figure 6: Results for PnP with synthetic data. Varying parameters are (a) outlier ratio, (b) noise SD, and (c) number of points.
Figure 7: Results for camera pose estimation on real image. (a) Feature matches. Left and right images contain 2D points and projection of 3D points, resp. (b-d) Projected shape with average time over 100 trials.

6.3.1 DO parametrization and training

To obtain the set of inliers, we solve for a matrix such that the geometric error [3] is zero (assuming is calibrated):


The optimization for solving is formulated by summing the error over the correspondences:


where is a penalty function. Following the derivation in Sec. 5, the function can be derived as:


After computing 43, we normalize it to a unit vector and use it our feature. Note that, although the Jacobian matrix of is a matrix, it has only 12 degrees of freedom. Thus, we need to consider only its 12 values instead of all 24.

We generated DO’s training data as follows. Each image was assumed to be 640 by 480 pixels. Generating 3D shapes: A 3D shape, composing of 100 to 500 points, was generated as random points in one of the following shapes: (i) in a box; (ii) on a spherical surface; and (iii) on multiple planes. For (iii), we randomly generated normal and shift vectors for 2 to 4 planes, then added points to them. All shapes were randomly rotated, then normalized to fit in . Generating camera matrix: We randomized the focal length in with the principal point at the center of the image. We sampled the rotation matrix from , while the translation was generated such that the projected 3D points lie in the image boundary. Generating image points: We first projected the 3D shape using the generated camera parameters, then randomly selected 0% to 80% of the image points as outliers by changing their coordinates to random locations. All random numbers were uniformly sampled. No noise is added for the training samples. To reduce the effect of varying sizes of images and 3D points, we normalized the inputs to lie in .333Camera matrix needs to be transformed accordingly, similar to [48]. Since the camera matrix is homogeneous, we normalize it to have a unit Frobenius norm. We use as the range for each dimension of , and discretize it to 10 bins. We generated 50000 training samples, and trained 30 maps with . The training time was 252 seconds.

We compared 3 DO-based approaches: DO, DO+P3P+RANSAC, and DO+RPnP. When DO returned as result, we transformed it back to a matrix , then projected the first three columns to obtain the rotation matrix. For DO+P3P+RANSAC and DO+RPnP, we used from DO to select matches with small projection errord as inliers, then calculated the camera parameters using P3P+RANSAC [44] and RPnP [45] (without RANSAC).

6.3.2 Baselines and evaluation metrics

We compared our approach against 5 baselines. EPnP [42] and REPPnP [47] are deterministic approaches. The other three baselines, P3P+RANSAC [44], RPnP+RANSAC [45], and EPnP+RANSAC [42] rely on RANSAC to select inliers and use the respective PnP algorithms to find the camera parameters. The minimum number of matches for each algorithm is 3, 4, and 6, resp. We use the code from [47] as implementation of the PnP algorithms. The RANSAC routine automatically determines the number of iterations to guarantee 99% chance of obtaining the inlier set. The performance are measured in terms of (i) mean computation time, (ii) mean rotation angle error, and (iii) mean inlier reprojection error.

6.3.3 Experiments and results

We first performed experiments using synthetic data. We generated the test data using the same approach as training samples. We vary 3 parameters: (i) the number of points from 200~2000 [default = 400]; (ii) the ratio of outliers from 0%~90% [default = 30%]; and (iii) the noise standard deviation from 0~10 pixels [default = 2]. When one parameter is varied, the other two parameters were set to the default values. We performed a total of 500 trials for each setting.

Fig. 6 shows the results of the experiments. In Fig. 6a, we can see that RANSAC-based approaches could obtain accurate results, but their computation time grows exponentially with the outlier ratio. On the other hand, EPnP and REPPnP which are deterministic performed very fast, but they are not robust against outliers even at 10%. For our approaches, it can be seen that DO alone did not obtain good rotations since it did not enforce any geometric constraints. However, DO could accurately align the 3D inlier points to their image points as can be seen by its low inlier reprojection errors. This is a good indication that DO can be used for identifying inliers. By using this strategy, DO+P3P+RANSAC could obtain accurate rotation up to 80% of outliers while maintaining low computation time. In contrast, DO+RPnP could obtain very accurate rotation when there were small outliers, but the error increases as it was easier to mistakenly include outliers in the post-DO step. For the noise case (Fig. 6b), DO+RPnP has constant time for all noise levels and could comparatively obtain good rotations under all noise levels, while DO+P3P+RANSAC required exponentially increasing time as points with very high noise may be considered as outliers. Finally, in Fig. 6c, we can see that computation times of all approaches grow linearly with the number of points, but those of DO approaches grow with faster rate, which is a downside of our approach.

Next, we performed experiments on real images. We used an image provided with the code in [43]. Fig. 7a shows the input matches. Notice that the matches are not one-to-one. Although DO is a deterministic algorithm, different configurations of the same 3D shape can affect the result. For example, we might consider either a 3D shape or its 90 rotated shape as the initial configuration with the identity transformation. To measure this effect, we performed 100 trials for DO-based algorithms, where we randomly rotate the 3D shape as the initial configuration. Similarly, we performed 100 trials for P3P+RANSAC.  7b-e show the results. It can be seen that DO can gives a rough estimate of the camera pose, then DO+P3P+RANSAC and DO+RPnP can postprocess to obtain accurate pose. P3P+RANSAC also obtained the correct pose, but it required 8 times the computation time of DO-based approaches. (More results provided in the appendex.)

Since DO is a learning-based approach, the main limitation of DO is that it may not work well with data that are not represented in training, e.g., when the depths and perturbations of training data and test data are different. It is not simple to generate training data to cover all possible cases. On the other hand, PnP solvers of RANSAC-based approaches can reliably obtain the correct pose since they directly solve the geometric problem.

6.4 Image Denoising

In this final experiment, we demonstrate the potential of DO for image denoising. This experiments serves to illustrate the potential of DO in multiple ways. First, we illustrate that an SUM trained in a simple fashion can compare favorably against state-of-the-art total variation (TV) denoising algorithms for impulse noises. Second, we show that a SUM can be used to estimate a large and variable number of parameters (number of pixels in this case). This differs from previous experiments that used DO to estimate a small, fixed number of parameters. Third, we show that it is simple for DO to incorporate additional information, such as intensity mask, during both training and testing. Finally, we demonstrate the effect of training data on the results.

6.4.1 DO parametrization and training

We based our design of on the TV denoising model [49], where we replace the penalty functions on both the data fidelity term and the regularization term with unknown functions and :


where is the image support, is the intensity at pixel of the noisy input image, is a given mask, and is the set of neighboring pixels of . The goal is to estimate the clean image .

In order to allow the learned SUM to work with images of different size, we will treat each pixel independently: Each pixel will have its own estimate .444The idea is similar to parameter sharing in deep neural network. Since we have two error terms, we follow Sec. 5 and concatenate the indicator of the two errors to form as


The first part of accounts for the data fidelity term, while the second part accounts for the regularization term.

In order to train DO, we randomly sample patches of size to from the training image, then randomly replace to of the pixels with impulse noise to create noisy images. We trained 3 SUMs: (i) DO-SP, where we used salt-pepper (SP) impulse noise in ; (ii) DO-RV, where we used random-value (RV) impulse noise in ; and (iii) DO-SPRV, where 50% of the images has RV noise, while the rest have SP noise. This is to study the effect of training data on learned SUMs. Following [50], for images with SP noise, we set the mask for pixels with intensity and and for others. For images with RV noise, we set for all pixels as we cannot determine whether a pixel is an impulse noise or not. The intensity of each pixel in the noisy image is treated as initial estimate , and is its noise-free counterpart. We use as the ranges for both and , and discretize them to 100 bins. We train a total of 30 maps for DO with . The training time took on average 367 seconds. During test, we use as the stopping criteria.

Figure 8: Results for image denoising for (a) salt-pepper impulse noise, and (b) random-value impulse noise.

6.4.2 Baseline and evaluation metrics

We compared our approach with two total variation (TV) denoising algorithms which are suitable for impulse noise. The first baseline is the convex  [51], which uses for the data fidelity term and isotropic TV as the regularization term. The optimization is solved by the ADMM algorithm. The second baseline is  [50], which uses the nonconvex for the data term and isotropic TV for the regularization term. The optimization is solved by the Proximal ADMM algorithm. The codes of both algorithms are provided in the toolbox of [50]. We used the same mask as in the DO algorithms. We compare the results in terms of Peak Signal-to-Noise Ratio (PSNR).

Figure 9: Examples of images denoising results for (top) salt-pepper impulse noise, and (bottom) random-value impulse noise. The PSNR for each image is shown on the top-right. (Best viewed electronically).

6.4.3 Experiments and results

We downloaded 96 grayscale images of size pixels from the Image Database555http://decsai.ugr.es/cvg/dbimagenes/g512.php of University of Granada’s Computer Vision Group. The first 30 images were used for training DO and selecting the best hyperparameters for the baselines and noise types, while the remaining 66 images were used for evaluation. For each image, we add impulse noise of to to measure the algorithm robustness.

Fig. 8 show the result PNSR over different noise ratios. It can be seen that DO trained with the correct noise type can match or outperform state-of-the-art algorithms, while using a wrong DO give a very bad result. Interestingly, DO-SPRV which was trained with both noise performed well for both cases. Fig. 9 shows examples of denoising results of each algorithm ( omitted for clarity of other approaches). For SP noise, , DO-SP, and DO-SPRV can recover small details, while oversmoothed the image and DO-SP returned an image with smudges. For RV noise, DO-RV returned the best result. DO-SPRV also returned an acceptable image but still contain intensity clumps, while DO-SP cannot recover the image at all. On the other hand, both baselines oversmoothed the image (notice the persons’ heads) and still have intensity clumps over the images. This experiment shows that DO can robustly handle different types of impulse noises, and that the training data have a strong effect on types and amount of noise that it can handle. The best approach for solving the problem is to select the correctly trained model. Still, training DO with both noise can return a good result, illustrating the potential of DO in solving a hard problem.

7 Conclusion and Discussions

We presented Discriminative Optimization (DO) as a new framework for solving computer vision problem. DO learns a sequence of update maps that update a set of parameters to a stationary point from training data. We study the relationship between DO and optimization-based and learning-based algorithms. A theoretical result on the convergence of the training error is given. We also proposed a framework for designing features for DO based on gradient descent, and provide a theoretical result that DO can learn to mimic gradient descent to solve some optimization problems with unknown cost functions. The result is supported by synthetic experiment that the maps learn the approximation of the gradients. In terms of applications, we show that DO can provide favorable results against state-of-the-art algorithms in the problems of 3D point cloud registration and tracking, camera pose estimation, and image denoising. This also shows that DO can deal with both ordered and unordered data.

Although algorithms similar to DO have been proposed previously, this paper opens the connection between DO and optimization. Future work may import ideas and intuition from optimization to DO, such as the incorporation of constraints and momentum methods. Also, our theoretical result provides only sufficient conditions, and monotonicity-at-a-point conditions suggest relationship to the variation inequality problems [52], which can be considered as a generalization of optimization and which might explain the robustness of DO. We also observe that DO’s update rule can be compared with the layer in deep residual network [53], since they both iteratively perform update of the form for some transformation . This can provide some connection to some deep learning algorithms. Future research may also address convergence in the test data or other approaches for designing feature function . With a strong theoretical foundation and practical potential, we believe DO opens a new exciting research area which would have strong impact to computer vision.


  • [1] P. J. Besl and H. D. McKay, “A method for registration of 3-D shapes,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, pp. 239–256, 1992.
  • [2] M. Harker and P. O’Leary, “Least squares surface reconstruction from measured gradient fields,” in

    Proc. IEEE Conf. Computer Vision and Pattern Recognition

    , 2008.
  • [3] R. Hartley and A. Zisserman, Multiple view geometry in computer vision.   Cambridge university press, 2003.
  • [4] R. Basri, L. Costa, D. Geiger, and D. Jacobs, “Determining the similarity of deformable shapes,” Vision Research, vol. 38, no. 15-16, pp. 2365–2385, 1998.
  • [5] P. Bergström and O. Edlund, “Robust registration of point sets using iteratively reweighted least squares,” Computational Optimization and Applicat., vol. 58, no. 3, pp. 543–561, 2014.
  • [6] J. T. Barron, “A more general robust loss function,” arXiv preprint arXiv:1701.03077, 2017.
  • [7] H. Mobahi and J. W. Fisher, “Coarse-to-fine minimization of some common nonconvexities,” in Proc. Int. Conf. Energy Minimization Methods in Computer Vision and Pattern Recognition, 2015.
  • [8] M. Black and A. Rangarajan, “On the unification of line processes, outlier rejection, and robust statistics with applications in early vision,” Int. J. Computer Vision, vol. 19, no. 1, pp. 57–91, 1996.
  • [9] M. Bertero, T. A. Poggio, and V. Torre, “Ill-posed problems in early vision,” Proceedings of the IEEE, vol. 76, no. 8, pp. 869–889, Aug 1988.
  • [10] S. Avidan, “Support vector tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 8, pp. 1064–1072, 2004.
  • [11] X. Liu, “Discriminative face alignment,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 11, pp. 1941–1954, 2009.
  • [12] M. H. Nguyen and F. De la Torre, “Metric learning for image alignment,” Int. J. Computer Vision, vol. 88, no. 1, pp. 69–84, 2010.
  • [13] K. Paliouras and A. A. Argyros, “Towards the automatic definition of the objective function for model-based 3d hand tracking,” in Man–Machine Interactions 4.   Springer, 2016, pp. 353–363.
  • [14] J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” Annals of Statistics, vol. 29, no. 5, pp. 1189–1232, 2001.
  • [15] L. Mason, J. Baxter, P. Bartlett, and M. Frean, “Boosting algorithms as gradient descent,” in Proc. Neural Information Processing Systems, 1999.
  • [16] P. Dollár, P. Welinder, and P. Perona, “Cascaded pose regression,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
  • [17] X. Cao, Y. Wei, F. Wen, and J. Sun, “Face alignment by explicit shape regression,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2012.
  • [18] X. Xiong and F. De la Torre, “Supervised descent method and its application to face alignment,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2013.
  • [19] ——, “Supervised descent method for solving nonlinear least squares problems in computer vision,” arXiv preprint arXiv:1405.0601, 2014.
  • [20] G. Tzimiropoulos, “Project-out cascaded regression with an application to face alignment,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015.
  • [21] E. Antonakos, P. Snape, G. Trigeorgis, and S. Zafeiriou, “Adaptive cascaded regression,” in Proc. IEEE Int. Conf. Image Processing, 2016.
  • [22] X. Xiong and F. De la Torre, “Global supervised descent method,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015.
  • [23] G. Trigeorgis, P. Snape, M. A. Nicolaou, E. Antonakos, and S. Zafeiriou, “Mnemonic descent method: A recurrent process applied for end-to-end face alignment,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016, pp. 4177–4187.
  • [24] K. Zimmermann, J. Matas, and T. Svoboda, “Tracking by an optimal sequence of linear predictors,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 4, pp. 677–692, 2009.
  • [25] J. Vongkulbhisal, F. D. la Torre, and J. P. Costeira, “Discriminative optimization: Theory and applications to point cloud registration,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2017.
  • [26] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.   MIT Press, 2016, http://www.deeplearningbook.org.
  • [27] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas, “Learning to learn by gradient descent by gradient descent,” in Proc. Neural Information Processing Systems, 2016, pp. 3981–3989.
  • [28] Y. Chen, M. W. Hoffman, S. G. Colmenarejo, M. Denil, T. P. Lillicrap, and N. de Freitas, “Learning to learn for global optimization of black box functions,” arXiv preprint arXiv:1611.03824, 2016.
  • [29] A. Kendall, M. Grimes, and R. Cipolla, “Posenet: A convolutional network for real-time 6-dof camera relocalization,” in Proc. IEEE Int. Conf. Computer Vision, 2015, pp. 2938–2946.
  • [30] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Deep image homography estimation,” arXiv preprint arXiv:1606.03798, 2016.
  • [31] J. Xie, L. Xu, and E. Chen, “Image denoising and inpainting with deep neural networks,” in Proc. Neural Information Processing Systems, 2012, pp. 341–349.
  • [32] S. Boyd and L. Vandenberghe, Convex Optimization.   Cambridge University Press, 2004.
  • [33] R. T. Rockafellar, Convex analysis.   Princeton University Press, 1970.
  • [34] S. Karamardian and S. Schaible, “Seven kinds of monotone maps,” J. Optimization Theory and Applicat., vol. 66, no. 1, pp. 37–46, 1990.
  • [35] P. Ochs, A. Dosoviskiy, T. Brox, and T. Pock, “An iterated algorithm for non-smooth non-convex optimization in computer vision,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2013.
  • [36] N. Hadjisavvas and S. Schaible, “Generalized monotone multivalued maps,” in Encyclopedia of Optimization.   Springer, 2001, pp. 1193–1197.
  • [37] B. Hall, Lie Groups, Lie Algebras, and Representations: An Elementary Introduction.   Springer, 2004.
  • [38] A. Myronenko and X. Song, “Point set registration: Coherent point drift,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 12, pp. 2262–2275, 2010.
  • [39]

    B. Jian and B. C. Vemuri, “Robust point set registration using gaussian mixture models,”

    IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1633–1645, 2011.
  • [40] Stanford Computer Graphics Laboratory, “The stanford 3D scanning repository,” https://graphics.stanford.edu/data/3Dscanrep/, Aug 2014, accessed: 2016-08-31.
  • [41] A. Mian, M. Bennamoun, and R. Owens, “On the repeatability and quality of keypoints for local feature-based 3D object retrieval from cluttered scenes,” Int. J. Computer Vision, vol. 89, no. 2, pp. 348–361, 2010.
  • [42] V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An accurate o(n) solution to the PnP problem,” Int. J. Computer Vision, vol. 81, no. 2, pp. 155–166, 2008.
  • [43] Y. Zheng, Y. Kuang, S. Sugimoto, K. Åström, and M. Okutomi, “Revisiting the PnP problem: A fast, general and optimal solution,” in Proc. IEEE Int. Conf. Computer Vision, 2013.
  • [44] L. Kneip, D. Scaramuzza, and R. Siegwart, “A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011.
  • [45] S. Li, C. Xu, and M. Xie, “A robust O(n) solution to the perspective-n-point problem,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1444–1450, 2012.
  • [46] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. of the ACM, vol. 24, no. 6, pp. 381–395, Jun. 1981.
  • [47] L. Ferraz, X. Binefa, and F. Moreno-Noguer, “Very fast solution to the PnP problem with algebraic outlier rejection,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2014.
  • [48] R. I. Hartley, “In defense of the eight-point algorithm,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 19, no. 6, pp. 580–593, 1997.
  • [49] T. Chan, S. Esedoglu, F. Park, and A. Yip, “Total variation image restoration: Overview and recent developments,” in Handbook of Mathematical Models in Computer Vision, N. Paragios, Y. Chen, and O. Faugeras, Eds.   Boston, MA: Springer US, 2006, pp. 17–31.
  • [50] G. Yuan and B. Ghanem, “TV: a new method for image restoration in the presence of impulse noise,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015.
  • [51] A. Chambolle, V. Caselles, M. Novaga, D. Cremers, and T. Pock, “An introduction to Total Variation for Image Analysis,” Nov. 2009, working paper or preprint. [Online]. Available: https://hal.archives-ouvertes.fr/hal-00437581
  • [52] A. Nagurney, Network Economics: A Variational Inequality Approach.   Springer US, 1999.
  • [53] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016.
  • [54] J.-C. Yao, “Multi-valued variational inequalities with k-pseudomonotone operators,” J. Optimization Theory and Applicat., vol. 83, no. 2, pp. 391–403, 1994.
  • [55] G. Romano, “New results in subdifferential calculus with applications to convex optimization,” Appl. Math. and Optimization, vol. 32, no. 3, pp. 213–234, 1995.
  • [56] S. Gold, A. Rangarajan, C.-P. Lu, P. Suguna, and E. Mjolsness, “New algorithms for 2D and 3D point matching: pose estimation and correspondence,” Pattern Recognition, vol. 38, no. 8, pp. 1019–1031, 1998.
  • [57] Y. Tsin and T. Kanade, “A correlation-based approach to robust point set registration,” in Proc. 8th European Conf. Computer Vision, 2004.
  • [58] T. Werner and A. Zisserman, “New techniques for automated architectural reconstruction from photographs,” in Proc. 7th European Conf. Computer Vision, 2002.
  • [59] ——, “Model selection for automated architectural reconstruction from multiple views,” in Proc. British Machine Vision Conf., 2002.

Appendix A Proofs for Theoretical Results

In this section, we provide the proofs for the theoretical results in the main DO paper.

a.1 Proof of the Thm. 1

Theorem 1.

(Convergence of SUM’s training error) Given a training set