We propose a novel spatially continuous framework for convex relaxations based on functional lifting. Our method can be interpreted as a sublabel-accurate solution to multilabel problems. We show that previously proposed functional lifting methods optimize an energy which is linear between two labels and hence require (often infinitely) many labels for a faithful approximation. In contrast, the proposed formulation is based on a piecewise convex approximation and therefore needs far fewer labels. In comparison to recent MRF-based approaches, our method is formulated in a spatially continuous setting and shows less grid bias. Moreover, in a local sense, our formulation is the tightest possible convex relaxation. It is easy to implement and allows an efficient primal-dual optimization on GPUs. We show the effectiveness of our approach on several computer vision problems.READ FULL TEXT VIEW PDF
Energy minimization methods have become the central paradigm for solving practical problems in computer vision. The energy functional can often be written as the sum of a data fidelity and a regularization term. One of the most popular regularizers is the total variation () due to its many favorable properties . Hence, an important class of optimization problems is given as
defined for functions with finite total variation, arbitrary, possibly nonconvex dataterms , label spaces which are closed intervals in , , and . The multilabel interpretation of the dataterm is that represents the costs of assigning label to point . For (weakly) differentiable functions equals the integral over the norm of the derivative, and therefore favors a spatially coherent label configuration. The difficultly of minimizing the nonconvex energy (1) has motivated researchers to develop convex reformulations.
Convex representations of (1) and more general related energies have been studied in the context of the calibration method for the Mumford-Shah functional . Based on these works, relaxations for the piecewise constant  and piecewise smooth Mumford-Shah functional  have been proposed. Inspired by Ishikawa’s graph-theoretic globally optimal solution to discrete variants of (1), continuous analogues have been considered by Pock et al. in [17, 18]. Continuous relaxations for multilabeling problems with finite label spaces have also been studied in .
Interestingly, the discretization of the aforementioned continuous relaxations is very similar to the linear programming relaxations proposed for MAP inference in the Markov Random Field (MRF) community[10, 22, 24, 26]. Both approaches ultimately discretize the range into a finite set of labels. A closer analysis of these relaxations reveals, however, that they are not well-suited to represent the continuous valued range that we face in most computer vision problems such as stereo matching or optical flow. More specifically, the above relaxations are not designed to assign meaningful cost values to non-integral configurations. As a result, a large number of labels is required to achieve a faithful approximation. Solving real-world vision problems therefore entails large optimization problems with high memory and runtime requirement. To address this problem, Zach and Kohli , Zach  and Fix and Agarwal  introduced MRF-based approaches which retain continuous label spaces after discretization. For manifold-valued labels, this issue was addressed by Lellmann et al. , however with the sole focus on the regularizer.
We propose the first sublabel–accurate convex relaxation
of nonconvex problems in a spatially continuous setting. It exhibits several favorable
Our formulation is compact and requires only half the amount of variables for the dataterm than the formulation in . We prove that the sublabel–accurate total variation can be represented in a very simple way, introducing no overhead compared to [17, 18]. In contrast, the regularizer in  is much more involved.
We make heavy use of the convex conjugate, which is given as for functions . The biconjugate denotes its convex envelope, i.e. the largest lower-semicontinuous convex under-approximation of . For a set we denote by the function which maps any element from to and is otherwise. For a comprehensive introduction to convex analysis, we refer the reader to 
. Vector valued functionsare written in bold symbols. If it is clear from the context, we will drop the inside the functions, e.g., we write for .
To derive a convex representation of (1), we rely on the framework of functional lifting. The idea is to reformulate the optimization problem in a higher dimensional space, in which the convex envelope approximates the nonconvex energy better than the one of the original low dimensional energy. We start by sampling the range at labels . This partitions the range into intervals so that . Clearly, any value in the range of can be written as
for and some label index . We represent such a value in the range by a -dimensional vector
where denotes a vector starting with ones followed by zeros. We call the lifted representation of , representing the graph of . This notation is depicted in Fig. 4 for . Back-projecting the lifted to the range of using the layer cake formula yields a one-to-one correspondence between and via
Due to the one-to-one correspondence between and it is clear that solving problem (5) is equivalent to finding a minimizer of the lifted energy:
Note that the constraint in (7) is essentially the nonconvex special ordered set of type 2 (SOS2) constraint . More precisely, we demand that the derivative is zero, except for two neighboring elements, which add up to one. In the following proposition, we derive the tightest convex relaxation of .
The convex envelope of (6) is given as:
where the conjugate of the individual is
with and .
See appendix. ∎
The above proposition reveals that the convex relaxation implicitly convexifies the dataterm on each interval . The equality implies that starting with yields exactly the same convex relaxation as starting with .
If is linear on each , then the convex envelopes of and coincide, where the latter is:
Consider an additional constraint for each , which corresponds to selecting in (7). The fact that our relaxation is independent of whether we choose or , along with the fact that the convex hull of two points is a line, yields the assertion. ∎
For the piecewise linear case, it is possible to find an explicit form of the biconjugate.
Let us denote by the vector with
Under the assumptions of Prop. 1, one obtains:
See appendix. ∎
Up to an offset (which is irrelevant for the optimization), one can see that (12) coincides with the dataterm of , the discretizations of [17, 18], and – after a change of variable – with . This not only proves that the latter is optimizing a convex envelope, but also shows that our method naturally generalizes the work from piecewise linear to arbitrary piecewise convex energies. Fig. (a)a and Fig. (b)b illustrate the difference of and on the example of a nonconvex stereo matching cost.
Because our method allows arbitrary convex functions on each , we can prove that, for the two label case, our approach optimizes the convex envelope of the dataterm.
In the case of binary labeling, i.e., , the convex envelope of (6) reduces to
See appendix. ∎
where the distributional derivative is a finite -valued Radon measure . We define
The individual are given by:
for some and . The intuition is that penalizes a jump from to in the direction of . Since is nonconvex we compute the convex envelope.
The convex envelope of (15) is
where is given as:
See appendix. ∎
The set from Eq. (36) involves infinitely many constraints which makes numerical optimization difficult. As the following proposition reveals, the infinite number of constraints can be reduced to only linearly many, allowing to enforce the constraint exactly.
In case the labels are ordered, i.e., , then the constraint set from Eq. (36) is equal to
See appendix. ∎
Discretizing as a -dimensional Cartesian grid, the relaxed energy minimization problem becomes
where denotes a forward-difference operator with . We rewrite the dataterm given in equation (8) by replacing the pointwise maximum over the conjugates with a maximum over a real number and obtain the following saddle point formulation of problem (20):
We numerically compute a minimizer of problem (21) using a first-order primal-dual method [6, 16] with diagonal preconditioning  and adaptive steps . It basically alternates between a gradient descent step in the primal variable and a gradient ascent step in the dual variable. Subsequently the dual variables are orthogonally projected onto the sets respectively . The projection onto the set is a simple -ball projection. To simplify the projection onto , we transform the -dimensional epigraph constraints in (22) into -dimensional scaled epigraph constraints by introducing an additional variable with:
We implement the newly introduced equality constraints (23) introducing a Lagrange multiplier . It remains to discuss the orthogonal projections onto the epigraphs of the conjugates . Currently we support quadratic and piecewise linear convex pieces . For the piecewise linear case, the conjugate is a piecewise linear function with domain . The slopes correspond to the -positions of the sublabels and the intercepts correspond to the function values at the sublabel positions.
The conjugates as well as the epigraph projections of both, a quadratic and a piecewise linear piece are depicted in Fig. 10. For the quadratic case, the projection onto the epigraph of a parabola is computed using [23, Appendix B.2].
We implemented the primal-dual algorithm in CUDA to run on GPUs. For , our implementation of the functional lifting framework , which will serve as a baseline method, requires optimization variables, while the proposed method requires variables, where is the number of points used to discretize the domain . As we will show, our method requires much fewer labels to yield comparable results, thus, leading to an improvement in accuracy, memory usage, and speed.
where denotes the input data. While there is no practical use in applying convex relaxation methods to an already convex problem such as the ROF model, the purpose of this is two-fold. Firstly, it allows us to measure the overhead introduced by our method by comparing it to standard convex optimization methods which do not rely on functional lifting. Secondly, we can experimentally verify that the relaxation is tight for a convex dataterm.
In Fig. 17 we solve (25) directly using the primal-dual algorithm , using the baseline functional lifting method  and using our proposed algorithm. First, the globally optimal energy was computed using the direct method with a very high number of iterations. Then we measure how long each method took to reach this global optimum to a fixed tolerance.
The baseline method fails to reach the global optimum even for labels. While the lifting framework introduces a certain overhead, the proposed method finds the same globally optimal energy as the direct unlifted optimization approach and generalizes to nonconvex energies.
The quadratic dataterm in (25
) is often not well suited for real-world data as it comes from a pure Gaussian noise assumption and does not model outliers. We now consider a robust truncated quadratic dataterm:
To implement (26), we use a piecewise polynomial approximation of the dataterm. In Fig. 26 we degraded the input image with additive Gaussian and salt and pepper noise. The parameters in (26) were chosen as , and . The proposed method requires significantly less labels to find lower energies than the baseline method.
We remark that Prop. 4 and Prop. 5 hold for arbitrary convex one-homogeneous functionals instead of in equation (16). In particular, they hold for the anisotropic total variation . This generalization allows us to directly compare our convex relaxation to the MRF approach of Zach and Kohli . In Fig. 39 we show the results of optimizing the two models entitled “DC-Linear” and “DC-MRF” proposed in , and of our proposed method with anisotropic regularization on the robust truncated denoising energy (26). We picked the parameters as , , and . The label space is also chosen as as described in .
Overall, all the energies are better than the ones reported in . It can be seen from Fig. 39 that the proposed relaxation is competitive to the one proposed in . In addition, The proposed relaxation uses a more compact representation and extends to isotropic regularizers. To illustrate the advantages of isotropic regularizations, Fig. (a)a and Fig. (b)b show a comparison of our proposed method for isotropic and anisotropic regularization in the next section.
Given a pair of rectified images, the task of finding a correspondence between the two images can be formulated as an optimization problem over a scalar field where each point denotes the displacement along the epipolar line associated with each . The overall cost functional fits Eq. (1). In our experiments, we computed for disparities on the Middlebury stereo benchmark  in a patch using a truncated sum of absolute gradient differences. For the dataterm of the proposed relaxation, we convexify the matching cost in each range by numerically computing the convex envelope using the gift wrapping algorithm.
The first row in Fig. 55 shows the result of the proposed relaxation using the convexified energy between two labels. The second row shows the baseline approach using the same amount of labels. Even for , the proposed method produces a reasonable depth map while the baseline approach basically corresponds to a two region segmentation.
Many sensors such as time-of-flight cameras or interferometric synthetic aperture radar (SAR) yield cyclic data lying on the circle . Here we consider the task of total variation regularized unwrapping. As is shown on the left in Fig. 76, the dataterm is a nonconvex function where each minimum corresponds to a phase shift by :
For the experiments, we approximated the nonconvex energy by quadratic pieces as depicted in Fig. 76. The label space is chosen as and the regularization parameter was set to . Again, it is visible in Fig. 76 that the baseline method shows label space discretization and fails to unwrap the depth map correctly if the number of labels is chosen too low. The proposed method yields a smooth unwrapped result using only labels.
In depth from focus the task is to recover the depth of a scene, given a stack of images each taken from a constant position but in a different focal setting, so that in each image only the objects of a certain depth are sharp. We achieve this by estimating the depth of a point by locally maximizing its contrast over the set of images. We compute the cost by using the modified Laplacian function as a contrast measure. Similar to the stereo experiments, we convexify the cost on each label range by computing the convex hull. The results are shown in Fig. 68. While the baseline method clearly shows the label space discretization, the proposed approach yields a smooth depth map. Since the proposed method uses a convex lower bound of the lifted energy, the regularizer has slightly more influence on the final result. This explains why the resulting depth maps in Fig. 68 and Fig. 55 look overall less noisy.
In this work we proposed a tight convex relaxation that can be interpreted as a sublabel–accurate formulation of classical multilabel problems. We showed that the local convex envelope involves infinitely many constraints, however we proved that it suffices to consider linearly many of those. The final formulation is a simple saddle-point problem that admits fast primal-dual optimization. Our method maintains sublabel accuracy even after discretization and for that reason outperforms existing spatially continuous methods. Interesting directions for future work include higher dimensional label spaces, manifold valued data and more general regularizers.
The proof follows from a direct calculation. We start with the definition of the biconjugate:
This shows the first equation inside the proposition. For the individual we again start with the definition of the convex conjugate:
Applying the substitution and consequently yields:
It is easy to see that
To compute the biconjugate, we write any input argument , and use to obtain
Instead of taking the supremum of all , we might as well take the supremum over all vectors p with . Care has to be taken of the first summand in the second term of the above formulation. We obtain
Note that for any being negative, the supremum immediately yields infinity by taking . Similarly, if yields infinity by taking all . For for all , and , we know that . Since equality can be obtained by choosing for all , we can reduce the above supremum to
where we used that the supremum over is attained at (still assuming that ). Let us now undo our change of variable. It is easy to see that , and for . The latter leads to
for . Considering the aforementioned constraints of , and , we finally find
We compute the individual conjugate as:
The inner maximum over is the conjugate of the -norm scaled by evaluated at . This yields:
For the overall biconjugate we have:
Since we have the over all conjugates, the set is given as the intersection of the sets described by the individual indicator functions :
First we rewrite (36) by expanding the matrix-vector product into sums:
Let . We show the equivalences:
Now let . Since it also holds that , thus it is equivalent to show (37) without the absolute value on the right hand side.