Minimizing Sum of Non-Convex but Piecewise log-Lipschitz Functions using Coresets

1 Introduction

We define below the general problem of minimizing sum of piecewise log-Lipschitz functions, and then suggest two example applications.

1.0.0.1 Minimizing sum of piecewise log-Lipschitz functions.

We consider the problem of minimizing the sum $\sum_{g \in G} g (x)$ over $x \in R^{d}$ of a set $G$ of $n$ real non-negative functions that may not be convex but satisfy the piecewise log-Lipschitz; see Definition 2. This condition means that we can partition the range of each function $g : R^{d} \to [0, \infty)$ into small $m$ subsets of $R^{d}$ (sub-ranges), such that $g$ satisfies the log-Lipschitz condition on each of these sub-ranges; see Definition 1. That is, $g$ has a single minimum in this sub-range, and increases in a bounded ratio around its local minimum. Note that $g$ might not be convex even in this sub-range. Formally, if the distance $x$ from the minimum in a sub-range is multiplied by $c > 0$ , then the value of the function increases by a factor of at most $c^{r}$ for some small (usually constant) $r > 0$ .

More generally, we wish to minimize the cost function $f (g_{1} (x), \dots, g_{n} (x))$ where $f : R^{n} \to [0, \infty)$ is a log-Lipschitz function as explained in the previous paragraph, and ${g_{1}, \dots, g_{n}}$ are the log-Lipschitz functions in $G$ .

As an application, we reduce the following problems to minimizing such a set $G$ of functions.

1.0.0.2 Aligning points-to-lines

, known as the Perspective-n-Point (PnP) problem, is a fundamental problem in computer vision which aims to compute the position of an object (formally, a rigid body) based on its position as detected in a 2D-Camera

[16, 18]. Here, we assume that the structure of the object (its 3D model) is known. This problem is equivalent to the problem of estimating the position of a moving camera, based on a captured 2D image of a known object, which is strongly related to the very common procedure of “camera calibration" that is used to estimate the external parameters of a camera using a chessboard.

Formally, the input to the problem is an ordered set $L = {ℓ_{1}, \dots, ℓ_{n}}$ of $n$ lines that intersect at the origin and an ordered set $P = {p_{1}, \dots, p_{n}}$ of $n$ points, both in $R^{3}$ . Each line represents a point in the 2D image. The output is an alignment $(R, t)$ that minimizes the sum of Euclidean distances over each point (column vector) and its corresponding line, i.e.,

min (R, t) n \sum i = 1 d i s t (R p_{i} - t, ℓ_{i}),

(1)

where the minimum is over every rotation matrix $R \in R^{3 \times 3}$ and translation vector $t \in R^{3}$ . Here, $d i s t (x, y) = {∥ x - y ∥}_{2}$ is the Euclidean distance but in practice we may wish to use non-Euclidean distances, such as distances from a point to the intersection of its corresponding line with the camera’s image plane.

While dozens of heuristics were suggested over the recent decades, this problem is open even when the points and lines are on the plane, e.g. when we wish to align a set of GPS points to a map of lines (say, highways).

We tackle a variant of this problem, when both the points and lines are in $R^{2}$ , and the lines do not necessarily intersect at the origin.

1.0.0.3 Constrained regression

is a fundamental problem in machine learning, which aims to compute a vector

x \in R^{d}

such that the inner product

(p^{T} x)

will predict the label or classification

b \in R

of a point (data record)

p \in R^{d}

. Without loss of generality we can assume that all the entries of

b

are non-negative. This motivates the problem of minimizing the error on a “training dataset" of

n

records which are the rows of an

n \times d

real matrix

A

, i.e,

{min}_{x \in R^{d}} {∥ A x - b ∥}_{z}

with respect to

ℓ_{z}

-norm where

z > 0

(including the non-standard norm

z < 1

). To avoid overfitting, or to decrease sparsity (number of non-zeroes in

x

) we may wish to keep the norm of

x

constant, say,

1

This yields the constrained optimization problem

(2)

Lagrange multiplier can then be used to obtain the problem

min λ \in R, x \in R^{d} ∥ A x - b ∥ + λ (∥ x ∥ - 1) .

(3)

Again, to our knowledge both (2) and (3) are open problems already for points on the plane $(d = 2)$ , and even for the $ℓ_{1}$ linear regression ( $z = 1$ ). A possible lee-way is to calibrate $λ$ manually, where large $λ$ implies better sparsity and less over fitting but higher fitting error, and replace the non-convex constraint $∥ x ∥ = 1$ by the convex constraint $∥ x ∥ \leq 1$ . This yields the common Lasso (least absolute shrinkage and selection operator) methods [30]

min ∥ x ∥ \leq 1 ∥ A x - b ∥ + λ ∥ x ∥

(4)

for different norms (usually combinations of $ℓ_{1}$ and $ℓ_{2}$ ). Due to the known value of $λ$ and convex constraint on $x$ , problem (4) can usually be solved using convex optimization.

1.1 Generalizations

We consider more generalizations of the above problems as follows.

1.1.0.1 Unknown matching

is the case where the matching $π : [n] \to [n]$ between the $i$ th point $p_{i}$ to its line $ℓ_{i}$ (in the PnP problem (1)) or to the $i$ th label $b_{i}$ of $b$ (in (3)) is unknown for every $i \in {1, \dots, n} = [n]$ . E.g., in the PnP problem there are usually no labels in the observed images, and in regression b may be an (unsorted) vector of anonymous votes or classifications for all the n users.

1.1.0.2 Non-distance functions

where for an error vector $v = (v_{1}, \dots, v_{n})$ of a candidate solution, the cost that we wish to minimize over all these solutions is $f (v)$ where $f : R^{n} \to [0, \infty)$ , such as $f (v) = {∥ v ∥}_{\infty}$ for “worst case" error, $f (v) = {∥ v ∥}_{1 / 2}$ which is more robust to noisy data, or $f (v) = {∥ v ∥}^{2}$ of squared distances for maximizing likelihood in the case of Gaussian noise. As in the latter two cases, the function $f$ may not be a distance functions.

1.1.0.3 Robustness to outliers

may be obtained by defining $f$ to be a function that ignores or at least put less weight on very large distance, maybe based on some given threshold. Such a function $f$ is called an M-estimator and is usually a non-convex function.

1.1.0.4 Coreset

in this paper refers to a small representation of a set $P$ of $n$ input points by a weighted (scaled) subset. The approximation is $(1 + ε)$ multiplicative factor, with respect to the cost of any item (query) in a given set $Q$ . E.g. in (1) it is the union over alignments $(R, t)$ , and in (2) $Q$ is the set of unit vectors in $R^{d}$ . Composable coresets have the property that they can be merged and re-reduced; see e.g. [1, 15].

Our main motivation for designing coresets is (i) to reduce the running time of our algorithms from polynomial to near-linear in $n$ , and (ii) handle big data computation models as follows, which is straightforward using composable coresets.

1.1.0.5 Handling big data

in our paper refers to the following computation models: (i) Streaming support for a single pass over possibly unbounded stream of items in $A$ using memory and update time that is sub-linear (usually poly-logarithmic) in $n$ . (ii) Parallel computations on distributed data that is streamed to $M$ machines where the running time is reduced by $M$ and the communication between the machines to the server should be also sub-linear in its input size $n / M$ . (iii) Dynamic data which includes deletion of pairs. Here $O (n)$ memory is necessary, but we still desire sub-linear update time.

1.2 Our contribution

1.2.0.1 Generic framework

for defining a cost function $c o s t (A, q)$ for any finite input subset $A = {a_{1}, \dots, a_{n}}$ from a set $X$ called ground set, and an item $q$ (called query) from a (usually infinite) set $Q$ . We show that this framework enables handling the generalization in Section 1.1 such as outliers, m-estimators and non-distance functions in a straightforward way. Formally, we define

c o s t (A, q) := f (l i p (D (a_{1}, q)), \dots, l i p (D (a_{n}, q))),

(5)

where $f : [0, \infty)^{n} \to [0, \infty)$ and $l i p : [0, \infty) \to [0, \infty)$ are an $O (1)$ -log-Lipschitz functions, and $D : X \times Q \to [0, \infty)$ ; See Definition 4.

1.2.0.2 Optimization of piecewise log-Lipschitz functions.

Given $x^{*} \in R$ and piecewise $O (1)$ -log-Lipschitz functions $g_{1}, \dots, g_{n}$ , we prove that one of their minima $x^{'} \in R$ approximates their value $g_{i} (x)$ in $x$ simultaneously (for every $i \in [n]$ ), up to a constant factor. See Theorem 3. This yields a finite set of candidate solutions (called centroid set) that contains an approximated solution, without knowing $x^{*}$ .

We use this result to compute $q^{'} \in Q$ that approximates $D (a_{i}, q^{*})$ simultaneously (for every $i \in [n]$ ), where $q^{*}$ is the query that minimizes $c o s t (A, q)$ over every $q \in Q$ . Observation 5 proves that $c o s t (A, q^{'})$ is the desired approximation for the optimal solution $c o s t (A, q^{*})$ in our framework.

1.2.0.3 Simultaneous optimization and matching

may be required for the special case that $A = {(a_{i}, b_{i})}_{i}^{n}$ is a set of pairs, and we wish to compute the permutation $π^{*}$ and query $q^{*}$ that minimize $c o s t (A_{π}, q)$ over every $π^{*} : [n] \to [n]$ and $q^{*} \in Q$ , where $A_{π} = {(a_{i}, b_{π (i)})}_{i}^{n}$ is the corresponding permutation of the pairs. We provide constant factor approximations for the case $f = {∥ \cdot ∥}_{1}$ in (5); See Theorems 8 and 13.

1.2.0.4 Constrained Regression

as defined in Section 1 is our first example application of the above framework, where $A = {(a_{i}, b_{i})}_{i}^{n} \subseteq R^{2} \times [0, \infty)$ , $Q = {x \in R^{2} ∣ {∥ x ∥}_{2} = 1}$ is the unit circle, and $D ((a_{i}, b_{i}), x) = | a_{i}^{T} x - b_{i} |$ for every $x \in Q$ and $i \in [n]$ . We provide the first constant factor approximation for the optimal solution ${min}_{x \in Q} c o s t (A, x)$ that takes time polynomial in $n$ . Such a solution can be computed for every cost function and $l i p$ as defined in (5), e.g., $l i p (x) = x^{1 / 2}$ where we wish to minimize the non-convex sum over $\sqrt{| a_{i}^{T} x - b_{i} |}$ ; see Theorem 7. Simultaneous optimization and matching for $f = {∥ \cdot ∥}_{1}$ are suggested in Theorem 8.

1.2.0.5 Approximated Points-to-Lines alignment

as defined in Section 1 is our second example, where $A = {(p_{i}, ℓ_{i})}_{i}^{n}$ is a set of paired point $p_{i}$ and line $ℓ_{i}$ on the plane, $Q=\textscAlignments$ is the union over every rotation matrix $R$ and a translation vector $t$ , $D ((p_{i}, ℓ_{i}), (R, t)) = {min}_{x \in ℓ_{i}} ∥ R p_{i} - t - x ∥$ for every $(R, t) \in Q$ and $i \in [n]$ . We provide the first constant factor approximation for the optimal solution ${min}_{(R, t) \in Q} c o s t (A, (R, t))$ that takes polynomial time. Such a solution can be computed for every cost function as defined in (5); see Theorem 12. Including simultaneous optimization and matching for $f = {∥ \cdot ∥}_{1}$ ; See Theorem 13.

1.2.0.6 Composable $ε$ -coresets

for aligning points-to-lines are suggested in Theorem 14 and 15, based on reduction to coresets for $ℓ_{z}$ regression.

1.2.0.7 Experimental Results

show that our algorithms performed better also in practice, compared to both existing heuristics and provable algorithms for related problems. A system for head tracking in the context of augmented reality shows that our algorithms can be applied in real-time using sampling and coresets. Existing solutions are either too slow or provide unstable images, as is demonstrated in the companion video [20].

2 Related Work

For the easier case of summing convex function, a framework was suggested in [11, 12]. However, for the case of summing non-convex functions as in this paper, each with more than one local minima, these techniques do not hold. This is why we had to use more involved algorithms in Section 4. Moreover, our generic framework such as handling outliers and matching can be applied also for the works in [11, 12].

The motivation was to obtain weak but faster $(α, β)$ -approximations that suffice for computing coresets. The polynomial time algorithms can then be applied on the coreset. A classic example is projective clustering where we wish to approximate $n$ points in $R^{d}$ by a set of $k$ affine $j$ -dimensional subspaces, such as $k$ -means ( $j = 0$ ) or PCA ( $k = 1$ ). Constant factor approximations can be computed by considering every set of $k$ subspaces, each spanned by $j + 1$ input points.

Summing of non-convex but polynomial or rational functions was suggested in [31]. This is by using tools from algebraic geometry such as semi-algebraic sets and their decompositions. For high degree polynomial such techniques may be used to compute the minima in Theorem 3. In this sense, piecewise log-Lipschitz functions can be considered as generalizations of such functions, and our framework may be used to extend them for the generalizations in Section 1.1 (outliers, matching, etc.).

2.0.0.1 Aligning points to lines.

The problem for aligning a set of points to a set of lines in the plane is natural e.g. in the context of GPS points [29, 27], finding sky patterns as in Fig. 1, or aligning pixels in a 2D image to an object that is pre-defined by linear segments [25], as in augmented reality applications [22, 32, 17].

The only known solutions are for the case of sum of squared distances and $d = 2$ dimensions, with no outliers, and when the matching between the points is given. In this case, the Lagrange multipliers method can be applied in order to get a set of $2$ nd order polynomials. For $d = 3$ the problem is called PnP (Perspective-n-Points) and has provable solutions only for the case of exact alignment (zero fitting error) [26, 19] and numerous heuristics. When the matching is unknown ICP (Iterative closest point) is the main common technique based on greedy nearest neighbours; see references in [6]. To handle outliers RANSAC [9] is heuristically used.

2.0.0.2 Constrained regression

is usually used to avoid overfitting and noise for linear regression, as explained in Section 1 and e.g. in [30, 34, 34, 33]; see references therein. The solution is usually based on relaxation to convex optimization. However, when the tradeoff parameter $λ$ is unknown, or when we want to ignore $k$ outliers, or use $M$ -estimators, the resulting problems are non-convex.

To our knowledge, no existing provable algorithms are known for handling outliers, unknown matching, or $ℓ_{z}$ norm for the case $z \in (0, 1)$ .

2.0.0.3 Coresets

have many different definitions. In this paper we use the simplest one that is based on a weighted subset of the input, which preserve properties such as sparsity of the input and numerical stability. Coresets for $ℓ_{p}$ regression were suggested in [8] using well-conditioned matrices that we cite in Theorem 14. We improve the bounds on the coreset size using the framework from [11, 7]. We also reduce the points-to-lines aligning problem to constrained $ℓ_{1}$ optimization in the proof of Theorem 31, which allows us to apply our algorithms on these coresets for the case of sum over point-line distances.

In most coresets papers, the main challenge is to compute the coreset. However, in this paper, the harder problem was to extract the desired constrained solution from the coresets which approximate every vector, and ignore the constraints.

3 Optimization Framework

In what follows, for every pair of vectors $v = (v_{1}, \dots, v_{n})$ and $u = (u_{1}, \dots, u_{n})$ in $R^{n}$ we denote $v \leq u$ if $v_{i} \leq u_{i}$ for every $i \in [n]$ . Similarly, $f : R^{n} \to [0, \infty)$ is non-decreasing if $f (v) \leq f (u)$ for every $v \leq u \in R^{d}$ .

The following definition is a generalization of Definition 2.1 in [14] from $n = 1$ to $n > 1$ dimensions, and from $R$ to $I \subseteq R^{n}$ .

Definition 1 (Log-Lipschitz function).

Let $r > 0$ , let $n \geq 1$ be an integer, and let $v = (v_{1}, \dots, v_{n}) \in R^{n}$ . Let $I$ be a subset of $R^{n}$ , and $h : I \to [0, \infty)$ be a non-decreasing function. Then $h (x)$ is $r$ -log-Lipschitz over $x \in I$ , if for every $c \geq 1$ and $x \in I \cap \frac{I}{c}$ , we have $h (c x) \leq c^{r} h (x) .$ The parameter $r$ is referred to as the log-Lipschitz constant.

Unlike previous papers, the loss fitting (“distance”) function that we want to minimize in this thesis is not a log-Lipshcitz function. However, it can be partitioned to a constant number of log-Lipschitz functions in the following sense.

Definition 2 (Piecewise log-Lipschitz).

Let $g : X \to [0, \infty)$ be a continuous function over a set $X$ , and $d i s t : X^{2} \to [0, \infty)$ be a distance function. Let $r \geq 0$ . The function $g$ is piecewise $r$ -log-Lipschitz if there is a partition of $X$ into $m$ subsets $X_{1}, \dots, X_{m}$ such that for every $i \in [m]$ :

$g$ has a unique infimum $x_{i}$ at $X_{i}$ , i.e., ${x_{i}} = {a r g m i n}_{x \in X_{i}} g (x)$ .
$h_{i} : [0, {max}_{x \in X_{i}} d i s t (x, x_{i})] \to [0, \infty)$ is an $r$ -log-Lipschitz function; see Definition 1.
$g (x) = h_{i} (d i s t (x_{i}, x))$ for every $x \in X_{i}$ .

The set of minima is denoted by $M (g) = {x_{1}, \dots, x_{m}}$ .

Suppose that we have a set of piecewise $r$ -log-Lipschitz functions, and consider the union $⋃ M (g)$ over every function $g$ in this set. The following lemma states that, for every $x \in R$ , this union contains a value $x^{'}$ such that $g (x^{'})$ approximates $g (x)$ up to a multiplicative factor that depends on $r$ .

Theorem 3 (simultaneous approximation).

Let $g_{1}, \dots, g_{n}$ be $n$ function, where $g_{i} : R \to [0, \infty)$ is a piecewise $r$ -log-Lipschitz function for every $i \in [n]$ , and let $M (g_{i})$ denote the minima of $g_{i}$ as in Definition 2. Let $x \in R$ . Then there is $x^{'} \in ⋃_{i \in [n]} M (g_{i})$ such that for every $i \in [n]$ ,

g_{i} (x^{'}) \leq 2^{r} g_{i} (x) .

(6)

Definition 4 (Optimization framework).

Let $X$ be a set called ground set, let $A = {a_{1}, \dots, a_{n}} \subset X$ be a finite input set and let $Q$ be a set of queries. Let $D : X \times Q \to [0, \infty)$ be a function. Let $l i p : [0, \infty) \to [0, \infty)$ be an $r$ -log-Lipschitz function $f : [0, \infty)^{n} \to [0, \infty)$ be an $s$ -log-Lipschitz function. Let $q \in Q$ . We define

c o s t (A, q) = f (l i p (D (a_{1}, q)), \dots, l i p (D (a_{n}, q))) .

The following observation states that if we find a query $q \in Q$ that approximates the function $D$ for every input element, then it also approximates the function $c o s t$ as defined in Definition 4.

Observation 5.

Let $c o s t (A, q) = f (l i p (D (a_{1}, q)), \dots, l i p (D (a_{n}, q)))$ be defined as in Definition 4. Let $q^{*}, q^{'} \in Q$ and let $c \geq 1$ . If $D (a_{i}, q^{'}) \leq c \cdot D (a_{i}, q^{*})$ for every $i \in [n]$ , then

c o s t (A, q^{'}) \leq c^{r s} \cdot c o s t (A, q^{*}) .

4 Algorithms for Aligning Points to Lines

In this section, we introduce our notations, describe our algorithms, and give an overview for each algorithm. See Sections D.1, D.2 and D.3 for an intuition of the algorithms presented in this section.

4.0.0.1 Notation.

Let $R^{n \times d}$ be the set of $n \times d$ real matrices. We denote by $∥ p ∥ = {∥ p ∥}_{2} = \sqrt{p_{1}^{2} + \dots + p_{d}^{2}}$ the length of a point $p = (p_{1}, \dots, p_{d}) \in R^{d}$ , by $d i s t (p, ℓ) = {min}_{x \in X} {∥ p - x ∥}_{2}$ the Euclidean distance from $p$ to an a line $ℓ$ in $R^{d}$ for every line $ℓ$ , by $p r o j (p, X)$ its projection on $X$ , i.e, $p r o j (p, X) = {a r g m i n}_{x \in X} d i s t (p, x)$ , and by $s p {p} = {k p ∣ k \in R}$ we denote the linear span of $p$ . For a matrix $V \in R^{d \times m}$ we denote by $V^{⊥} \in R^{d \times (d - m)}$ an arbitrary matrix whose columns are mutually orthogonal unit vectors, and also orthogonal to every vector in $V$ . Hence, $[V ∣ V^{⊥}]$

is an orthogonal matrix. If

p \in R^{2}

, then

p^{⊥} := q

such that

q^{T} p = 0

and

q_{x} > 0

. We denote

[n] = {1, \dots, n}

for every integer

n \geq 1

In this paper, every vector is a column vector, unless stated otherwise. A matrix $R \in R^{2 \times 2}$ is called a rotation matrix if it is orthogonal and its determinant is $1$ , i.e., $R^{T} R = I$ and $d e t (R) = 1$ . For $t \in R^{2}$ that is called a translation vector, the pair $(R, t)$ is called an alignment. We define Alignments to be the union of all possible alignments in $2$ -dimensional space.

For $π : [n] \to [n]$ and a set $A = {(a_{1}, b_{1}), \dots, (a_{n}, b_{n})}$ of $n$ pairs of elements, $A_{π}$ is defined as $A_{π} = {(a_{1}, b_{π (1)}), \dots, (a_{n}, b_{π (n)})}$ .

4.0.0.2 Algorithms.

We now present algorithms that compute a constant factor approximation for the problem of aligning points to lines, when the matching is either known or unknown. Algorithm 2 handles the case when the matching is given, i.e. given an ordered set $P = {p_{1}, \dots, p_{n}}$ of $n$ points, and a corresponding set $L = {ℓ_{1}, \dots, ℓ_{n}}$ of $n$ lines, both in $R^{2}$ , we wish to find an alignment that minimizes, for example, the sum of distances between each point in $P$ and it’s corresponding line in $L$ .

Formally, let $A = {(p_{i}, ℓ_{i})}_{i}^{n}$ be a set of $n \geq 3$ point-line pairs, $z \geq 1$ , and $Dz:A×\textscAlignments→[0,∞)$ such that $D_{z} ((p, ℓ), (R, t)) = {min}_{q \in ℓ} {∥ R p - t - q ∥}_{z}$ is the $ℓ_{z}$ distance between $R p - t$ and $ℓ$ for every $(p, ℓ) \in A$ and $(R,t)∈\textscAlignments$ . Let $c o s t, s, r$ be as defined in Definition 4 for $D = D_{z}$ . Then Algorithm 2 outputs a set of alignments that is guaranteed to contain an alignment which approximates $min(R,t)∈\textscAlignmentscost(A,(R,t))$ up to a constant factor; See Theorem 12.

Algorithm 3 handles the case when the matching is unknown, i.e. given unordered sets $P$ and $L$ consisting of $n$ points and $n$ lines respectively, we wish to find a matching function $π : [n] \to [n]$ and an alignment $(R, t)$ that minimize, for examples, the sum of distances between each point $p_{i} \in P$ and its corresponding line $ℓ_{π (i)} \in L$ .

Formally, let $c o s t$ be as defined above but with $f = {∥ \cdot ∥}_{1}$ . Then Algorithm 3 outputs a set of alignments that is guaranteed to contain an alignment which approximates $min (R, t, π) c o s t (A_{π}, (R, t))$ up to a constant factor, where the minimum is over every alignment $(R, t)$ and matching function $π$ ; See Theorem 13.

4.1 Algorithm 1: Z-Configurations

Input:	A unit vector $v = (v_{x}, v_{y})^{T}$ such that
	$\| v_{y} \| > 0$ , and $p, q, z \in R^{2}$ such that $p \neq q$ .
Output:	A tuple of matrices $P, Q, Z \in R^{2 \times 2}$
	that satisfy Lemma 9.

Set

r_{1} \leftarrow ∥ p - q ∥, r_{2} \leftarrow ∥ p - z ∥, r_{3} \leftarrow ∥ q - z ∥

2 Set

d_{1} \leftarrow \frac{r_{1}^{2} + r_{2}^{2} - r_{3}^{2}}{2 r_{1}}

;

d_{2} \leftarrow \sqrt{| r_{2}^{2} - d_{1}^{2} |}

3 Set

s \leftarrow \frac{v_{x}}{v_{y}}

;

R \leftarrow (\begin{matrix} 0 & - 1 1 & 0 \end{matrix})

;

P \leftarrow r_{1} (\begin{matrix} s & 1 0 & 0 \end{matrix})

4 Set

Q \leftarrow P^{T}

5 Set

b \leftarrow {\begin{matrix} 1, & (z - p)^{T} (q - p)^{⊥} > 0 0, & otherwise \end{matrix}

6 Set

Z \leftarrow P + \frac{d_{1}}{r_{1}} (Q - P) + b \cdot \frac{d_{2}}{r_{1}} R (Q - P)

return

(P, Q, Z)

Algorithm 1

\textscZ−Configurations(v,p,q,z)

4.1.0.1 Overview of Algorithm 1.

Algorithm 1 takes as input a unit vector $v$ and three points $p, q, z \in R^{2}$ . The vector $v$ represents a direction of a line $ℓ$ that intersects the origin. It computes $3$ matrices $P, Q, Z$ that satisfy Lemma 9; See Section D.1 for intuition and interpretation of those matrices. In Lines 1- 1 we define constants. In Line 1 we define a rotation matrix that rotates the coordinates system by $π / 2$ radians counter clockwise around the origin. In Lines 1 and 1 we compute the output matrices $P$ and $Q$ respectively.

In Line 1, $b = 1$ if $z$ is in the halfplane to the left of the vector $q - p$ , and $b = 0$ otherwise. In Line 1 we compute the output matrix $Z$ ; See Fig. 2 for an illustration.

4.2 Algorithm 2: Align

In this section we present the main algorithm, called Align; See Algorithm 2. The output of this algorithm satisfies Lemma 10. The algorithm uses our main observations and general technique for minimizing the sum of distances between the point-line pairs.

Input:	A set $A = {(p_{1}, ℓ_{1}), \dots, (p_{n}, ℓ_{n})}$ of $n$ pairs, where for every $i \in [n] = {1, \dots, n}$ ,
	we have that $p_{i}$ is a point and $ℓ_{i}$ is a line, both on the plane.
Output:	A set $C⊂\textscAlignments$ of alignments that satisfies Lemma 10.

Set

C \leftarrow \emptyset

2 Set

v_{i} \in R^{2}

and

b_{i} \geq 0

such that

ℓ_{i} = {q \in R^{2} ∣ v_{i}^{T} q = b_{i}}

for every

i \in [n]

3 for every $j, k, l \in [n]$ s.t $j \neq k$ do

4 Set

C_{1}, C_{2} \leftarrow \emptyset

5 if $| v_{j}^{T} v_{k} | \neq 1$ then

ℓ_{j}

and

ℓ_{k}

are not parallel. */

6 Set

v_{j}^{⊥} \leftarrow

a unit vector in

R^{2}

that is orthogonal to

v_{j}

Set

R_{v_{j}} \leftarrow (\begin{matrix} - v_{j}^{T} - - {v_{j}^{⊥}}^{T} - \end{matrix})

. /*

R_{v_{j}}

aligns

v_{j}

with the

x

-axis. */

Set

(P′,Q′,Z′)←\textscZ−Configurations(Rvjvk,pj,pk,pl)

. // See Algorithm 1

7 Set

P \leftarrow R_{v_{j}}^{T} P^{'}, Q \leftarrow R_{v_{j}}^{T} Q^{'}, Z \leftarrow R_{v_{j}}^{T} Z^{'}

8 Set

s \leftarrow ℓ_{j} \cap ℓ_{k}

ℓ_{j} \cap ℓ_{k}

contains one point since

ℓ_{j}

and

ℓ_{k}

are not parallel.

9 Set

c_{l} \leftarrow d i s t (s, ℓ_{l})

Set . // The set of unit vectors that minimize the distance between

p_{l}

and

ℓ_{l}

while maintaining

p_{j} \in ℓ_{j}

and

p_{k} \in ℓ_{k}

Set

C1←{(R,t)∈\textscAlignments∣Rpj−t=Px and Rpk−t=Qx and Rpl−t=Zx for every x∈X}

. /* The set of alignments that align the points

(p_{j}, p_{k}, p_{l})

with points

(P x, Q x, Z x)

for every

x \in X

. */

11 else

12 Set

C_{2} \leftarrow {(R, t)}

such that

(R,t)∈\textscAlignments

R p_{j} - t \in ℓ_{j}

and

(R, t) \in a r g m i n d i s t (R p_{k} - t, ℓ_{k})

13 Set

C \leftarrow C \cup C_{1} \cup C_{2}

return

C

Algorithm 2 Align

(A)

4.2.1 Overview of Algorithm 2

The input for Algorithm 2 is a set of $n$ pairs, each consists of a point and a line on the plane. The algorithm runs exhaustive search on all the possible tuples and outputs candidate set $C$ of $O (n^{3})$ alignments. Alignment consists of a rotation matrix $R$ and a translation vector $t$ . Theorem 10 proves that one of these alignments is the desired approximation. See Section D.2 for intuition.

Line 2 identifies each line $ℓ_{i}$ by its direction (unit vector) $v_{i}$ and distance $b_{i} > 0$ from the origin. Lines 2–2 iterates over every triple $(j, k, l)$ of input pairs such that $j \neq k$ , and turns it into a constant number of alignments $C_{1}$ . In Lines 2–2 we handle the case where the lines $ℓ_{j}$ and $ℓ_{k}$ are not parallel. In Line 2 we handle the case where $ℓ_{j}$ and $ℓ_{k}$ are parallel.

The case where $ℓ_{j}$ and $ℓ_{k}$ are not parallel. Lines 2–2 compute a rotation matrix $R_{v_{j}}$ that rotates $v_{j}$ to the $x$ -axis. Line 2 calls the sub-procedure Algorithm 1 for computing three matrices $P^{'}, Q^{'}$ and $Z^{'}$ . In Line 2 we revert the effect of the rotation matrix $R_{v_{j}}$ . Lines 2–2 compute the distance between $ℓ_{l}$ and the intersection between $ℓ_{j}$ and $ℓ_{k}$ since we assumed this intersection point is the origin in Algorithm 1. The matrix $Z$ and the line $ℓ_{l}$ are used to compute a set $X$ of $O (1)$ unit vectors in Line 2. Every $x \in X$ defines a possible positioning for the triplet. In Line 2 we define an alignment $(R, t)$ for each $x \in X$ . The union of the alignments in $C_{1}$ is then added to the output set $C$ in Line 2.

The case where $ℓ_{j}$ and $ℓ_{k}$ are parallel. In the case, we place $p_{j} \in ℓ_{j}$ , and place $p_{k}$ as close as possible to $ℓ_{k}$ . If there are more than one alignment that satisfies those conditions, then we pick an arbitrary one. This is done in Line 2.

4.3 Algorithm 3: Align+Match

Input:	A set $A = {(p_{1}, ℓ_{1}), \dots, (p_{n}, ℓ_{n})}$ and
	a cost function as in Theorem 13.
Output:	An element $(~ R, ~ t, ~ π)$ that satisfies
	Theorem 13.

Set

X \leftarrow \emptyset

2 for every $i_{1}, i_{2}, i_{3}, j_{1}, j_{2}, j_{3} \in [n]$ do

3 Set

X′←\textscAlign({(pi1,ℓj1),(pi2,ℓj2),(pi3,ℓj3)})

// See Algorithm 2

4 Set

X \leftarrow X \cup X^{'}

6Set

S←{(R,t,^π(A,(R,t),cost))∣(R,t)∈X}

/* see Definition 11 */

Set

(~ R, ~ t, ~ π) \in a r g m i n (R^{'}, t^{'}, π^{'}) \in S c o s t (A_{π^{'}}, (R^{'}, t^{'}))

. return

(~ R, ~ t, ~ π)

Algorithm 3

\textscAlign+Match(A,cost)

4.3.0.1 Overview of Algorithm 3.

Algorithm 3 takes as input a set of $n$ points and lines in $R^{2}$ , and a cost function as defined in Theorem 13. The algorithm computes an alignment $(^R,^t)∈\textscAlignments$ and a matching function $^π$ that approximate the minimal value of the given cost function; See Theorem 13.

In Line 3 we iterate over every $i_{1}, i_{2}, i_{3}, j_{1}, j_{2}, j_{3} \in [n]$ . In Lines 3- 3 we match $p_{i_{1}}$ to $ℓ_{j_{1}}, p_{i_{2}}$ to $ℓ_{j_{2}}$ and $p_{i_{3}}$ to $ℓ_{j_{3}}$ , compute their corresponding set of alignments $X^{'}$ by a call to Algorithm 2, and then add $X^{'}$ to the set $X$ . Finally, in Lines 3- 3 we compute the optimal matching for every alignment in $X$ , and pick the alignment and corresponding matching that minimize the given cost function.

5 Statements of Main Results

5.1 Constrained regression

The following lemma proves that for every two paired sets $a_{1}, \dots, a_{n} \subseteq R^{2}$ and $b_{1}, \dots, b_{n} \geq 0$ and unit vector $x \in R^{2}$ , there exists $x^{'} \in {a r g m i n}_{∥ y ∥ = 1} | a_{k}^{T} y - b_{k} |$ for some $k \in [n]$ that approximates $| a_{i}^{T} x - b_{i} |$ for every $i \in [n]$ .

Lemma 6.

Let $a_{1}, \dots, a_{n} \subseteq R^{2}$ and $b_{1}, \dots, b_{n} \geq 0$ . Then there is a set $C$ of $| C | \in O (n)$ unit vectors that can be computed in $O (n)$ time such that (i) and (ii) hold as follows:

For every unit vector $x \in R^{2}$ there is a vector $x^{'} \in C$ such that for every $i \in [n]$ ,

$| a_{i}^{T} x^{'} - b_{i} | \leq 4 \cdot | a_{i}^{T} x - b_{i} | .$ (7)
There is $k \in [n]$ such that $x^{'} \in {a r g m i n}_{∥ y ∥ = 1} | a_{k}^{T} y - b_{k} |$

The following theorem generalizes the approximation obtained in Lemma 6 to the family of cost functions defined in Definition 4.

Theorem 7.

Let $A = {(a_{1}, b_{1}), \dots, (a_{n}, b_{n})}$ be a set of $n \geq 1$ pairs, where for every $i \in [n]$ , we have that $a_{i} \in R^{2}$ and $b_{i} \geq 0$ . Let $D ((a_{i}, b_{i}), x) = | a_{i}^{T} x - b_{i} |$ for every $i \in [n]$ and unit vector $x \in R^{2}$ . Let $c o s t, s, r$ be as defined in Definition 4. Then in $n^{O (1)}$ time we can compute a unit vector $x^{'} \in R^{2}$ such that

Recall that for $π : [n] \to [n]$ and a set $A = {(a_{1}, b_{1}), \dots, (a_{n}, b_{n})}$ of $n$ pairs of elements, $A_{π}$ is defined as $A_{π} = {(a_{1}, b_{π (1)}), \dots, (a_{n}, b_{π (n)})}$ . Lemma 6 proves that for every set $A = {(a_{1}, b_{1}), \dots, (a_{n}, b_{n})} \subset R^{2} \times R$ and unit vector $x \in R^{2}$ , there is $(a_{k}, b_{k}) \in A$ , such that the minimizes $x^{'} \in {a r g m i n}_{∥ y ∥ = 1} | a_{k}^{T} y - b_{k} |$ approximates $| a_{i} x - b_{i} |$ for every $i \in [n]$ . Hence, by computing the union $C$ of minimizers of $| a_{i} x - b_{j} |$ for every $i, j \in [n]$ , we are guaranteed that for any permutation $π : [n] \to [n]$ and unit vector $x$ , one of the vectors in $C$ will approximate $| a_{i} x - b_{π (i)} |$ for every $i \in [n]$ .

Theorem 8.

Let $A = {(a_{1}, b_{1}), \dots, (a_{n}, b_{n})}$ be a set of $n \geq 1$ pairs, where for every $i \in [n]$ , we have that $a_{i} \in R^{2}$ and $b_{i} \geq 0$ . Let $D ((a, b), x) = | a^{T} x - b |$ for every $a \in R^{2}$ , $b \geq 0$ and unit vector $x \in R^{2}$ . Let $c o s t, r$ be as defined in Definition 4 for $D$ and $f (v) = {∥ v ∥}_{1}$ . Then in $n^{O (1)}$ we can compute a unit vector $x^{'} \in R^{2}$ and $π^{'} : [n] \to [n]$ that satisfy the following

c o s t (A_{π^{'}}, x^{'}) \leq 4^{r} \cdot min x, π c o s t (A_{π}, x),

where the minimum is over every unit vector $x \in R^{2}$ and $π : [n] \to [n]$ .

5.2 Aligning Points-To-Lines

Table 1 summarizes the important results that we obtained for the problem of aligning points-to-lines.

Function

Name

f (v)

l i p (x)

z

r

Time

Approximation

factor

Ref.

Related Work

Sum of Squared

Euclidean distances

{∥ v ∥}_{1}

x^{2}

2

2

O (n^{3})

16^{2}

See Section 2

Sum of Euclidean

distances

{∥ v ∥}_{1}

x

2

1

O (n^{3})

16

None

Sum of

ℓ_{z}

distances

to the power of

r

{∥ v ∥}_{1}

x^{r}

z \geq 1

r \geq 1

O (n^{3})

(\sqrt{2} \cdot 16)^{r}

None

Sum of Euclidean

distances with

k

outliers

{∥ v ∥}_{1, n - k}

x

z \geq 1

1

O (n^{3})

16

RANSAC [9]

Sum of Euclidean

distances with unknown

matching

{∥ v ∥}_{1}

x

2

1

O (n^{9})

16 ⋆

ICP [6]

Sum of

ℓ_{z}

distances

to the power of

r

with unknown matching

{∥ v ∥}_{1}

x^{r}

z \geq 1

r \geq 1

O (n^{9})

(\sqrt{2} \cdot 16)^{r} ⋆

None

Table 1: Main results of this paper. Let

P

be a set of

n

points and

L

be a set of

n

lines, both in

R^{2}

. Let

π : [n] \to [n]

be a matching function and let

A_{π} = {(p_{1}, ℓ_{π (1)}), \dots, (p_{n}, ℓ_{π (n)})}

. In this table we assume

x \in [0, \infty)

and

v \in [0, \infty)^{n}

. For every point-line pair

(p, ℓ) \in A

and

(R,t)∈\textscAlignments

, let

D_{z} ((p, ℓ), (R, t)) = {min}_{x^{'} \in ℓ} {∥ R p - t - x^{'} ∥}_{z}^{'}

for

z \geq 1

. Let

c o s t

be as defined in Definition 4, for

D = D_{z}

A = A_{π}

and

s = 1

. The approximation factor is relative to the minimal value of the

c o s t

function. Rows marked with a

⋆

have that the minimum of the function

c o s t

is computed both over every

(R,t)∈\textscAlignments

and matching function

π

The following Lemma proves that the matrices computed in Algorithm 1 satisfy some set of properties.

Lemma 9.

Let $v$ be a unit vector and $ℓ = s p {v}$ be the line in this direction. Let $p, q, z \in R^{2}$ be the vertices of a triangle such that $∥ p - q ∥ > 0$ . Let $P, Q, Z \in R^{2 \times 2}$ be the output of a call to Z-Configurations $(v, p, q, z)$ ; see Algorithm 1. Then the following hold:

$p \in x$ -axis and $q \in ℓ$ iff there is a unit vector $x \in R^{2}$ such that $p = P x$ and $q = Q x$ .
For every unit vector $x \in R^{2}$ , we have that $z = Z x$ if $p = P x$ and $q = Q x$ .

What follows is the main Lemma of Align; See Algorithm 2. The proof of this lemma is divided into 3 steps that correspond to the steps discussed in the intuition for Algorithm 2 in Section D.2.

Lemma 10.

Let $A = {(p_{1}, ℓ_{1}), \dots, (p_{n}, ℓ_{n})}$ be set of $n \geq 3$ pairs, where for every $i \in [n]$ , we have that $p_{i}$ is a point and $ℓ_{i}$ is a line, both on the plane. Let $C⊆\textscAlignments$ be an output of a call to
Align $(A)$ ; see Algorithm 2. Then for every alignment $(R∗,t∗)∈\textscAlignments$ there exists an alignment $(R, t) \in C$ such that for every $i \in [n]$ ,

d i s t (R p_{i} - t, ℓ_{i}) \leq 16 \cdot d i s t (R^{*} p_{i} - t^{*}, ℓ_{i}) .

(8)

Moreover, $| C | \in O (n^{3})$ and can be computed in <

Minimizing Sum of Non-Convex but Piecewise log-Lipschitz Functions using Coresets

Authors

Related Research

Provable Approximations for Constrained ℓ_p Regression

k-Means Clustering of Lines for Big Data

Provably Approximated ICP

Efficient Convex Optimization Requires Superlinear Memory

Optimizing a low-dimensional convex function over a high-dimensional cube

Sublinear Time Numerical Linear Algebra for Structured Matrices

Introduction to Core-sets: an Updated Survey

This week in AI

1 Introduction

1.0.0.1 Minimizing sum of piecewise log-Lipschitz functions.

1.0.0.2 Aligning points-to-lines

1.0.0.3 Constrained regression

1.1 Generalizations

1.1.0.1 Unknown matching

1.1.0.2 Non-distance functions

1.1.0.3 Robustness to outliers

1.1.0.4 Coreset

1.1.0.5 Handling big data

1.2 Our contribution

1.2.0.1 Generic framework

1.2.0.2 Optimization of piecewise log-Lipschitz functions.

1.2.0.3 Simultaneous optimization and matching

1.2.0.4 Constrained Regression

1.2.0.5 Approximated Points-to-Lines alignment

1.2.0.6 Composable ε-coresets

1.2.0.7 Experimental Results

2 Related Work

2.0.0.1 Aligning points to lines.

2.0.0.2 Constrained regression

2.0.0.3 Coresets

3 Optimization Framework

Definition 1 (Log-Lipschitz function).

Definition 2 (Piecewise log-Lipschitz).

Theorem 3 (simultaneous approximation).

Definition 4 (Optimization framework).

Observation 5.

4 Algorithms for Aligning Points to Lines

4.0.0.1 Notation.

4.0.0.2 Algorithms.

4.1 Algorithm 1: Z-Configurations

4.1.0.1 Overview of Algorithm 1.

4.2 Algorithm 2: Align

4.2.1 Overview of Algorithm 2

4.3 Algorithm 3: Align+Match

4.3.0.1 Overview of Algorithm 3.

5 Statements of Main Results

5.1 Constrained regression

Lemma 6.

Theorem 7.

Theorem 8.

5.2 Aligning Points-To-Lines

Lemma 9.

Lemma 10.

1.2.0.6 Composable $ε$ -coresets