Log In Sign Up

Learning Less is More - 6D Camera Localization via 3D Surface Regression

by   Eric Brachmann, et al.

Popular research areas like autonomous driving and augmented reality have renewed the interest in image-based camera localization. In this work, we address the task of predicting the 6D camera pose from a single RGB image in a given 3D environment. With the advent of neural networks, previous works have either learned the entire camera localization process, or multiple components of a camera localization pipeline. Our key contribution is to demonstrate and explain that learning a single component of this pipeline is sufficient. This component is a fully convolutional neural network for densely regressing so-called scene coordinates, defining the correspondence between the input image and the 3D scene space. The neural network is prepended to a new end-to-end trainable pipeline. Our system is efficient, highly accurate, robust in training, and exhibits outstanding generalization capabilities. It exceeds state-of-the-art consistently on indoor and outdoor datasets. Interestingly, our approach surpasses existing techniques even without utilizing a 3D model of the scene during training, since the network is able to discover 3D scene geometry automatically, solely from single-view constraints.


page 1

page 3

page 5

page 8

1 Introduction

Tasks such as inserting a virtual character into the smartphone view of your living room in a plausible way or guiding a self-driving taxi through New York all require precise localization. While different devices, like smartphones or taxis, come with different senors, an RGB camera is often present because of its low cost and rich output. In this work, we present a system which estimates the 6D camera pose, consisting of position and orientation, within a known 3D environment from a single RGB image.

In the last five years, we have seen an impressive leap forward of what computer vision can achieve due to advances in machine learning. Camera localization is a particularly difficult learning problem because training data is usually limited. Recording a dense or even regular sampling of 6D camera views for any given scene is impractical. Therefore, generalization to unseen views is a central capability of any camera localization system.

Because of scarce training data, learning the direct mapping of global image appearance to camera pose with a general purpose CNN (convolutional neural net) has seen only limited success. Approaches like PoseNet [11] and its derivatives [29, 10] exhibit low localization accuracy so far.

An alternative paradigm decomposes camera localization into a sequence of less complex tasks of which only some are learned. Recently, Brachmann et al. [2] presented the differentiable RANSAC (DSAC) pipeline for camera localization. It builds upon the scene coordinate regression framework originally proposed by Shotton et al. [23]. The main idea is to map image patches to corresponding points in 3D scene space, so called scene coordinates. This step can be learned even from limited data, since local patch appearance is relatively stable w.r.t. to view change. The camera pose, which aligns the image and predicted scene coordinates, can be estimated using RANSAC.

Specifically, in the case of DSAC [2], one CNN predicts scene coordinates, and then random subsets of scene coordinates are used to create a pool of camera pose hypotheses. Each hypothesis is scored by a second CNN (“scoring CNN”) according to its consensus with the global, i.e. image-wide, scene coordinate predictions. Based on these scores, one hypothesis is probabilistically chosen, refined and returned as the final camera pose estimate. The pipeline can be trained end-to-end by optimizing the expected loss of chosen hypotheses.

Brachmann et al. report state-of-the-art accuracy for indoor camera localization [2], but we see three main short-comings of the DSAC pipeline. Firstly, the scoring CNN is prone to overfit because it can memorize global patterns of consensus to differentiate good from bad pose hypotheses. For example, the CNN might focus on where errors occur in the image rather than learning to assess the quality of errors. However, error location does not generalize well to unseen views. Secondly, initializing the pipeline for end-to-end training requires RGB-D training data or a 3D model of the scene to generate scene coordinate ground truth. Neither might be available in some application scenarios. Thirdly, end-to-end learning is unstable because in the DSAC pipeline [2]

pose refinement is differentiated via finite differences which leads to high gradient variance.

In this work, we propose a new, fully differentiable camera localization pipeline which has only one learnable component, a fully convolutional neural net for scene coordinate regression. The output neurons of this network have a limited receptive field, preserving the patch-based nature of scene coordinate prediction. For hypothesis scoring, we utilize a soft inlier count instead of a learnable CNN. We show that this simple, differentiable scoring strategy is a reliable measure of pose quality, and yet impervious to overfitting. We present a new entropy control method to automatically adapt the magnitude of score values to ensure broad hypotheses distributions for stable end-to-end learning. We also deploy a new, analytical approximation of pose refinement gradients for additional training stability. Our pipeline is fast and substantially more accurate than state-of-the-art camera localization methods.

Additionally, and in contrast to previous works [23, 28, 9, 5, 3, 2], we explore learning scene coordinate regression without utilizing a 3D scene model or RGB-D training data. RGB-D data might not be available for outdoor scenes, and creating a scene reconstruction often requires tedious trial and error parameter search and manual corrections. Our system is able to discover an approximate scene geometry automatically due to a coarse initialization followed by optimization of scene coordinate reprojection errors. We can still utilize a 3D model if available but do not depend on it for accurate camera localization, see Fig. 1.

In the following, we summarize our main contributions.

  • We present a new camera localization pipeline where a CNN regressing scene coordinates is the only learnable component. We implement hypothesis scoring with a new, entropy controlled soft inlier count without learnable parameters, which massively increases generalization capabilities.

  • We show that pose refinement can be effectively differentiated using a local linearization which results in stable end-to-end learning.

  • To the best of our knowledge, we are the first to show that scene coordinate regression can be learned using RGB images with associated ground truth poses, alone. Using a 3D model of the scene is optional since the system can discover scene geometry automatically.

  • We improve accuracy of RGB-based 6D camera localization on three datasets, both indoor and outdoor, independent of training with or without a 3D model.

Related Work.

Image-based localization has been addressed using image retrieval techniques,

e.g. in [22] or more recently in [4, 1]. These methods match a query image to an image database annotated with pose information like geolocation. While these methods can scale to extremely large environments, they usually provide only a coarse estimate of the camera location.

Instead of matching to a database, Kendall et al. [11] proposed PoseNet, a CNN which learns to map an image directly to a 6D camera pose. The method has been improved in [10] by using a reprojection loss, and in [29] by using a more expressive architecture. Although accuracy increased somewhat in these recent works, they are still inferior to competing techniques discussed next.

Accurate 6D camera poses can be recovered using sparse feature-based pipelines. Matching local image descriptors to 3D points of a Structure-from-Motion scene reconstruction yields a set of 2D-3D correspondences, from which an accurate pose estimate can be recovered [13]. Research focused on making descriptor matching efficient [15], robust [25, 19] and scale to large outdoor environments [14, 18, 20]. However, local feature detectors rely on sufficiently textured scenes and good image quality [11]. Scene reconstruction can also be difficult for some environments [29] such as scenes with repeated texture elements. We surpass the accuracy of state-of-the-art feature-based methods for indoor and outdoor camera localization tasks, often even without using a 3D reconstruction.

The original scene coordinate regression pipeline for RGB-D camera localization of Shotton et al. [23] is related to sparse feature approaches by recovering camera pose by means of 2D-3D correspondences. But instead of matching a discrete set of points, Shotton et al

. formulate correspondence prediction as a continuous regression problem. A random forest learns to map any image patch to a 3D scene point. The scene coordinate regression pipeline has been improved in several follow-up works,

e.g. in terms of accuracy [28, 9] or learning camera localization on-the-fly [5]. However, these methods heavily depend on a depth channel which greatly simplifies the problem due to strong geometric constrains. An RGB version of the scene coordinate regression pipeline was proposed in [3] using an auto-context random forest. Similarly, Massiceti et al. [17] use a random forest mapped to a neural net within a similar pipeline. However, both systems can only be trained with scene coordinate ground truth, using either RGB-D data or a 3D scene model, and were not trained in an end-to-end fashion. Our approach is most closely related to the DSAC pipeline [2] which was introduced in detail above.

2 Method

Figure 2: System Overview. Given an RGB image, we estimate the 6D camera pose in two stages. Firstly, a fully convolutional network (FCN) regresses 3D scene coordinates (XYZ mapped to RGB for visualization). This CNN is the only learnable component of our system. Secondly, the system optimizes the pose by sampling a pool of hypotheses, scoring them using a soft inlier count, selecting one according to the scores, and refining it as the final estimate. The second stage contains no learnable parameters but is fully differentiable.

Our pipeline follows the basic framework of differentiable RANSAC (DSAC) [2] which we describe in Sec. 2.1 for the task of camera pose estimation. In Sections 2.2 and 2.3, we explain the key architectural differences to the DSAC pipeline, namely using a fully convolutional network for scene coordinate regression, and scoring pose hypothesis using a soft inlier count without learnable parameters. See also Fig. 2 for an overview of our approach. We discuss the training procedure of our pipeline, with and without the use of a 3D scene model, in Sec. 2.4.

2.1 Background

Given an RGB image , we aim at finding an estimate of the 6D camera pose consisting of a 3D translation and a 3D rotation . Our system has learnable parameters which control the search for pose estimate . Differentiable RANSAC [2] estimates in the following steps:

  1. Scene Coordinate Regression. A CNN predicts for each pixel with position the corresponding 3D point in the local coordinate frame of the scene. This scene coordinate defines a 2D-3D correspondence between the image and the scene.

  2. Pose Hypothesis Sampling. Four scene coordinates suffice to define a unique camera pose by solving the perspective-n-point problem [8]. Since predictions can be erroneous, a pool of pose hypotheses is generated by selecting random 4-tupels of scene coordinate predictions. Each hypothesis depends on parameters via the corresponding scene coordinates.

  3. Hypothesis Selection. A function scores the consensus of each hypothesis with all scene coordinate predictions. One hypothesis with index

    is selected according to a probability distribution

    which is derived from the score values. Hypotheses with a high score are more likely to be selected. Hyper-parameter controls the broadness of the distribution, and will be discussed together with the details of scoring in Sec. 2.3. Selecting hypothesis probabilistically facilitates end-to-end learning, as we will discuss shortly.

  4. Hypothesis Refinement. Refinement is an iterative procedure which alternates between determining inlier pixels using the current pose estimate, and optimizing the estimate w.r.t. the inliers. We discuss the details of refinement in Sec. 2.4. The selected and refined pose hypothesis is the final estimate of the system, i.e. .

Learning the Pipeline. We assume a set of training images with ground truth poses . The probabilistic selection of a pose hypothesis in step 3 allows for optimizing learnable parameters by minimizing the expected pose loss of the final estimate over the training set [2]:


Any differentiable loss qualifies but we follow [2] by using , i.e. the maximum of rotational and translational error. Partial derivatives of Eq. 1 w.r.t. parameters are given by


where we use () as a stand-in for corresponding arguments of Eq. 1. Eq. 2 allows us to learn our pipeline in an end-to-end fashion using standard back propagation.

2.2 Scene Coordinate Regression

The DSAC pipeline [2] uses a CNN for scene coordinate regression which takes an image patch of px as input and produces one scene coordinate prediction for the center pixel. This design is inefficient because the CNN processes neighboring patches independently without reusing computations. They alleviate the problem by sampling patches per image instead of making a dense prediction for all patches. Instead, we use a fully convolutional network [16], although without upsampling layers. Our CNN takes an RGB image of px as input and produces scene coordinate predictions, i.e. we regress more scene coordinates in less time. Similar to DSAC [2], we use a VGG-style [24] architecture with 30M parameters, and our output neurons have a receptive field of px.

2.3 Hypothesis Scoring

Scoring determines which camera pose hypothesis is chosen and refined to yield the final estimate. DSAC [2] uses a separate CNN for this task. This scoring CNN takes a image of reprojection errors, and regresses a score value for each hypothesis. The reprojection error for pixel and hypothesis is defined as


where is the camera calibration matrix. We assume homogeneous coordinates, and application of perspective division before calculating the norm. The final hypothesis is chosen according to the softmax distribution :


where hyper-parameter is a fixed scaling factor that ensures a broad distribution. Parameter controls the flow of gradients in end-to-end learning by limiting or enhancing the influence of hypotheses with smaller scores compared to hypotheses with larger scores.

We identify two problems when learning function . Firstly, the image of reprojection errors contains information about the global image structure, e.g. where errors occur. Scene coordinate regression generalizes well because only local, i.e. patch-based, predictions are being made. The scoring CNN of DSAC [2] however learns patterns in the global error image that do not generalize well to unseen views. Secondly, during end-to-end learning, the score CNN has an incentive to produce increasingly large scores which puts more weight on the best hypothesis over all other hypotheses in the pool. At some point, one hypothesis will have probability 1 resulting in the minimal expected task loss, see Eq. 1. Note that the scaling factor is fixed in [2]. Hence, distribution can collapse, leading to overfitting and training instability. Regularization might be able to alleviate both problems but we show that a simple and differentiable inlier counting schema is an effective and robust measure of camera pose quality, see Fig. 3.

Figure 3: Generalization Capabilities. We train two CNNs for scene coordinate regression and pose hypothesis scoring following DSAC [2] but using only two training images (gray). The system cannot generalize to an unseen test image (green), and produces an estimate far off (orange, 3.7m and 12.6 pose error). Exchanging the scoring CNN for a soft inlier count (see Sec. 2.3), we obtain an accurate estimate (blue, 0.1m and 0.3 pose error). Note that, apart from the scoring function, we fix all components of the test run, including pose hypotheses sampling. This experiment illustrates that scene coordinate regression generalizes well, while score regression does not.

Soft Inlier Counting. The original RANSAC-schema [6] measures hypothesis consensus by counting data points explained by the model, so called inliers. In our application, the inlier count is given by , where denotes the Heaviside step function, denotes a reprojection error (see Eq. 3), and is the inlier threshold. Several earlier scene coordinate regression works relied on the inlier count to score hypotheses, e.g. [23, 3]

. For our purpose, we construct a differentiable version by substituting the step function with a sigmoid function.


where hyper-parameter controls the softness of the sigmoid. This scoring function is similar to MSAC [26] but does not use a hard cut-off. We use Eq. 5 in distribution to select a pose hypothesis.

Controlling Entropy. The magnitude of inlier scores can vary significantly depending on the difficulty of the scene, usually ranging from to for different environments. The magnitude can also change during end-to-end learning when scene coordinate regression improves and produces smaller reprojection errors. As mentioned earlier, keeping scores within a reasonable range is important for having a broad distribution , and hence stabilizing end-to-end training. Setting manually per scene is a tedious task, hence we adapt automatically during end-to-end training. We measure distribution broadness via the Shannon entropy as a function of :


We optimize according to with target entropy value via gradient descent in parallel to end-to-end training of the pipeline. This schema establishes the target entropy within the first few iterations of end-to-end training and keeps it stable throughout.

2.4 Training Procedure

Our pipeline can be trained in an end-to-end fashion using pairs of RGB images and ground truth poses, but doing so from scratch will fail as the system quickly reaches a local minimum. The DSAC pipeline is initialized using scene coordinate ground truth extracted from RGB-D training data [2]. We propose a new 3-step training schema with different objective functions in each step. Depending on whether a 3D scene model is available or not, we use rendered or approximate scene coordinates to initialize the network in the first step. Training steps two and three improve the accuracy of the system which is crucial when no 3D model was provided for initialization.

Scene Coordinate Initialization. In the first training step, we initialize our pipeline similar to DSAC [2] by optimizing


We render scene coordinates using ground truth poses , and a 3D scene model, if available. Without a 3D model, we approximate scene coordinate by , where and are the 2D coordinates of pixel , denotes the focal length, and

represents a constant depth prior. This heuristic assumes that all scene points have a constant distance from the camera plane, see Fig. 

4 for a visualization. The heuristic ignores scene geometry completely. However, it effectively disentangles camera views by coarsely assigning the correct range of scene coordinate values to different spatial parts of the 3D environment. The heuristic itself will yield poor localization accuracy, but serves as basis for the next training step.

Figure 4: Initial Scene Coordinate Heuristic. When ground truth scene coordinates (middle) are unknown, we use a heuristic (right) for initializing our pipeline. This heuristic assumes that each scene point has a constant distance to the camera plane.

Optimization of Reprojection Error. In a second training step, we optimize the reprojection error, calculated using the ground truth pose. It encodes single view constraints that are effective for recovering the correct depth of scene points in case we used the heuristic in the first training step. Thus, we optimize


which lets our system learn about the scene geometry without using a 3D scene model. However, we found that optimizing the reprojection error can improve accuracy, even if a 3D model was available for the initialization.

End-to-End Optimization. In a third step, we train our pipeline in an end-to-end fashion according to Eq. 1. This requires all components to be differentiable, including pose refinement. In DSAC [2], refinement gradients are calculated via finite differences. Note that refinement depends on thousands of inputs, namely all scene coordinate predictions, which makes calculation of finite differences slow, and results in high gradient variance due to numerical instabilities. In the following, we discuss pose refinement in detail, and explain an efficient, analytical approximation for refinement gradients, resulting in stable end-to-end training.

As mentioned earlier, pose refinement alternates between, firstly, determining a set of inlier pixels w.r.t. to the current pose estimate, and, secondly, optimizing the pose w.r.t. to reprojection errors over the inlier set. We define the inlier set to contain all pixels with a reprojection error below a threshold , i.e.

. For pose optimization, we combine the reprojection errors of all inliers within one residual vector:


We optimize the pose according to:


We use the iterative Gauss-Newton algorithm, which gives the following update rule:


where denotes the iteration number, and we abbreviate by . The Jacobean matrix contains partial derivatives . We optimize until convergence, re-calculate the inlier set, and repeat until the inlier set converges, too. Note that for DSAC [2] the number of refinement iterations and the number of inliers considered are limited to reduce the computational demand of finite differences. We do not have to make similar concessions, here.

The model linearization of Gauss-Netwon allows us to approximate refinement gradients around the optimum found in the last optimization iteration [7]. We fix the optimum as which makes the last update step , and the corresponding derivatives


which allow for stable end-to-end training according to Eq. 1.

3 Experiments

Dataset Training w/ 3D Model w/o 3D Model
PoseNet [10]
(Geom. Loss)
Search [20]
DSAC [2]
(RGB Training)
PoseNet [10]
(Pose Loss)
LSTM [29]
Chess 0.13m, 4.5 0.04m, 2.0 0.02m, 1.2 0.02m, 0.5 0.14m, 4.5 0.24m, 5.8 0.02m, 0.7
Fire 0.27m, 11.3 0.03m, 1.5 0.04m, 1.5 0.02m, 0.8 0.27m, 11.8 0.34m, 11.9 0.04m, 1.2
Heads 0.17m, 13.0 0.02m, 1.5 0.03m, 2.7 0.01m, 0.8 0.18m, 12.1 0.21m, 13.7 0.24m, 10.0
Office 0.19m, 5.6 0.09m, 3.6 0.04m, 1.6 0.03m, 0.7 0.20m, 5.8 0.30m, 8.1 0.03m, 0.8
Pumpkin 0.26m, 4.8 0.08m, 3.1 0.05m, 2.0 0.04m, 1.0 0.25m, 4.8 0.33m, 7.0 0.04m, 1.1
Kitchen 0.23m, 5.4 0.07m, 3.4 0.05m, 2.0 0.04m, 1.0 0.24m, 5.5 0.37m, 8.8 0.05m, 1.3
Stairs 0.35m, 12.4 0.03m, 2.2 1.17m, 33.1 0.10m, 2.5 0.37m, 10.6 0.40m, 13.7 0.27m, 5.4
Great Court 7.00m, 3.7 - *2.80m, 1.5 0.52m, 0.3 6.83m, 3.5 - 0.98m, 0.8
K. College 0.99m, 1.1 0.42m, 0.6 *0.30m, 0.5 0.18m, 0.4 0.88m, 1.0 0.99m, 1.0 0.22m, 0.4
Old Hospital 2.17m, 2.9 0.44m, 1.0 0.33m, 0.6 0.21m, 0.4 3.20m, 3.3 1.51m, 4.3 0.22m, 0.4
Shop Facade 1.05m, 4.0 0.12m, 0.4 0.09m, 0.4 0.06m, 0.3 0.88m, 3.8 1.18m, 7.4 0.08m, 0.4
St M. Church 1.49m, 3.4 0.19m, 0.5 *0.55m, 1.6 0.14m, 0.5 1.57m, 3.2 1.52m, 6.7 0.22m, 0.8
Street 20.7m, 25.7 0.85m, 0.8 - - 20.3m, 25.5 - -
Table 1: Median 6D Localization Errors. We report results for the 7Scenes dataset [23] and the Cambridge Landmarks dataset [11]. We mark best results bold (if both, translational and rotational error, are lowest). Results of DSAC marked with an asterisk (*) are before end-to-end optimization which did not converge. A dash (-) indicates that a method failed completely.
Figure 5: Indoor Localization Accuracy. We show the percentage of test frames of the 7Scenes (a) resp. the 12Scenes (b) dataset with a pose error below 5cm and 5. We mark our method blue. Note that all competitors depend on a 3D model. For our method, we show results after each of our 3 training steps on the 7Scenes dataset (c).

We evaluate our approach on three publicly available camera localization datasets, both indoor and outdoor.

7Scenes [23]. This RGB-D dataset comprises of 7 difficult indoor scenes with motion blur, repeating structures and texture-less surfaces. Several thousand frames are given as training and test sequences for each scene. The dataset includes ground truth poses and accurate 3D scene models. We ignore depth channels and utilize only RGB images.

12Scenes [27]. This dataset is very similar to 7Scenes but has larger indoor environments, and contains smaller training sets of several hundred frames per scene.

Cambridge Landmarks [11]. The dataset contains RGB images of six large outdoor environments, divided in training and test sequences of several hundred frames. Coarse SfM reconstructions are also provided.

Parameter Settings. We train our pipeline for a fixed number of iterations using ADAM [12] on full training sets, and select hyper-parameters that achieve the lowest training loss. During test time we always choose the hypothesis with maximum score. See the appendix (Sec. A) for a full parameter listing.

Competitors. We compare to the latest incarnation of PoseNet [10] which can be trained using a standard pose loss or, utilizing a 3D model, a geometric loss for improved accuracy. We also compare to the Spatial LSTM of [29]. We include results of several sparse feature baselines, most notably Active Search [20]. We compare to DSAC [2] trained using RGB-D training data as in [2], and using rendered scene coordinates (denoted “RGB training”).

3.1 Camera Localization Accuracy

We list our main experimental results in Table 1 for the 7Scenes and Cambridge datasets. Compared to the PoseNet variants [10, 29] we improve accuracy by a factor of 10 for many scenes, and compared to the sparse feature-based Active Search [20] by a factor of 2. Compared to DSAC [2], which is the strongest competitor, we massively improve accuracy for the Cambridge dataset. We observe only a small to moderate loss in accuracy when our method is trained without a 3D scene model. Note that the only two competitors that do not depend on a 3D model, namely PoseNet [10] and the spatial LSTM [29], achieve a much lower accuracy. In fact, for most scenes, our method trained without a 3D model surpasses the accuracy of competitors utilizing a 3D model. Similar to DSAC [2] and the Spatial LSTM [29], we were not able to estimate reasonable poses for the Cambridge Street scene which is one order of magnitude larger than the other outdoor scenes. The capacity of our neural network might be insufficient for this scene scale but we did not explore this possibility.

The median pose accuracy used in Table 1 does not reflect the frequency of wrong pose estimates very well. Therefore, we show the percentage of test images with an pose error below 5cm and 5 for the 7Scenes dataset in Fig. 5 a). Note that all competitors listed require a 3D model of the scene. PoseNet [10] and the Spatial LSTM [29] do not report results using this measure, but based on their median accuracy they presumably achieve less than 50% on 7Scenes. We outperform all competitors, most notably DSAC [2] trained with RGB-D data (+13.6%). When training DSAC using a 3D model (“RGB Training”), its performance drops by 6.6% due to inaccuracies in the 3D model. Our method trained without a 3D model exceeds the accuracy of DSAC trained with a 3D model by 3.0%.

We show results for the 12Scenes dataset in Fig. 5 b). We achieve an accuracy of 96.4% with a 16.7% margin to DSAC. Training without a 3D model still achieves a good accuracy comparable to a sparse feature baseline (SIFT+PNP [27]). See Fig. 6 for a qualitative comparison of DSAC and our method trained with and without a 3D model, respectively. We include additional qualitative results in the supplementary video111

Figure 6: Estimated Camera Trajectories. We plot estimated camera locations as trajectories within the respective 3D scene model (untextured for 7Scenes). We show ground truth in green, DSAC [2] estimates in orange, and our results in blue and cyan when trained with and without a 3D model, respectively. Note that DSAC produces many wrong estimates despite being trained with a 3D model.
Figure 7: Learning Scene Geometry. We show scene coordinate predictions for test images after end-to-end optimization. For comparison, we calculate ground truth scene coordinates (b) using measured depth (7Scenes) or the 3D scene model (Cambridge Hospital). When trained using a 3D model (c), our method learns the scene geometry accurately. When trained without a 3D model (d), our method discovers an approximate geometry, automatically. The last two columns show a 3D mesh representation of column (d) and (b).

3.2 Detailed Studies

Inlier Count vs. Scoring CNN. We retrain DSAC, substituting the scoring CNN for our soft inlier count. We measure the percentage of test frames with an error below 5cm and 5. Results improve from 55.9% to 58.9% for 7Scenes. The effect is strongest for the Heads and Stairs scenes (+19% resp. +8%) which have the smallest training sets. Accuracy for 12Scenes, where all training sets are small, also increases from 79.7% to 89.6%. We conclude that the soft inlier count helps generalization, considerably.

Impact of Training Steps. See Fig. 5 c) for a breakdown of our 3-step training procedure. When a 3D scene model is available for training, the first training step already achieves high accuracy. Optimizing the reprojection error and end-to-end training improve accuracy slightly. When no 3D model is available, optimizing the reprojection error is imperative for good results, since discovering scene geometry is necessary to generalize to unseen views. End-to-end training can improve accuracy further but effects are small. We observed that end-to-end training alone is insufficient to recover from the heuristic initialization.

Learning Scene Geometry. We visualize the approximate scene geometry discovered by our system when trained without a 3D model in Fig. 7. Although our heuristic initialization ignores scene geometry entirely, the system recovers depth information through optimization of reprojection errors.

Stability of End-to-End Training. For three out of six Cambridge scenes, DSAC’s end-to-end training did not converge despite manually tuning training parameters. In contrast, end-to-end training of our method converges for all scenes due to learning with broad hypothesis distributions, and our improved approximation of refinement gradients.

4 Conclusion

We have presented a method for 6D camera pose estimation from RGB images which surpasses accuracy of sparse feature-based and learning-based competitors. The system can learn from small training sets, and still generalize well to unseen views. Training can utilize a 3D scene model, or discover scene geometry automatically. The system processes an image in 200ms on a Tesla K80 GPU.

Our method scales to large outdoor scenes but fails on city-scale scenes like the challenging Cambridge Street scene [11]. In the future, we could pair our system with an image retrieval approach as proposed in [21].


This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 647769). The computations were performed on an HPC Cluster at the Center for Information Services and High Performance Computing (ZIH) at TU Dresden. We thank Tobias Witt for his help visualizing our results, and Lynton Ardizzone for inspiring discussions on the differentiability of iterative optimization.

Appendix A Parameter Listing

We list the main parameter settings for executing (Sec. A.1) and learning (Sec. A.2) our pipeline. With very few exceptions (stated below), we use the same parameter settings for all scenes, indoor and outdoor.

a.1 Pipeline Parameters

Our FCN [16] network architecture takes an image of px as input. We re-scale larger images to px height. Should an image be wider than px after re-scaling, we crop it centrally to px width. After scene coordinate prediction, we sample hypotheses using random 4-tuples of points and the algorithm of [8]. We reject hypotheses where the reprojection error of the corresponding 4-tuple is larger than the inlier threshold , and sample again. We set the inlier threshold px for all scenes, except “Cambridge St Mary’s Church” and “Cambridge Great Court” where we use px. For the soft inlier count (Eq. 5) we use a softness factor . We refine the selected hypothesis until convergence or for a maximum of 100 iterations.

a.2 Learning Parameters

Our FCN [16] network architecture predicts scene coordinates for a input image, i.e. it predicts one scene coordinate for each px image block. To make full use of the training data, we randomly shift training images by a maximum of px, horizontally and vertically. We re-scale training images to px height. Should an image be wider than px after re-scaling, we crop it to px width using random horizontal offsets.

We optimize using ADAM [12] and a batch size of 1 image. The remaining learning hyper-parameters differ for the three different training steps described in the main paper.

Scene Coordinate Initialization. We use an initial learning rate of , and train for 300k iterations. After 100k iterations we halve the learning rate every 50k iterations.

When initializing the pipeline using our scene coordinate heuristic instead of rendered scene coordinates, we reduce training to 100k iterations and utilize only of the training data. This is to avoid overfitting to the heuristic. For the heuristic, we use a constant depth prior of m for indoor scenes, and m for outdoor scenes.

Optimization of Reprojection Error. We use an initial learning rate of , and train for 300k iterations. After 100k iterations we halve the learning rate every 50k iterations. We clamp gradients to before passing them to the network.

End-to-End Optimization. We use an initial learning rate of , and train for 50k iterations222We had to reduce the learning rate to for the scene “Cambridge Great Court”, when learning without a 3D model. This is the only experiment where adjusting the training parameters was necessary for convergence.. We halve the learning rate after 25k iterations. We clamp gradients to before passing them to the network.

For our entropy control schema, we set scale parameter , initially. We optimize using ADAM [12] and a learning rate of for a target entropy of bit.


  • [1] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In CVPR, 2016.
  • [2] E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother. DSAC-Differentiable RANSAC for camera localization. In CVPR, 2017.
  • [3] E. Brachmann, F. Michel, A. Krull, M. Y. Yang, S. Gumhold, and C. Rother. Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In CVPR, 2016.
  • [4] S. Cao and N. Snavely. Graph-based discriminative learning for location recognition. In CVPR, 2013.
  • [5] T. Cavallari, S. Golodetz, N. A. Lord, J. Valentin, L. Di Stefano, and P. H. Torr. On-the-fly adaptation of regression forests for online camera relocalisation. In CVPR, 2017.
  • [6] M. A. Fischler and R. C. Bolles. Random Sample Consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 1981.
  • [7] W. Förstner and B. P. Wrobel. Photogrammetric Computer Vision – Statistics, Geometry, Orientation and Reconstruction. 2016.
  • [8] X.-S. Gao, X.-R. Hou, J. Tang, and H.-F. Cheng. Complete solution classification for the perspective-three-point problem. TPAMI, 2003.
  • [9] A. Guzman-Rivera, P. Kohli, B. Glocker, J. Shotton, T. Sharp, A. Fitzgibbon, and S. Izadi. Multi-output learning for camera relocalization. In CVPR, 2014.
  • [10] A. Kendall and R. Cipolla.

    Geometric loss functions for camera pose regression with deep learning.

    In CVPR, 2017.
  • [11] A. Kendall, M. Grimes, and R. Cipolla. PoseNet: A convolutional network for real-time 6-DoF camera relocalization. In ICCV, 2015.
  • [12] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, 2014.
  • [13] Y. Li, N. Snavely, and D. P. Huttenlocher. Location recognition using prioritized feature matching. In ECCV, 2010.
  • [14] Y. Li, N. Snavely, D. P. Huttenlocher, and P. Fua. Worldwide pose estimation using 3D point clouds. In ECCV. 2012.
  • [15] H. Lim, S. N. Sinha, M. F. Cohen, and M. Uyttendaele. Real-time image-based 6-dof localization in large-scale environments. In CVPR, 2012.
  • [16] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
  • [17] D. Massiceti, A. Krull, E. Brachmann, C. Rother, and P. H. S. Torr. Random forests versus neural networks - what’s best for camera localization? In ICRA, 2017.
  • [18] T. Sattler, M. Havlena, F. Radenovic, K. Schindler, and M. Pollefeys. Hyperpoints and fine vocabularies for large-scale location recognition. In ICCV, 2015.
  • [19] T. Sattler, M. Havlena, K. Schindler, and M. Pollefeys. Large-scale location recognition and the geometric burstiness problem. In CVPR, 2016.
  • [20] T. Sattler, B. Leibe, and L. Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization. TPAMI, 2016.
  • [21] T. Sattler, A. Torii, J. Sivic, M. Pollefeys, H. Taira, M. Okutomi, and T. Pajdla. Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization? In CVPR, 2017.
  • [22] G. Schindler, M. Brown, and R. Szeliski. City-scale location recognition. In CVPR, 2007.
  • [23] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In CVPR, 2013.
  • [24] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 2014.
  • [25] L. Svarm, O. Enqvist, M. Oskarsson, and F. Kahl. Accurate localization and pose estimation for large 3D models. In CVPR, 2014.
  • [26] P. H. S. Torr and A. Zisserman. MLESAC: A new robust estimator with application to estimating image geometry. CVIU, 2000.
  • [27] J. Valentin, A. Dai, M. Nießner, P. Kohli, P. Torr, S. Izadi, and C. Keskin. Learning to navigate the energy landscape. CoRR, 2016.
  • [28] J. Valentin, M. Nießner, J. Shotton, A. Fitzgibbon, S. Izadi, and P. H. S. Torr. Exploiting uncertainty in regression forests for accurate camera relocalization. In CVPR, 2015.
  • [29] F. Walch, C. Hazirbas, L. Leal-Taixé, T. Sattler, S. Hilsenbeck, and D. Cremers. Image-based localization with spatial LSTMs. In ICCV, 2017.