In the last years a lot of research has been done on improving computer vision and LiDAR systems. Applications range from autonomous cars to unmanned drones. One of the main areas of development is mapping and localization. Mainly two sensors are used for this application: LiDARs, which provide highly accurate distance measurements but are still rather expensive, and cameras, which are cheaper but they lack accuracy when used for mapping. In this work we propose a step towards a new concept of localization, namely finding a common representation of LiDAR and camera data to be used with standard computer vision algorithms. This common representation provides a useful application in swarm robotics like swarm drones or fleets of autonomous cars. The idea is the following: A mapping device is equipped with one highly accurate LiDAR sensor and utilized to create a precise and consistent 3D model of the environment, the map. The swarm robots are equipped with an affordable, lightweight camera. The images taken by the camera can be transformed into the proposed common representation, together with nearby point cloud data of the map. Next, a standard 2D-2D feature matching is performed using the common representations. The 3D positions of the matched features are retrieved from the stored point cloud based map data, which was originally used to generate one of the common representations. Using this data, the 6DoF pose of the robot can be estimated. Our contribution to this concept are two methods for finding a common representation of image and point cloud data in the form of a 2D image. The process of finding this common representation is learning based and unsupervised. Figure1 provides a high-level overview of the system. Figure 2 shows the used input data. A point cloud created by multiple LiDAR sensors is projected onto the corresponding image of a forward facing camera with a fisheye lens.
2 Related Work
Finding an abstract representation of images is an extensively studied topic in the field of computer vision and deep learning. A very common approach to this problem is the utilization of convolutional layers combined in an encoder-decoder network . The encoder aims to learn an encoding of the input in a high dimensional hyperspace, while the decoder ensures that this code contains all relevant information to generate the desired output image. The simplest example are auto-encoders, which reconstruct the input image from the intermediate code. Those networks are often used as the backbone architecture of more complex structures . By constraining the learned encoding, variational auto-encoders (VAE) are restricted to include certain information in the low level representation . Another approach to enforce a specific representation of the output leverages the feedback loop in generative adversarial networks (GANs). A GAN is composed of two subnetworks: while the generator creates artificial data, the discriminator tries to differentiate them from another data set that represents the target domain. Recent research based on GANs has shown astonishing results when converting images from one space to another space in an unsupervised fashion . In both  and  multiple encoding-decoding networks are coupled to transfer images between several domains where no trivial relations exist. In order to find a common structure between data taken from multiple modalities, it is important to guide the network during the training process by providing both similar and non-similar pairs of the respective domains. Siamese networks consist of two networks that share all weights and that are trained simultaneously. A final cost layer computes the similarity score between the output of the different input streams. In  Siamese networks are used to identify persons in images taken by different cameras and across time. Another field of application is image matching when dealing with changing weather conditions or diverse viewpoints . An example of combining the aforementioned work is presented in . Using output data from a VAE as first input for a GAN’s discriminator and having a traditional auto-encoder as the second stream, the encoding learned by the auto-encoder is constrained to contain the desired information. This overcomes the problem of handcrafted features based on a pixel level. However, this approach cannot be directly applied to data from multiple modalities. In SuperPoint  the idea of Siamese training is applied to find keypoint correspondences between images in an completely unsupervised fashion. Classical computer vision methods, e.g., SIFT features, fall short behind the capabilities of SuperPoint. Even though the visual results are very promising, the method requires the computational power of recent GPU models, which conflicts with equipping low-cost swarm robots.
In this section two approaches for finding a common representation (CR) between camera and LiDAR data are proposed. The first approach (section 3.1) is purely unsupervised and utilizes a double Siamese network architecture. The second approach (section 3.2) tries to find a CR that resembles an edge image, to facilitate the learning process. Lastly, in section 3.3 the network layers utilized in the proposed architectures are explained.
3.1 Architecture 1 - Double Siamese Networks
Figure 3 shows the proposed architecture to learn a CR of image and point cloud data in a purely unsupervised manner. The framework builds on ideas from Siamese learning and encoder-decoder architectures, which are commonly used in unsupervised approaches. The Siamese learning approach ensures that the results are consistent, i.e., changes of the input result in similar changes of the output. Firstly, two images (image 1 and image 2) are sampled from the database and used to compute a difference score. This metric describes the similarity between both images and returns a high value for samples of the same scene of slightly different viewpoints. The Chebyshev distance 
is utilized as it provides the desired behavior. During the sampling process, a tuning parameter is used as the probability whether to randomly sample from a set of similar or non-similar images. Secondly, the system fetches the corresponding depth maps (depth map 1 and depth map 2) belonging to image 1 and image 2, respectively. The depth maps are computed by projecting the point cloud data onto the same image plane as given by the corresponding images. The pixels encode the depth measurement of the respective point. Finally, image 1 and image 2 are fed into the image CR generator, whereas depth map 1 and depth map 2 are fed into the depth map CR generator. Both generators are designed in an encoding-decoding fashion, which is explained in section3.3. The four forward passes through the subnetworks generate four grayscale output images representing the respective CR of the input pair 1 and input pair 2. The double Siamese networks are trained using the following loss :
where and are the CRs generated from the first and second image, respectively. and are the CRs generated from the first and second depth map, respectively. is the Chebyshev distance of images 1 and 2, and the norm
can theoretically be any matrix/image comparison norm. To this end the Manhattan distance is utilized. The intuition behind this formulation of the loss function is the following: The loss should be small when the CR computed from the image and the depth map is similar and, additionally, one of the two following conditions holds: either when the Siamese pairs are similar, i.e., a small difference score, and the CR of the respective Siamese pairs are similar, or when the Siamese pairs are not similar, i.e., large difference score, and the CR of the respective Siamese pairs differ a lot. Similarly, the loss should be large in the respective inverse cases.
3.2 Architecture 2 - Common Edges
Our second approach of finding a common representation utilizes a Canny edge image in the loss function to solve the consistency problem. Figure 4 shows the proposed architecture of this approach. The input, which is randomly fetched from a database, consists of a grayscale image and the corresponding depth map. The Canny edge algorithm is utilized to generate an edge image based on the grayscale input image. The inputs are fed through two separate encoder-decoder networks, which are explained in more detail in section 3.3. The following loss function is utilized:
where and are the CRs generated from the image and the depth map, respectively, and resembles the edge image. The norm operator is implemented as the Manhattan distance, but other matrix/image comparison norms could work as well. This approach ensures that the generated CRs are consistent and resemble an edge image. Therefore, a change in the input image leads to a similar change in the output image.
3.3 Encoder-Decoder Layers
. The input into this network could be any image that fits the input dimension. For our purposes, the input is either an image or a depth map, and the output is a 2D image, the CR. The layers are chosen such that the network learns a higher dimensional representation of the input data in some latent space by extracting the relevant features in an unsupervised fashion. Since the CR should preserve local information with respect to the input data, the network consists only of convolutional layers. After each convoulutional layer, a max pooling operation reduces the first two dimensions by a factor of two.
The proposed architectures are trained on the UP-Drive data set . The point cloud data is projected onto a 2D plane and utilized as depth image. The RGB image coming from the front facing camera is converted to grayscale and scaled to a feasible size. The dimensions are 320x160 pixels for the Double Siamese Networks and 640x320 pixels for the Common Edges method, facilitating the training process.
4.1 Architecture 1 - Double Siamese Networks
Figure 6 shows the results generated by the Double Siamese Networks. It can be seen that the CR is very similar when generated from either the grayscale image or depth image, which is what was one of the goals of this work. On the other hand, it is is not very descriptive and detailed anymore. A lot of information is lost in the compression process of the encoder-decoder networks and due to the large domain gap between the two modalities. To demonstrate the usability of the CR, two standard feature matching methods are applied. The results are averaged over 100 images from the test data set. Figure 7 shows the result when using ORB and SIFT features for matching. The left image pairs are matched using ORB, the right image pairs using SIFT. Of each pair, the data is taken of the same scene from different time instances resulting in small variations in the point of view. The most significant result is the reprojection error given in table 1
. It is significantly higher for the CR in both cases (ORB and SIFT), but still in an acceptable range. The loss of detail in the CR can be quantified when counting the number of matches that the algorithms find. Normal images achieve 10 times as many matches as the CR. On the other hand, the quality of the matches is surprisingly good. In fact, the average L2 distance of the matched descriptor vectors is even lower for the CRs, demonstrating the strength of the matches.
|Matching inputs||SIFT avg. dist.||SIFT avg. matches||SIFT avg. reprojection error (pixels)|
|image vs. image||1||148||0.65|
|image CR vs. depth map CR||0.67||13||1.46|
|Matching inputs||ORB avg. dist.||ORB avg. matches||ORB avg. reprojection error (pixels)|
|image vs. image||1||270||1.49|
|image CR vs. depth map CR||0.81||23||3.68|
4.2 Architecture 2 - Common Edges
The results of the Common Edges approach are depicted in Figure 8. The CR generated from the grayscale image looks similar to an edge image, which agrees with intuition, as the edge image has been generated from the input grayscale image in the first place. The CR generated from the depth image is blurry, which does not resemble how an edge image looks like. More details on this behavior and potential solutions are presented in section 5. In general, the results of this architecture are too inconsistent to be used for any matching algorithms. A quantitative evaluation is therefore omitted.
The results of the Double Siamese Networks show that there is a lack of detail in the common representation, limiting the usability for matching operations. Still, the evaluations using ORB and SIFT matches show that the quality of the matches is not too far off the quality of purely image based matches. The Common Edges architecture does not perform well at all since the CR generated from the depth image is very blurry. There are two potential causes for this issue. Firstly, when utilizing encoder-decoder architectures, it is a common problem that the results look blurry. This can potentially be resolved under utilization of GANs, as proposed in . Secondly, there is a conceptual flaw in this approach. Namely, many objects have edges in the sense of a Canny edge detector, e.g., red brick buildings, which a depth sensor is not capable to capture. The reduction of information when generating a common representation seems to be unavoidable when utilizing an architecture as proposed in this work. In order to find a representation that retains more information, it is necessary to incorporate more information in either of the pipelines. For instance, one could perform monocular depth estimation from images to reduce the domain gap between camera and LiDAR data. A novel approach for this is presented in . Another approach could include semantic information. On those lines it could be useful to apply object instance detection on both modalities to generate a higher-level representation of the scene. A common method for this is proposed in , called Mask R-CNN. The issues with approaches like this are that the generalizability of the whole system depends on the generalization capabilities of the individual parts. The training time and the data set size, which is necessary to still be able to generalize well, might not be feasible anymore.
In this work we have proposed two architectures to learn common representations of LiDAR and camera data, in the form of a 2D image. We find that for one of our approaches the results are not as detailed as initially hoped, but that they are still sufficient for basic feature matching algorithms. The Double Siamese Network architecture showed to be very effective in ensuring that the consistency and distinctiveness of the common representation is given. We have further shown the limitations of using just geometric data, and propose the incorporation of additional semantic information for future work.
-  (2017-12) SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (12), pp. 2481–2495. External Links: Cited by: §2.
StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. CoRR abs/1711.09020. External Links: Cited by: §2.
A two stream siamese convolutional neural network for person re-identification. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §2.
-  (2017) SuperPoint: self-supervised interest point detection and description. CoRR abs/1712.07629. External Links: Cited by: §2.
-  (2017) Countering adversarial images using input transformations. CoRR abs/1711.00117. External Links: Cited by: §3.1.
-  (2017) Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988. Cited by: §5.
-  (2017) Learning to discover cross-domain relations with generative adversarial networks. CoRR abs/1703.05192. External Links: Cited by: §2.
-  (2015) Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (Eds.), pp. 2539–2547. Cited by: §2.
-  (2015) Autoencoding beyond pixels using a learned similarity metric. CoRR abs/1512.09300. External Links: Cited by: §2.
Siamese network features for image matching.
2016 23rd International Conference on Pattern Recognition (ICPR), Vol. , pp. 378–383. External Links: Cited by: §2.
-  (2018) SuperDepth: self-supervised, super-resolved monocular depth estimation. CoRR abs/1810.01849. External Links: Cited by: §5.
Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions. CoRR abs/1807.06650. Cited by: §5.
-  (2017-10) Super-sensor for 360-degree environment perception: point cloud segmentation using image features. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Vol. , pp. 1–8. External Links: Cited by: §4.
Coupled deep autoencoder for single image super-resolution. IEEE Transactions on Cybernetics 47 (1), pp. 27–37. External Links: Cited by: §2.
-  (2017-10) Unpaired image-to-image translation using cycle-consistent adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §2.