Visual perception of 3D scene geometry using a monocular camera is a fundamental problem with numerous applications, including autonomous driving. We focus on the ability to infer accurate geometry (depth and motion) of static and moving objects in a 3D scene. Supervised deep learning models have been proposed for geometry predictions, yielding “robust” and favorable results against the traditional approaches (SfM)[38, 39, 10, 2, 1, 26]. However, supervised models require a data-set labeled with geometrically informative annotations, which is extremely challenging as the collection of geometrically annotated ground truth (e.g. depth, motion) requires expensive equipment (e.g. LIDAR) and careful calibration procedures.
Recent works combine the geometric-based SfM methods with end-to-end unsupervised trainable deep models to utilize abundantly available unlabeled monocular camera data. In [54, 41, 51, 9] deep models predict depth and motion per pixel simultaneously from a short sequence of images and typically use photo-metric reconstruction loss of a target scene from neighboring scenes as the surrogate task. However, these solutions fail often when dealing with dynamic objects111Section 5 presents empirical results that explicitly illustrate this shortcoming of state-of-the-art unsupervised approaches.. Furthermore, the prediction quality is negatively affected by the imperfections like dynamic objects, Lambertian reflectance, and varying intensity, all of these occur in the real world. In short, no robust solution is known.
In Fig 1, we highlight the challenges that unsupervised learning faces for dynamic objects. Unsupervised model learns the movement of every pixel in a sequence of images and by taking ego-motion into consideration produces a single feedback (i.e. photo-metric reconstruction loss) to make depth and motion predictions. Therefore, it aims to learn both depth and motion per pixel from a single feedback. SIGNet relies on the key observation that inherent spatial constraints exist in the visual perception problem as shown in Fig 1. Specifically, we exploit the fact that pixels belonging to the same object have additional inherent constraints on the depth and motion profile.
SIGNet encodes the spatial constraints to improve geometric perception without requiring geometrically informative labels. How can the spatial constraints of the pixels be encoded? We leverage the semantic information as seen in Fig 1 for unsupervised frameworks. Intuitively, semantic information can be interpreted as defining boundaries around a group of pixels whose geometry is closely related. The knowledge of semantic information between different segments of a scene could allow us to easily learn which pixels are correlated, while the object edges could imply sharp depth transition. Furthermore, note that this learning paradigm is practical 222Semantic labels can be easily curated on demand on unlabeled data. On the contrary, geometrically informative labels such as motion and depth require additional sensors and careful annotation at the data collection stage.
as annotations for semantic prediction tasks such as semantic segmentation are relatively cheaper and easier to acquire. To the best of our knowledge, our work is the first to utilize semantic information in the context of unsupervised learning for geometry prediction (depth and motion) estimation.
A natural question is how do we combine semantic information with an unsupervised geometric prediction? Our approach to combine the semantic information with RGB input is two-fold: First, we propose a novel way to augment RGB images with semantic information. Second, we propose new loss functions, architecture, and training method. The two-fold approach precisely accounts for spatial constraints in making geometric predictions:
Feature Augmentation: We concatenate the RGB input data with both per-pixel class predictions and instance-level predictions. We use per pixel class predictions to define semantic mask which serves as a guidance signal that eases unsupervised geometric predictions. Moreover, we use the instance-level prediction and split them into two inputs, instance edges and object masks. Instance edges and object masks enable the network to learn the object edges and sharp depth transitions.
Loss Function Augmentation: Second, we augment the loss function to include various semantic losses, which reduces the reliance on semantic features in the evaluation phase. This is crucial when the environment contains less common contextual elements (like in dessert navigation or mining exploitation). We design and experiment with various semantic losses, such as semantic warp loss, masked reconstruction loss, and semantic-aware edge smoothness loss. However, manually designing a loss term which can improve the performance over the feature augmentation technique turns out to be very difficult. The challenge comes from the lack of understanding of error distributions because we are generally biased towards simple, interpretable loss functions that can be sub-optimal in unsupervised learning. Hence, we propose an alternative approach of incorporating a transfer network that learns how to predict semantic mask via a semantic reconstruction loss and provides feedback to improve the depth and pose estimations, which shows considerable improvements in depth and flow prediction.
We empirically evaluate the feature and loss function augmentations on KITTI dataset  and compare them with the state-of-the-art unsupervised learning framework . In our experiments we use class-level predictions from DeepLabv3+  trained on Cityscapes  and Mask R-CNN  trained on MSCOCO . Our key findings:
By using semantic segmentation for both feature and loss augmentation, our proposed algorithms improves squared relative error in depth estimation by % compared to the strong baseline set by state-of-the-art unsupervised GeoNet .
Feature augmentation alone, combining semantic with instance-level information, leads to larger gains. With both class-level and instance-level features, the squared relative error of the depth predictions improves by % compared to the baseline.
Finally, as for common dynamic object classes (e.g. vehicles) SIGNet shows % improvement (in squared relative error) for depth predictions and % improvement in the flow prediction, thereby showing that semantic information is very useful for improving the performance in the dynamic categories of objects. Furthermore, SIGNet is robust to noise in image intensity compared to the baseline.
2 Related Work
Deep Models for Understanding Geometry: Deep models have been widely used in supervised depth estimation [8, 29, 36, 53, 5, 49, 50, 11, 46], tracking, and pose estimation [43, 47, 2, 17] , as well as optical flow predictions [7, 20, 25, 40]. These models have demonstrated superior accuracy and typically faster speed in modern hardware platforms (especially in the case of optical flow estimation) compared to traditional methods. However, achieving good performance with supervised learning requires a large amount of geometry-related labels. The current work addresses this challenge by adopting an unsupervised learning framework for depth, pose, and optical flow estimations.
Deep Models for Semantic Predictions: Deep models are widely applied in semantic prediction tasks, such as image classification , semantic segmentation , and instance segmentation . In this work, we utilize the effectiveness of the semantic predictions provided by DeepLab v3+  and Mask R-CNN  in encoding spatial constraints to accurately predict geometric attributes such as depth and flow. We note that while we particularly, utilize DeepLab v3+  and Mask R-CNN , similar gains can be obtained by using any state-of-the-art semantic prediction system.
Unsupervised Deep Models for Understanding Geometry: Several recent methods propose to use unsupervised learning for geometry understanding. In particular, Garg et al.  uses a warping method based on Taylor expansion. In the context of unsupervised flow prediction, Yu et al.  and Ren et al.  introduce image reconstruction loss with spatial smoothness constraints. Similar methods are used in Zhou et al.  for learning depth and camera ego-motions by ignoring object motions. This is partially addressed by Vijayanarasimhan et al. , despite the fact, we note, that the modeling of motion is difficult without introducing semantic information. This framework is further improved with better modeling of the geometry. Geometric consistency loss is introduced to handle occluded regions, in binocular depth learning , flow prediction  and joint depth, ego-motion and optical flow learning . Mahjourian et al.  focuses on improved geometric constraints, Godard et al.  proposes several architectural and loss innovations, while Zhan et al.  uses reconstruction in the feature space rather than the image space. In contrast, the current work explores using semantic information to resolve ambiguities that are difficult for pure geometric modeling. Methods proposed in the current work are complementary to these recent methods, but we choose to validate our approach on the state-of-the-art GeoNet framework .
Multi-Task Learning for Semantic and Depth: Multi-task learning  achieves better generalization by allowing the system to learn features that are robust across different tasks. Recent methods focus on designing efficient architectures that can predict related tasks using shared features while avoiding negative transfers [35, 19, 30, 34, 23, 12]. In this context, several prior works report promising results combining scene geometry with semantics. For instance, similar to our method Liu et al.  uses semantic predictions to provide depth. However, this work is fully supervised and only uses sub-optimal traditional methods. Wang et al. , Cross-Stitching , UberNet  and NDDR-CNN 
all report improved performance over single-task baselines. But they did not address outdoor scenes and unsupervised geometry understanding. Our work is also related to PAD-Net. PAD-Net reports improvements by combining intermediate tasks as inputs to final depth and segmentation tasks. Our method of using semantic input similarly introduces an intermediate prediction task as input to the depth and pose predictions, but we tackle the problem setting where depth labels are not provided.
3 State-of-the-art Unsupervised Geometry Prediction
Prior to presenting our technical approach, we provide a brief overview of state-of-the-art unsupervised depth and motion estimation framework, which is based on image reconstruction from geometric predictions [54, 51]. It trains the geometric prediction models through the reconstructions of a target image from source images. The target and source images are neighboring frames in a video sequence. Note that such a reconstruction is possible only when certain elements of the 3D geometry of the scene are understood: (1) The relative 3D location (and thus the distance) between the camera and each pixel. (2) The camera ego-motion. (3) The motion of pixels. Thus this framework can be used to train a depth estimator and an ego-motion estimator, as well as a optical flow predictor.
Technically, each training sample consists of contiguous video frames where the center frame is the “target frame” and the other frames serve as the “source frame”. In training, a differentiable warping function is constructed from the geometry predictions. The warping function is used to reconstruct the target frame from source frame
via bilinear sampling. The level of success in this reconstruction provides training signals through backpropagation to the various ConvNets in the system. A standard loss function to measure reconstruction success is as follows:
where SSIM denotes the structural similarity index  and is typically set to .
To filter out erroneous predictions while preserving sharp details, the standard practice is to include an edge-aware depth smoothness loss weighted by image gradients
where denotes element-wise absolute operation,
is the vector differential operator, anddenotes transpose of gradients. These losses are usually computed from a pyramid of multi-scale predictions. The sum is used as the training target.
While the reconstruction of RGB images is an effective surrogate task for unsupervised learning, it is limited by the lack of semantic information as supervision signals. For example, the system cannot learn the difference between the car and the road if they have similar colors or two neighboring cars with similar colors. When object motion is considered in the models, the learning can mistakenly assign motion to non-moving objects as the geometric constraints are ill-posed. We augment and improve this system by leveraging semantic information.
In this section, we present solutions to enhance geometry predictions with semantic information. Semantic labels can provide rich information on 3D scene geometry. Important details such as 3D location of pixels and their movements can be inferred from a dense representation of the scene semantics. The proposed methods are applicable to a wide variety of recently proposed unsupervised geometry learning frameworks based on photometric reconstruction [54, 16, 51] represented by our baseline framework introduced in Section 3. Our complemented pipeline in test time is illustrated in Fig 2.
4.1 Semantic Input Augmentation
Semantic predictions can improve geometry prediction models by being used as input features. Unlike RGB images, semantic predictions mark objects and contiguous structures with consistent blobs, which provide important information for the learning problem. However, it is uncertain that using semantic labels as input could indeed improve depth and motion predictions since training labels are not available. Semantic information could be lost or distorted, which would end up being a noisy training signal. An important finding of our work is using semantic prediction as inputs significantly improves the accuracy in geometry prediction, despite the presence of noisy training signal. Input representation and the type of semantic labels have a large impact on the performance of the system. We further illustrate this by Fig 3, where we show various semantic labels (semantic segmentation, instance segmentation, and instance edge) that we use to augment the input. This imposes additional constraint such as depth of the pixels belonging to a particular object (e.g. a vehicle) which helps the learning process. Furthermore, sudden changes in the depth predictions can be inferred from the boundary of vehicles. The semantic labels of the pixels can provide important information to associate pixels across frames.
Encoding Pixel-wise Class Labels:
We explored two input encoding techniques for class labels: dense encoding and one-hot encoding. In dense encoding, dense class labels are concatenated along the input channel dimension. The added semantic features are centralized to the range ofto be consistent with RGB inputs. In the case of one-hot encoding, the class-level semantic predictions are first expanded to one-hot encoding and then concatenated along the input channel dimension. The labels are represented as one-hot sparse vectors. In this variant, semantic features are not normalized since they have similar value range as the RGB inputs,
Encoding Instance-level Semantic Information: Both dense and one-hot encoding are natural to class-level semantic prediction, where each pixel is only assigned a class label rather than an instance label. Our conjecture is that instance-level semantic information is particular well-suited to improve unsupervised geometric predictions, as it provides accurate information on the boundary between individual objects of the same type. Unlike class-level label, instance label in itself does not have well-defined meaning. Across different frames, the same label could refer to different object instances. To efficiently represent the instance-level information, we compute the gradient map of a dense instance map and use it as an additional feature channel, along the class label input (dense or one-hot encoding).
Direct Input versus Residual Correction: Complementary to the choice of encoding, we also experiment with different methods to input semantic information to the geometry prediction models. In particular, we make a residual prediction using a separate branch that takes in only semantic inputs. Notably, using residual depth prediction leads to further improvement on top of the gains from direct input methods.
4.2 Semantic Guided Loss Functions
The information from semantic predictions could be diminished due to noisy semantic labels and very deep architectures. Hence, we design training loss functions that are guided by semantic information. In such design, the semantic predictions provide additional loss constraints to the network. In this subsection, we introduce a set of semantic guided loss functions designed to improve the learning of depth and motion prediction.
Semantic Warp Loss: Semantic predictions can help learn scenarios where reconstruction of the RGB image is correct in terms of pixel values but violates obvious semantic correspondences, e.g. matching pixels to incorrect semantic classes and/or instances. In light of this, we propose to reconstruct the semantic predictions in addition of doing so for RGB images. We call this “semantic warping loss” as it is based on warping of the semantic predictions from source frames to the target frame. Let be the source frame semantic prediction and be the warped semantic image, we define semantic warp loss as:
The warped loss is added to the baseline framework using a hyper-tuned value of the weight .
Masking of Reconstruction Loss via Semantics: As described in Section 3, ambiguity in object motion can lead to sub-optimal learning. Semantic labels can partially resolve this ambiguity by separating non-moving objects. Motivated by this observation, we mask the foreground region out to form a set of new images where represents the RGB-channel index, is the element-wise multiplication for matrix and is the -th channel of the binary semantic segmentation result. Similarly we can obtain
. The image similarity loss is defined as:
Semantic-Aware Edge Smoothness Loss: Equation 2 uses RGB to infer edge locations when enforcing smooth regions of depth. This could be improved by including an edge map computed from semantic predictions. Given a semantic segmentation result , we define a weight matrix where the weight is low (close to zero) on class boundary regions and high (close to one) on other regions. We propose a new image similarity loss as:
Semantic Loss by Transfer Network: Motivated by the observation that high-quality depth maps usually depict object classes and background region, we designed a novel transfer network architecture. As shown in Fig 4 the transfer network block receives predicted depth maps along with the original RGB images and outputs semantic labels. The transfer network introduces a semantic reconstruction loss term to the objective function to force the predicted depth maps to be richer in contextual sense, hence refine the depth estimation. For implementation, we choose the ResNet50 as the backbone and alter the dimensions for the input and output convolutional layers to be consistent with the segmentation task. The network generates one-hot encoded heatmaps and use cross-entropy as the semantic similarity measure.
To quantify the benefits that semantic information brings to geometry-based learning, we designed experiments based on GeoNet framework . First, we showed our model’s depth prediction performance on KITTI dataset , which outperformed state-of-the-art unsupervised and supervised models. Then we designed ablation studies to analyze each individual component’s contribution. Finally, we presented improvements in motion predictions and revisited the performance gain using category-specific evaluation.
5.1 Implementation Details
To make a fair comparison with state-of-the-art models [8, 54, 51], we divided KITTI 2015 dataset into train set (40238 images) and test set (697 images) according to split rules from Eigen et al . We utilized DeepLabv3+  to obtain semantic segmentation results and Mask-RCNN  for instance-level segmentation. Similar to the hyper-parameter settings in , we used Adam optimizer  with initial learning rate as 2e-4, set batch size to 4 and trained our modified DepthNet and PoseNet modules for 250000 iterations with random shuffling and data augmentation. The training took 10 hours on two GTX1080Ti.
5.2 Monocular Depth Evaluation on KITTI
We augmented the image sequences with corresponding semantic and instance segmentation sequences and adopted the scale normalization suggested in . In the evaluation stage, the ground truth depth maps were generated by projecting 3D Velodyne LiDAR points to the image plane. Followed by , we clipped our depth predictions within 0.001m to 80m and calibrated the scale by the medium number of the ground truth. The evaluation results are shown in Table 1, where all the metrics are introduced in . Our model benefits significantly from feature augmentation and surpasses the state-of-the-art methods substantially in both supervised and unsupervised fields.
Moreover, we found a correlation between the improvement region and object classes. We visualized the absolute relative error (AbsRel) among image plane from our model and from the baseline. As shown in Fig 5, most of the improvements come from regions containing objects. This indicates that the network is able to learn the concept of objects to improve the depth prediction by rendering extra semantic information.
|Method||Supervised||Error-related metrics||Accuracy-related metrics|
|Abs Rel||Sq Rel||RSME||RSME log|
|Eigen et al.  Coarse||Depth||0.214||1.605||6.653||0.292||0.673||0.884||0.957|
|Eigen et al.  Fine||Depth||0.203||1.548||6.307||0.282||0.702||0.890||0.957|
|Liu et al. ||Depth||0.202||1.614||6.523||0.275||0.678||0.895||0.965|
|Godard et al. ||Pose||0.148||1.344||5.927||0.247||0.803||0.922||0.964|
|Zhou et al.  updated||No||0.183||1.595||6.709||0.270||0.734||0.902||0.959|
|Yin et al. ||No||0.155||1.296||5.857||0.233||0.793||0.931||0.973|
5.3 Ablation Studies
Here we took a deeper look of our model, testified its robustness under noise from observations, and presented variations of our framework to show promising explorations for future researchers. In the following experiments, we kept all the other parameters the same in  and applied the same training/evaluation strategies mentioned in Section 5.2
, our first conclusion is that any meaningful form of extra inputs can ameliorate the model, which is straightforward. Secondly, when we use “Semantic” and “Instance class” for feature augmentation, one-hot encoded form tends to behave better than dense map form. Conceivably one-hot encoding stores richer information in its structural formation, whereas dense map only contains discrete labels which may be more difficult for learning. Moreover, using both “Semantic” and “Instance class” can provide further gain, possibly due to the different label distributions from two datasets. Labels from Cityscape cover both background and foreground concepts, while the COCO dataset focuses more on objects. At last, when we combined one-hot encoded “Semantic” and “Instance class” information along with “Instance id” edge features, the network exploited the most from scene understanding hence greatly enhanced the performance.
|Semantic||Instance||Instance||Error-related metrics||Accuracy-related metrics|
|class||id||Abs Rel||Sq Rel||RSME||RSME log|
Can our model handle noise?
To testify our model’s robustness for varied lighting condition, we multiplied a scalar between 0 and 1 to RGB inputs in the evaluation. Fig 6 showed that our model still holds equal performance to GeoNet when the intensity decrease to %.
Which module needs extra information the most?
We fed semantics to only DepthNet or PoseNet to see the difference in their performance gain. From Table 3 we can see that compared to DepthNet, PoseNet learns little from the semantics to help depth prediction. Therefore we tried to feed the semantics to a new PoseNet with the same structure as the original one and compute the predicted poses by taking the sum from two different PoseNets, which led to performance gain; however, performance gain was not observed from applying the same method to DepthNet.
|DepthNet||PoseNet||Error-related metrics||Accuracy-related metrics|
|Abs Rel||Sq Rel||RSME||RSME log|
How to be “semantic-free” in evaluation?
Though semantic helps depth prediction, this idea relies on semantic features during the evaluation phase. If semantic is only utilized in the loss, it would not be needed in evaluation. We attempted to introduce a handcrafted semantic loss term as a weight guidance among image plane but it didn’t work well. Also we designed a transfer network which uses the predicted depth to predict semantic maps along with a reconstruction error to help in the training stage. The result in Table 4 shows a better result can be obtained by training from pretrained models.
|Checkpoint||Transfer||Error-related metrics||Accuracy-related metrics|
|Network||Abs Rel||Sq Rel||RSME||RSME log|
|Yin et al. ||0.155||1.296||5.857||0.233||0.793||0.931||0.973|
|Yin et al. ||Yes||0.150||1.141||5.709||0.231||0.792||0.934||0.974|
|Yin et al.  +sn||0.149||1.060||5.567||0.226||0.796||0.935||0.975|
|Yin et al.  +sn||Yes||0.145||0.994||5.422||0.222||0.806||0.939||0.976|
5.4 Optical Flow Estimation on KITTI
Using our best model for DepthNet and PoseNet in Section 5.2, we conducted rigid flow and full flow evaluation on KITTI . We first generated the rigid flow from estimated depth and pose, which were compared with GeoNet . Our model performed better in all the metrics shown in Table 5.
|Method||End Point Error||Accuracy|
|Yin et al. ||23.5683||29.2295||0.2345||0.2237|
We appended the semantic warping loss introduced in Section 4.2 to ResFlowNet in  and trained our model on KITTI stereo for 1600000 iterations. As demonstrated in Table 6, flow prediction improved in non-occluded region compared to GeoNet  and produced comparable results in overall regions.
5.5 Category-Specific Metrics Evaluation
This section will present the improvements by semantic categories. As shown in the bar-chart in Fig 7, most improvements were shown in “Vehicle” and “Dynamic” classes, where errors are generally large. Our network did not improve much for other less frequent categories, such as “Motorcycle”, which are generally more difficult to segment in images.
In SIGNet, we strive to achieve robust performance for geometry perception (depth and motion) without using any geometric labels. To achieve this goal, SIGNet utilizes the well-known supervised learning semantic segmentation frameworks to create spatial constraints on the geometric attributes of the pixels. We present novel methods of feature augmentation and loss augmentation to include semantic labels in the geometry predictions. This work presents a first of a kind approach which moves away from pixel-level to object-level depth and motion prediction tasks. Most notably, this approach significantly improves the state-of-the-art solution for estimating the depth and motion of dynamic objects.
Supplementary Material for SIGNet
Here we present extra visualization results to help readers understand where our semantic-aided model improved the most. We compared the prediction result from our best model in Table 1 with Yin et al.  and ground truth. We followed  to plot the prediction result using disparity heatmaps. The following results show that our model can gain improvement from regions belonging to cars and other dynamic classes.
-  M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and A. J. Davison. Codeslam-learning a compact, optimisable representation for dense visual slam. arXiv preprint arXiv:1804.00874, 2018.
A. Byravan and D. Fox.
Se3-nets: Learning rigid body motion using deep neural networks.In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 173–180. IEEE, 2017.
-  R. Caruana. Multitask learning. Mach. Learn., 28(1):41–75, July 1997.
-  L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv preprint arXiv:1802.02611, 2018.
-  W. Chen, Z. Fu, D. Yang, and J. Deng. Single-image depth perception in the wild. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 730–738, USA, 2016. Curran Associates Inc.
-  M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In , pages 3213–3223, 2016.
-  A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
-  D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems, pages 2366–2374, 2014.
-  X. Fei, A. Wang, and S. Soatto. Geo-supervised visual depth prediction. arXiv preprint arXiv:1807.11130, 2018.
-  P. Fischer, A. Dosovitskiy, E. Ilg, P. Häusser, C. Hazırbaş, V. Golkov, P. Van der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. arXiv preprint arXiv:1504.06852, 2015.
-  H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao. Deep ordinal regression network for monocular depth estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  Y. Gao, Q. She, J. Ma, M. Zhao, W. Liu, and A. L. Yuille. Nddr-cnn: Layer-wise feature fusing in multi-task cnn by neural discriminative dimensionality reduction. CoRR, abs/1801.08297, 2018.
-  R. Garg, G. VijayKumarB., and I. D. Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue. In ECCV, 2016.
-  A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3354–3361. IEEE, 2012.
-  C. Godard, O. M. Aodha, and G. J. Brostow. Digging into self-supervised monocular depth estimation. CoRR, abs/1806.01260, 2018.
-  C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, volume 2, page 7, 2017.
-  D. Gordon, A. Farhadi, and D. Fox. Re : Real-time recurrent regression networks for visual tracking of generic objects. IEEE Robotics and Automation Letters, 3(2):788–795, 2018.
-  K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980–2988. IEEE, 2017.
-  K. He, Z. Wang, Y. Fu, R. Feng, Y.-G. Jiang, and X. Xue. Adaptively weighted multi-task deep network for person attribute classification. In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, pages 1636–1644, New York, NY, USA, 2017. ACM.
-  E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  J. Y. Jason, A. W. Harley, and K. G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In European Conference on Computer Vision, pages 3–10. Springer, 2016.
-  D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory.In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5454–5463, July 2017.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84–90, May 2017.
-  W.-S. Lai, J.-B. Huang, and M.-H. Yang. Semi-supervised learning for optical flow with generative adversarial networks. In NIPS, 2017.
-  K.-N. Lianos, J. L. Schönberger, M. Pollefeys, and T. Sattler. Vso: Visual semantic odometry. In Proceedings of the European Conference on Computer Vision (ECCV), pages 234–250, 2018.
-  T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
-  B. Liu, S. Gould, and D. Koller. Single image depth estimation from predicted semantic labels. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1253–1260, 2010.
-  F. Liu, C. Shen, G. Lin, and I. D. Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans. Pattern Anal. Mach. Intell., 38(10):2024–2039, 2016.
-  Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  R. Mahjourian, M. Wicke, and A. Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  S. Meister, J. Hur, and S. Roth. UnFlow: Unsupervised learning of optical flow with a bidirectional census loss. In AAAI, New Orleans, Louisiana, Feb. 2018.
-  M. Menze and A. Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3061–3070, 2015.
-  E. Meyerson and R. Miikkulainen. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. CoRR, abs/1711.00108, 2017.
-  I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994–4003, 2016.
-  N.Mayer, E.Ilg, P.Häusser, P.Fischer, D.Cremers, A.Dosovitskiy, and T.Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016. arXiv:1512.02134.
-  Z. Ren, J. Yan, B. Ni, B. Liu, X. Yang, and H. Zha. Unsupervised deep learning for optical flow estimation. In AAAI, volume 3, page 7, 2017.
-  A. Saxena, S. H. Chung, and A. Y. Ng. Learning depth from single monocular images. In Advances in neural information processing systems, pages 1161–1168, 2006.
-  A. Saxena, M. Sun, and A. Y. Ng. Make3d: Learning 3d scene structure from a single still image. IEEE transactions on pattern analysis and machine intelligence, 31(5):824–840, 2009.
-  D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint arXiv:1704.07804, 2017.
-  C. Wang, J. M. Buenaposada, R. Zhu, and S. Lucey. Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2022–2030, 2018.
-  L. Wang, W. Ouyang, X. Wang, and H. Lu. Visual tracking with fully convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 3119–3127, 2015.
-  P. Wang, X. Shen, Z. Lin, S. Cohen, B. Price, and A. L. Yuille. Towards unified depth and semantic prediction from a single image. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
-  Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
-  K. Xian, C. Shen, Z. Cao, H. Lu, Y. Xiao, R. Li, and Z. Luo. Monocular relative depth perception with web stereo data supervision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017.
-  D. Xu, W. Ouyang, X. Wang, and N. Sebe. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. CoRR, abs/1805.04409, 2018.
-  D. Xu, E. Ricci, W. Ouyang, X. Wang, and N. Sebe. Multi-scale continuous crfs as sequential deep networks for monocular depth estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  D. Xu, W. Wang, H. Tang, H. Liu, N. Sebe, and E. Ricci. Structured attention guided convolutional neural fields for monocular depth estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  Z. Yin and J. Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2018.
H. Zhan, R. Garg, C. Saroj Weerasekera, K. Li, H. Agarwal, and I. Reid.
Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  Z. Zhang, A. G. Schwing, S. Fidler, and R. Urtasun. Monocular object instance segmentation and depth ordering with cnns. 2015 IEEE International Conference on Computer Vision (ICCV), pages 2614–2622, 2015.
-  T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, volume 2, page 7, 2017.