One of the important challenges in computer vision applications is obtaining high resolution depth maps of observed scenes. A number of common tasks, such as object reconstruction, robotic navigation, and automotive driver assistance can be significantly improved by complementing intensity information from optical cameras with high resolution depth maps. However, with current sensor technology, direct acquisition of high-resolution depth maps is very expensive.
The cost and limited availability of such sensors imposes significant constraints on the capabilities of vision systems and has dampened the adoption of methods that rely on high-resolution depth maps. Thus, the literature has flourished with methods that provide numerical alternatives to boost the spatial resolution of the measured depth data.
One of the most popular and widely investigated class of techniques for improving the spatial resolution of depth is guided depth superresolution. These techniques jointly acquire the scene using a low-resolution depth sensor and a high-resolution optical camera. The information acquired from the camera is subsequently used to superresolve the low-resolution depth map. These techniques exploit the property that both modalities share common features, such as edges and joint texture changes. Thus, such features in the optical camera data provide information and guidance that significantly enhances the superresolved depth map.
To-date, most of these methods operate on a single snapshot of the optical image and the low-resolution depth map. However, most practical uses of such systems acquire a video from the optical camera and a sequence of snapshots of the depth map. The key insight in our paper is that information about one particular frame is replicated, in some form, in nearby frames. Thus, frames across time can be exploited to superresolve the depth map and significantly improve such methods. The challenge is finding this information in the presence of scene, camera, and object motion between frames. Figure 1 provides an example, illustrating the similarity of images and depth maps across frames.
A key challenge in incorporating time into depth estimation is that depth images change significantly between frames. This results in abrupt variations in pixel values along the temporal dimension and may lead to significant degradation in the quality of the result. Thus, it is important to compensate for motion during estimation. To that end, the method we propose exploits space-time similarities in the data using motion adaptive regularization. Specifically, we identify and group similar depth patches, which we superresolve and regularize using a rank penalty.
Our method builds upon prior work on patch-based methods and low-rank regularization, which were successfully applied to a variety of practical estimation problems. It further exploits the availability of optical images which provide a very robust guide to identify and group similar patches, even if the depth map has very low resolution. Thus, the output of our iterative algorithms is robust to operating conditions. Our key contributions are summarized as follows:
We provide a new formulation for guided depth superresolution, incorporating temporal information. In this formulation, the high resolution depth is determined by solving an inverse problem that minimizes a cost. This cost includes a quadratic data-fidelity term, as well as a new motion adaptive regularizer based on a low-rank penalty on groups of similar patches.
We develop two optimization strategies for solving our estimation problem. The first approach is based on exact optimization of the cost via alternating direction method of multipliers (ADMM). The second approach uses a simplified algorithm that alternates between enforcing data-consistency and low-rank penalty.
We validate our approach experimentally and demonstrate it delivers substantial improvements. In particular, we compare several algorithmic solutions to the problem and demonstrate that: (a) availability of temporal information significantly improves the quality of estimated depth; (b) motion adaptive regularization is crucial for avoiding artifacts along temporal dimension; (c) using intensity during block matching is essential for optimal performance.
2 Related Work
In the last decade, guided depth superresolution has received significant attention. Early work by Diebel and Thrun  showed the potential of the approach by modeling the co-occurence of edges in depth and intensity with Markov Random Fields (MRF). Kopf et al.  and Yang et al.  have independently proposed an alternative approach based on joint bilaterial filtering, where intensity is used to set the weights of the filter. The bilaterial filtering approach was further refined by Chan et al.  who incorporated the local statistics of the depth and by Liu et al.  who used geodesic distances for determining the weights. Dolson et al.  extended the approach to dynamic sequences to compensate for different data rates in the depth and intensity sensors. He et al. [7, 8] proposed a guided image filtering approach, improving edge preservation.
More recently, sparsity-promoting regularization—an essential component of compressive sensing [9, 10]—has provided more dramatic improvements in the quality of depth superresolution. For example, Li et al.  demonstrated improvements by combining dictionary learning and sparse coding algorithms. Ferstl et al.  relied on weighted total generalized variation (TGV) regularization for imposing a piecewise polynomial structure on depth. Gong et al.  combined the conventional MRF approach with an additional term promoting transform domain sparsity of the depth in an analysis form. In their recent work, Huang et al.  use the MRF model to jointly segment the objects and recover a higher quality depth. Schuon et al.  performed depth superresolution by taking several snapshots of a static scene from slightly displaced viewpoints and merging the measurements using sparsity of the weighted gradient of the depth.
Many natural images contain repetitions of similar patterns and textures. Current state-the-art image denoising methods such as nonlocal means (NLM)  and block matching and 3D filtering (BM3D)  take advantage of this redundancy by processing the image as a structured collection of patches. The original formulation of NLM was extended by Yang and Jacob  to more general inverse problems via introduction of specific NLM regularizers. Similarly, Danielyan et al.  have proposed a variational approach for general BM3D-based image reconstruction that inspired the current work. In the context of guided depth superresolution, NLM was used by Huhle et al.  and Park et al.  for reducing the amount of noise in the estimated depth. Lu et al.  combined a block-matching procedure with low-rank constraints for enhancing the resolution of a single depth image.
Our paper extends prior work on depth supperresolution by introducing a new variational formulation that imposes low-rank constraints in the regularization. Furthermore, our formulation is motion-adaptive, resulting in substantial improvement of the quality of the estimated depth.
3 Proposed Approach
Our approach estimates the high-resolution depth map by minimizing a cost function that—as typical in such problems—combines a data-fidelity term and a regularizer. Specifically, we impose a quadratic data fidelity term that controls the error between the measured and estimated depth values. The regularizer groups similar depth patches from multiple frames and penalizes the rank of the resulting structure. Since we use patches from multiple frames, our method is implicitly motion adaptive. Thus, by effectively combining multiple views of the scene, it yields improved depth estimates.
3.1 Problem Formulation
The depth sensing system collects a set of measurements denoted . Each measurement is considered as a downsampled version of a higher resolution depth map using a subsampling operator . Our end goal is to recover this high-resolution depth map for all .
In the remainder of this work, we use to denote the number of pixels in each frame, to denote the number of temporal frames, and to denote the total number of depth measurements. Furthermore,
denotes the vector of all the measurements,the complete sequence of high-resolution depth maps, and the complete subsampling operator. We also have available the sequence of high-resolution intensity images from the optical camera, denoted .
Using the above, a forward model for the depth recovery problem is given by
where denotes the measurement noise. Thus, our objective becomes to recover high-resolution depth given the measured data and , and the sampling operator .
As typical in such problems, we formulate the depth estimation task as an optimization problem
where enforces data fidelity and is a regularization term that imposes prior knowledge about the depth map.
We form the regularization term by constructing sets of patches from the image. Specifically, we first define an operator , for each set of patches , where is the number of such sets constructed. The operator extracts patches of size pixels from the depth image frames in . As illustrated in Fig. 2, each block is obtained by first selecting a reference patch and then finding similar patches within the current frame as well as the adjacent temporal frames.
To determine similarity and to group similar patches together we use the intensity image as a guide. To reduce the computational complexity of the search, we restrict it to a space-time window of fixed size around the reference patch. We perform the same block matching procedure for the whole space-time image by moving the reference patch and by considering overlapping patches in each frame. Thus, each pixel in the signal may contribute to multiple blocks.
The adjoint of simply corresponds to placing the patches in the block back to their original locations in . It satisfies the following property
where and denotes the total number of references to the th pixel by the matrices . Therefore, the depth image can be expressed in terms of an overcomplete representation using the blocks
3.2 Rank regularization
Each block, represented as a matrix, contains multiple similar patches, i.e., similar columns. Thus, we expect the matrix to have a low rank, making rank a natural regularizer for the problem
By seeking a low-rank solution to (2), we exploit the similarity of blocks to guide superresolution while enforcing consistency with the observed data. However, the rank regularizer (5) is of little practical interest since its direct optimization is intractable. The most popular approach around this, first proposed by Fazel in , is to convexify the rank by replacing it with the nuclear norm:
where denotes the
th largest singular value ofand is a parameter controlling the amount of regularization.
In addition to its convexity, the nuclear norm is an appealing penalty to optimize because it also has a closed form proximal operator:
is the singular value decomposition (SVD) ofand is the soft-thresholding function applied to the diagonal matrix .
Recent work has shown that nonconvex regularizers consistently outperform nuclear norm by providing stronger denoising capability without losing important signal compoenents [24, 25, 26]. In this paper, we use the nonconvex generalization to the nuclear norm proposed by Chartrand 
Here, the scalar function is designed to satisfy
where is the -Huber function
with . Although is nonconvex and has no closed form formula, its proximal operator does admit a closed form expression
where is a poitwise -shrinkage operator defined as
, it approaches hard thresholding, which is similar to principal component analysis (PCA) in the sense that it retains the significant few principal components.
Thus, the regularizer (8) is a computationally tractable alternative to the rank penalty. While the regularizer is not convex, it can still be efficiently optimized due to closed form of its proximal operator (3.2). Note that due to nonconvexity of our regularizer for , it is difficult to theoretically guarantee global convergence. However, we have empirically observed that our algorithms converge reliably over a broad spectrum of examples presented in Section 4.
3.3 Iterative optimization
The minimization is performed using an augmented-Lagrangian (AL) method . Specifically, we seek the critical points of the following AL
where is the quadratic parameter and is the dual variable that imposes the constraint . Traditionally, an AL scheme solves (2) by alternating between a joint minimization step and an update step as
However, the joint minimization step (14a) is typically computationally intensive. To reduce complexity, we separate (14a) into a succession of simpler steps using the well-established by now alternating direction method of multipliers (ADMM) . The steps are as follows
By ignoring the terms that do not depend on , (15a) amounts to solving a quadratic problem
where . Solving this quadratic equation is efficient since the inversion is performed on a diagonal matrix. Similarly, (15b) is solved by
with . This step can be solved via block-wise application of the proximal operator (3.2) as
for all .
3.4 Simplified algorithm
The algorithm in Section 3.3 can be significantly simplified by decoupling the enforcement of the data-fidelity from the enforcement of the rank-based regularization. The simplified algorithm reduces computational complexity while making estimation more uniform across the whole space-time depth image.
In particular, Danielyan  has argued that, due to inhomogeneous distribution of pixel references generated by matching across the image, using a penalty with a single regularization parameter highly penalizes pixels with a large number of references. The resulting nonuniform regularization makes the algorithm potentially oversensitive to the choice of the parameter . Instead, we rely on the simplified algorithm
where , and is the regularization and is the quadratic parameters.
To verify our development, we report results on extensive simulations using our guided depth superresolution algorithms. In particular, we compare results of both the ADMM approach (denoted ADMM-3D) and its simplified variant (denoted GDS-3D) against six alternative methods.
As the first and the simplest reference method, we consider standard linear interpolation (Linear). In addition to linear interpolation, we consider methods relying in some form of total variation (TV) regularization, one of the most widely used regularizers in the context of image reconstruction due to its ability to reduce noise while preserving image edges . Specifically, we consider depth interpolation using TV-regularized least squares on a frame-by-frame basis (TV-2D). We also consider the weighted-TV formulation proposed by Ferstl et al.  (WTGV-2D), also operating on a frame-by-frame basis. This formulation uses a weighted anisotropic total generalized variation, where weighting is computed using the guiding intensity image, thus promoting edge co-occurrence in both modalities. Finally, we consider a weighted-TV formulation which includes time, i.e., multiple frames (WTV-3D), with weights computed using the guiding intensity image, as before.
We also compare these methods to two variations of our algorithm. To illustrate the potential gains of our method due to temporal information, we run our simplified algorithm on a frame-by-frame basis (GDS-2D), i.e., using no temporal information. Similarly, to illustrate the gains due to intensity information, we also run the simplified algorithm by performing block matching on the low-resolution depth as opposed to intensity (DS-3D). This last approach conceptually corresponds to denoising the initial depth using our motion adaptive low-rank prior.
To improve convergence speed, we initialized all the iterative algorithms with the solution of linear interpolation. The regularization parameters of TV-2D, WTGV-2D, and WTV-3D were optimized for the best signal-to-noise ratio (SNR) performance. Similarly, the regularization parameters for GDS-2D, DS-3D, ADMM-3D, and GDS-3D were optimized for the best SNR performance. However, in order to reduce the computational time, the selection was done from a restricted set of three predetermined parameters. The methods TV-2D, WTV-3D, GDS-2D, DS-3D, ADMM-3D, and GDS-3D were all run for a maximum of iterations with an additional stopping criterion based on measuring the relative change of the solution in two successive iterations
where is the desired tolerance level. We selected other parameters of WTGV-2D as suggested in the code provided by the authors; in particular, we run the algorithm for a maximum of iterations with the stopping tolerance of . In all experiments, the patch size was set to pixels, the space-time window size to pixels, and the number of similar patches was fixed to . Parameters and were hand selected to and , respectively.
For quantitative comparison of the algorithms, we rely on the data-set by Zhang et al. , which consists of three video sequences Flower, Lawn, and Road, containing both intensity and depth information on the scenes. We considered images of size with time frames. The ground truth depth was downsized by factors of , , , and , and was corrupted by additive Gaussian noise corresponding to SNR of dB. Table 1 reports the SNR of superresolved depth for all the algorithms and downsizing factors. Figures 4 and 5 illustrate the evolution of SNR against the frame number for Road and Flower, respectively, at downsizing factor of . The effectiveness of the proposed approach can also be appreciated visually in Figs. 6 and 7.
For additional validation, we test the algorithm on the KITTI dataset . The dataset contains intensity images from PointGray Flea2 cameras and 3D point clouds from a Velodyne HDL-64E, which have been calibrated a priori using specific known targets. We consider a region of interest of pixels with time frames. The total lidar measurements are randomly and uniformly split into a reconstruction and a validation sets. Note that this implies a depth measurement rate of just . We then estimate depth from the reconstruction set using WTV-3D, as well as GDS-3D, and use the validation set to evaluate the quality of the results; these are illustrated in Fig. 8.
The examples and simulations in this section, validate our claim: the quality of estimated depth can be considerably boosted by properly incorporating temporal information into the reconstruction procedure. Comparison of GDS-2D against GDS-3D highlights the importance of additional temporal information. The approach we propose is implicitly motion adaptive and can thus preserve temporal edges substantially better than alternative approaches such as WTV-3D. Moreover, comparison between DS-3D and GDS-3D highlights that the usage of intensity significantly improves the performance of the algorithm. Note also the slight improvement in SNR of GDS-3D over ADMM-3D. This is consistent with the arguments in  that suggest to decouple data-fidelity from the enforcement of prior constraints when using block-matching-based methods.
We presented a novel motion-adaptive approach for guided superresolution of depth maps. Our method identifies and groups similar patches from several frames, which are then supperresolved using a rank regularizer. Using this approach, we can produce high-resolution depth sequences from significantly down-sized low-resolution ones. Compared to the standard techniques, the proposed method preserves temporal edges in the solution and effectively mitigates noise in practical configurations.
While our formulation has higher computational complexity than standard approaches that process each frame individually, it allows us to incorporate a very effective regularization for stabilizing the inverse problem associated with superresolution. The algorithms we describe enable efficient computation and straightforward implementation by reducing the problem to a succession of straightforward operations. Our experimental results demonstrate the considerable benefits of incorporating time and motion adaptivity into inverse-problems for depth estimation.
-  J. Diebel and S. Thrun, “An application of markov random ffield to range sensing,” in Proc. Advances in Neural Information Processing Systems 18, vol. 18, Vancouver, BC, Canada, December 5-8, 2005, pp. 291–298.
-  J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” in ACM Trans. Graph. (Proc. SIGGRAPH 2007), vol. 26, San Diego, CA, USA, August 5-9, 2007, p. 96.
Q. Yang, R. Yang, J. Davis, and D. Nister, “Spatial-depth super resolution for range images,” in
Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Minneapolis, MN, USA, June 17-22, 2007, pp. 1–8.
-  D. Chan, H. Buisman, C. Theobalt, and S. Thrun, “A noise-aware filter for real-time depth upsampling,” in ECCV Workshop on Multicamera and Multimodal Sensor Fusion Algorithms and Applications, 2008.
-  M.-Y. Liu, O. Tuzel, and Y. Taguchi, “Joint geodesic upsampling of depth images,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA, June 23-28, 2013, pp. 169–176.
-  J. Dolson, J. Baek, C. Plagemann, and S. Thrun, “Upsampling range data in dynamic environments,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), vol. 23, San Francisco, CA, USA, June 13-18, 2010, pp. 1141–1148.
-  K. He, J. Sun, and X. Tang, “Guided image filtering,” in Proc. ECCV, Hersonissos, Greece, September 5-11, 2010, pp. 1–14.
-  ——, “Guided image filtering,” IEEE Trans. Patt. Anal. and Machine Intell., vol. 35, no. 6, pp. 1397–1409, June 2013.
-  E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, February 2006.
-  D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289–1306, April 2006.
-  Y. Li, T. Xue, L. Sun, and J. Liu, “Joint example-based depth map super-resolution,” in Proc. IEEE Int. Con. Multi., Melbourne, VIC, Australia, July 9-13, 2012, pp. 152–157.
-  D. Ferstl, C. Reinbacher, R. Ranftl, M. Ruether, and H. Bischof, “Image guided depth upsampling using anisotropic total generalized variation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Sydney, NSW, Australia, December 1-8, 2013, pp. 993–1000.
-  X. Gong, J. Ren, B. Lai, C. Yan, and H. Qian, “Guided depth upsampling via a cosparse analysis model,” in Proc. IEEE CVPR WKSHP, Columbus, OH, USA, June 23-28, 2014, pp. 738–745.
-  W. Huang, X. Gong, and M. Y. Yang, “Joint object segmentation and depth upsampling,” IEEE Signal Process. Lett., vol. 22, no. 2, pp. 192–196, February 2015.
-  S. Schouon, C. Theobalt, J. Davis, and S. Thrun, “LidarBoost: Depth superresolution for ToF 3D shape scanning,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, June 20-25, 2009, pp. 343–350.
-  A. Buades, B. Coll, and J. M. Morel, “Image denoising methods. A new nonlocal principle.” SIAM Rev, vol. 52, no. 1, pp. 113–147, 2010.
-  K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 16, pp. 2080–2095, August 2007.
-  Z. Yang and M. Jacob, “Nonlocal regularization of inverse problems: A unified variational framework,” IEEE Trans. Image Process., vol. 22, no. 8, pp. 3192–3203, August 2013.
-  A. Danielyan, V. Katkovnik, and K. Egiazarian, “BM3D frames and variational image deblurring,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 1715–1728, April 2012.
-  B. Huhle, T. Schairer, P. Jenke, and W. Strasser, “Fusion of range and color images for denoising and resolution enhancement with a non-local filter,” Comput. Vis. Image Underst., vol. 114, no. 12, pp. 1336–1345, December 2010.
-  J. Park, H. Kim, Y.-W. Tai, M. S. Brown, and I. Kweon, “High quality depth map upsampling for 3D-TOF cameras,” in Proc. IEEE Int. Conf. Comp. Vis., Barcelona, Spain, November 6-13, 2011, pp. 1623–1630.
-  S. Lu, X. Ren, and F. Liu, “Depth enhancement via low-rank matrix completion,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, June 23-28, 2014, pp. 3390–3397.
-  M. Fazel, “Matrix rank minimization with applications,” Ph.D. dissertation, PhD thesis, Stanford University, 2002.
-  R. Chartrand, “Nonconvex splitting for regularized Low-Rank + Sparse decomposition,” IEEE Trans. Signal Process., vol. 60, no. 11, pp. 5810–5819, November 2012.
-  Y. Hu, S. G. Lingala, and M. Jacob, “A fast majorize-minimize algorithm for the recovery of sparse and low-rank matrices,” IEEE Trans. Image Process., vol. 21, no. 2, pp. 742–753, February 2012.
-  H. Yoon, K. S. Kim, D. Kim, Y. Bresler, and J. C. Ye, “Motion adaptive patch-based low-rank approach for compressed sensing cardiac cine MRI,” IEEE Trans. Med. Imag., vol. 33, no. 11, pp. 2069–2085, November 2014.
-  J. Nocedal and S. J. Wright, Numerical Optimization, 2nd ed. Springer, 2006.
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed
optimization and statistical learning via the alternating direction method of
Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.
-  A. Danielyan, “Block-based collaborative 3-D transform domain modeing in inverse imaging,” Publication 1145, Tampere University of Technology, May 2013.
-  L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, vol. 60, no. 1–4, pp. 259–268, November 1992.
-  L. Zhang, S. Vaddadi, H. Jin, and S. K. Nayar, “Multiple view image denoising,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA, June 20-25, 2009, pp. 1542–1549.
-  A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, November 2013.