Depth map super-resolution (DSR) (c.f., Fig. 1) aims at super-resolving a high-resolution depth map from a low-resolution depth map input [1, 2, 3, 4], which is a challenging task especially under large up-sampling factors (, , and beyond). This is mainly due to the great information loss in down-sampling. For example, under the up-sampling factor of , 256 () depth values have to be inferred from a single depth value on average. To tackle this highly under-constrained problem, various methods have been proposed by exploiting the availability of large-scale training datasets. Even though deep convolutional neural network (DCNN) based methods have achieved great success in various vision tasks such as image deblurring , image denoising 
, monocular depth estimation[46, 56], saliency prediction [28, 27], and even color image super-resolution (CSR) , it is only very recently that the success of DCNN in color image super-resolution  has been extended to the task of depth map super-resolution . This is mainly due to the intrinsic differences between color images and depth maps, where the depth maps generally contain less textures and more sharp boundaries, and are usually degraded by noise due to the imprecise consumer depth cameras. The difficulty in capturing high-resolution depth map further increases the challenge.
Under the pipelines of current DCNN based DSR methods , a depth map usually needs to be up-sampled before feeding into the network. However, the up-sampled depth maps do not necessarily provide a proper good initialization for the network learning. And the problem becomes even worse when the resolution of the input low-resolution depth map is too low and the up-sampling factor is too large. Hence, to improve the representative ability of DCNN, Riegler et al.  resorted to increasing the depth of the network. Unfortunately, this deep structure may suffer from the vanishing gradient issue as the supervision is only enforced at the very end of the network. Besides, deconvolution strategy  has also been used in DSR to improve the quality of the resultant feature maps, which can be viewed as an inverse operation of convolution. The deconvolution operator generally needs a large number of parameters. In this paper, we would like to argue that neither the hand-designed up-sampling nor the deconvolution is necessary for depth map super-resolution.
In this paper, we adopt a different way by representing the task of depth map super-resolution as a series of novel view synthesis sub-tasks at different positions, where each sub-task generates a depth map at a different camera pose. Take the depth map super-resolution task with an up-sampling factor as an example, where we would like to generate new depth values from one depth measurement in the low-resolution depth map. We partition the desired high-resolution depth map with size () into four parts: , , and , where . These four depth maps own the same spatial resolution and could be viewed as depth maps captured by virtual depth cameras at four different positions, which has the same resolution as the low-resolution depth map with resolution (). Therefore, instead of learning a direct nonlinear mapping from to , we propose to learn four separate nonlinear mappings (e.g.novel view synthesis) from to , , and separately, where each of the nonlinear mapping could be learned through DCNN in parallel. Then the super-resolution task can be formulated as predicting the depth maps corresponding to the four different virtual cameras from the input low-resolution depth map. In this way, the input and output of each novel view synthesis task have the same resolution, which makes the network structure easy to design and implement. In Fig. 2, we illustrate our idea of representing depth map super-resolution as novel view synthesis.
Furthermore, to handle large up-sampling factors, a deeply supervised learning strategy is proposed. In training each sub-task, the supervision from the output could be used independently. Thus, we achieve a deeply supervised learning framework to DSR. As the supervision has been deeply enforced at different layers of the network, strong supervision is well expected. As each novel view synthesis sub-task is learned independently, there will be blocking effect between different parts, which only happens in the final output depth map. To reduce the blocking effect and effectively exploit the feature maps at different stages, we propose a multi-scale fusion strategy (MSF) to further improve the learned depth map, where inter-media depth maps at different scales are fused. Finally, to better utilize the individual information from each depth map, we impose a global depth field statistic prior (DFS) to further optimize the obtained depth maps. Our method does not need the support of color information, which reduces the restriction of corresponding color images.
As illustrated in Fig. 1, our method outperforms state-of-the-art DSR methods especially under large up-sampling factors (). Our main contributions can be summarized as:
We represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks, where each sub-task can be efficiently solved in an end-to-end learning manner;
A deeply supervised learning framework is proposed to handle large up-sampling factors in depth map super-resolution, where strong supervisions are applied at different stages;
A multi-scale fusion strategy (MFS) and depth field statistic (DFS) are proposed to effectively exploit the feature maps at different scales and handle the blocking effect;
Experiments on various benchmarking datasets demonstrate the superiority of our method over state-of-the-art depth map super-resolution methods, including the DSR methods using the color images.
2 Related Work
2.1 Depth image super-resolution
Depth map super-resolution (DSR) methods can be roughly classified into three categories: conventional learning based DSR, high-resolution (HR) intensity image guided DSR and deep convolutional neural network (DCNN) based DSR.
Conventional learning based DSR: It is proven that HR depth maps can be generated by low-resolution (LR) depth maps based on prior information. Inspired by Freeman et al. , Aodha et al.  proposed a patch based MRF method to DSR by using prior information learned from depth map datasets. Hornácek et al.  generated HR depth maps by searching low- and high-resolution patch-pairs of arbitrary size in the depth map itself. What’s more, sparse representation and dictionary learning have also been utilized in DSR. Ferstl et al.  proposed to generate HR depth edges by learning from an external database of high and low-resolution examples, where the Total Generalized Variation (TGV) was employed as regularization. Xie et al.  used the MRF optimization approach to generate sharp HR depth edges from LR depth edges, where the HR depth edges were used as guidance in generating the HR depth maps.
HR intensity image guided DSR: Pre-aligned HR intensity images can always provide effective guidance for DSR since they contain plenty of useful high-frequency components which assist the process of DSR. Park et al.  generated HR depth maps by non-local means filter with an HR intensity image as auxiliary information. Yang et al.  used an adaptive color-guided auto-regressive (AR) model to generate HR depth maps from LR depth maps. Ferstl et al. 
utilized an anisotropic diffusion tensor, which used HR color image as guidance. Kiechleet al.  made use of a bimodal co-sparse analysis model to generate an HR depth map from an LR depth map and an HR color image. Additionally, Matsuo et al.  generated HR depth maps by using auxiliary information extracted from HR color images to compute local tangent planes in depth maps. What’s more, Lu et al.  utilized the consistency between color images and depth maps to generate HR depth maps. However, notwithstanding the appealing results that such approaches could generate, the lack of high-resolution color images fully registered with the depth maps in many cases makes the color image guided approaches less general.
DCNN based DSR:
The success of DCNN in high-level computer vision tasks has only been extended to DSR very recently. Songet al.  proposed a progressive DCNN based end-to-end learning method to generate an HR depth map from an LR depth map, where SRCNN  was used as the mapping unit in the progressive process. Meanwhile, Riegler et al.  proposed to combine DCNN with total variations in a novel ATGV-Net to generate HR depth maps. The total variations were expressed by layers with fixed parameters. Besides, Riegler et al.  also proposed a novel DCNN based method by combining a DCNN with a non-local variational method. The corresponding HR color images were also utilized in the method. LR depth maps were all needed to be up-sampled before feeding them to the network . Very recently, Hui et al.  proposed to use a multi-scale fusion strategy in a Multi-Scale Guided convolutional network for DSR with and without the guidance of the color images.
2.2 DCNN based CSR
The success of DCNN has been extended to color image super-resolution (CSR). Using DCNN, effective frameworks have been proposed to super-resolve the low-resolution color images.
Dong et al.  proposed an end-to-end deep CNN framework to learn the nonlinear mapping between low- and high-resolution images. Based on , Kim et al.  proposed to use a deeper network to represent the non-linear mapping and improved performance has been achieved. Meanwhile, in , a deeply-recursive convolutional network for CSR is proposed, which uses a deep recursive layer to obtain better results. Shi et al.  proposed a sub-pixel DCNN for CSR instead of using the deconvolution strategy, which utilized a sub-pixel layer to learn an array of up-sampling filters to upscale the LR feature maps to HR output. Ledig et al.  proposed a residual framework to infer photo-realistic natural images, where an adversarial loss and a content loss were used. Recently, Tai et al.  proposed a Deep Recursive Residual Network to mitigate the difficulty of training deep networks and used Recursive learning to control the model parameters. Lai et al.  proposed a Laplacian Pyramid Super-Resolution Network to progressively solve the problem. To handle high magnification ratios and create realistic textures, Sajjadi et al.  proposed to use feed-forward fully CNN and perceptual loss to achieve automated texture synthesis. Tong et al.  introduced dense skip connections in a very deep network for CSR. Huang et al.  proposed to utilize weakly-supervised joint convolutional sparse coding to solve the problem of multi-modal image super-resolution.
3 Our Approach
In this paper, we propose a new framework to tackle the problems in handling large up-sampling factors (e.g., , ) and the need of deconvolution or pre-processing. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. Each novel view synthesis sub-task aims at generating (synthesizing) a depth map from a slightly different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, to exploit the feature maps learned in different stages with different scales, we propose to use a multi-scale fusion (MFS) strategy, which fuses the inter-media feature maps at different scales. In this way, our framework could tackle the challenging task depth map super-resolution with large up-sampling factors efficiently. In Fig. 3, we illustrate the whole pipeline of our method under an up-sampling factor of . Our method takes a low-resolution depth map as input and does not require the corresponding color image.
3.1 Deconvolution in DCNN based DSR
, as an inverse operation of convolution, is a novel operation to recover an HR depth map from an LR depth map. The deconvolution layer employs a set of deconvolution filters to up-sample the feature maps, and, the filter is convolved with the image by a stride of, and the output is times of the input. Most of the current DCNN based depth super-resolution methods  need the up-sampled LR depth images as input, the process of up-sampling LR depth maps can also be viewed as a deconvolution layer whose parameters are fixed. However, up-sampling the low-resolution depth map does not necessarily provide a proper initialization of the network.
Recently, Shi et al.  proposed a sub-pixel DCNN for color image super-resolution without deconvolution. Feature maps are extracted in the LR space, and a subpixel layer which learns an array of up-sampling filters to upscale LR feature maps to HR output is utilized. Inspired by , we present a novel view to deal with DSR as deeply supervised novel view synthesis task.
3.2 DSR as Novel View Synthesis
To avoid up-sampling the LR depth maps before feeding them into the network, we represent DSR as a series of novel view synthesis sub-tasks. Rather than treating it as learning a nonlinear mapping between the low-resolution depth map and the high-resolution depth map, we decompose the desired output as a collection of low-resolution depth maps acquired at slightly different viewpoints from different “virtual cameras”. Under this setup, each virtual camera owns the same spatial resolution which is the same as the input low-resolution depth map. For example, given a DSR task with an up-sampling factor , we could decompose the DSR task to novel view synthesis tasks. This is because each virtual camera can be explained as having a translation between them, or forming a light field camera as illustrated in Fig. 2.
Specifically, as shown in Fig. 4, given the current network input, under , we denote the task of DSR as predicting the depth maps viewed at the position of and . Under a pure down-sampling version, the low-resolution depth map is exactly the depth map viewed at the position of
. Then the first novel view synthesis is directly an identical mapping. The other three tasks could be directly viewed as predicting the depth map with a motion vector, sayfor the horizontal direction, for the vertical direction and for the diagonal direction. With different down-sampling operators, the detailed meaning may be different, but the principle is pretty the same. The above formulation could be generalized to other up-sampling factors with ease.
Through this representation, these low-resolution depth maps of the virtual cameras could be learned in parallel. Meanwhile, as the whole DSR task has been decoupled to several different novel view synthesis sub-tasks, where for each sub-task, the output and input have the same spatial resolution. Thus, neither deconvolution nor the up-sampled input is needed in our approach.
As illustrated in Fig. 4, given an LR depth map as input, layers DCNN unit is first employed to estimate the depth values at each novel view synthesis in parallel (, , and ), where is the result of each sub-net. Then a re-organization operation is utilized to generate the output . Note that we use a residual DCNN unit here.
Specifically, for each novel view synthesis sub-task, the process can be described as follows:
where denotes the nonlinear mapping between the LR depth map and the HR depth map. , are the learnable network weights and biases respectively, denotes convolution. is the input LR depth map, is a nonlinear function and is the estimated novel view synthesis output for position .
Then, the final output can be obtained by using the re-organization operation which is described as follows:
where is a periodic operation that re-arranges low-resolution depth maps of dimension to a high-resolution depth map of dimension .
Training each sub-task: Take an up-sampling factor of as an example, the ground truth high-resolution depth map is decomposed into four sub-ground truth depth maps, (the reverse process of the re-organization operation as illustrated in Fig. 4), and in each sub-task, the LR depth map has the same resolution with the corresponding high-resolution supervision depth sub-map, which makes the DCNN easy to design and implement. Thus, neither deconvolution or up-sampling of the LR input are needed.
3.3 Deeply Supervised Learning
During training, besides the high-resolution target/ground truth depth map used to supervise the final output, its down-sampled versions have also been used at different stages of the depth map learning framework, denoted as , , ,, which are expressed as follows:
where is the number of supervision stages and
is the down-sampling factor between two consecutive stages. Here the bicubic interpolation is used as the down-sampling strategy, which is commonly used in depth map super-resolution and color image super-resolution. In this way, the aim of each sub-task can be viewed as learning the residual between two consecutive stages (two consecutive down-sampled versions of high-resolution target/ground truth) using the novel view synthesis strategy. The residual of each stage is much smaller than the residual between the input low-resolution depth map and the ground truth depth map, which does not need very deep layers to present. Hence, deeply supervised learning can effectively handle the gradient vanishing issue and obtain better results. In this way, the network could receive direct supervision in learning each factor, which could better regularize the learned depth map.
3.4 Multi-scale Fusion
In the above deeply supervised learning structure, the ground truth depth map has been down-sampled to different resolutions to provide supervisions at different scales. While providing deep supervision and guiding the generation of high-resolution depth maps, the quality of the down-sampled depth maps has been gradually decreased due to the downsampling effect, i.e., the more downsampling has applied, the more smooth the obtained depth maps are. Therefore, the supervision depth maps actually encode the depth map details at different scales. To handle this side-effect and better utilize the inter-media feature maps, we propose a multi-scale fusion strategy (MSF) (c.f. Fig. 3), which fuses the feature maps (depth maps) at different stages. Specifically, the predicted depth maps are upsampled to the same resolution as the output and then are concatenated together as the input to another DCNN unit, which not only achieves multi-scale fusion of the feature maps at different scales but also effectively handles the blocking effect introduced by the individual novel view synthesis sub-tasks. By integrating both deeply supervised learning and multi-scale fusion strategies, our network is able to effectively exploit the supervision provided by the ground truth high-resolution depth map and the multi-scale feature maps.
3.5 Network Architecture
For each novel view synthesis sub-task, we utilize the network VDSR-Net  as the DCNN unit due to its high performance in color image super-resolution. Furthermore, we propose to learn the residual between the input and the ground truth rather than learning the depth map itself. Note that the DCNN unit can be replaced by any other DCNN networks. Fig. 4 shows the process of one sub-task. Taking low-resolution depth maps as input, depth maps corresponding to the positions , , and are trained by their corresponding DCNN units in parallel. Then, the output is obtained by re-organizing the four depth maps.
Taking the DSR task with an up-sampling factor as an example, we demonstrate the network structure in Fig. 3. The low-resolution depth maps are fed into the first stage, and the resultant depth maps are regarded as input for the next stage. Down-sampled ground truth is used as supervision to refine the depth maps at each stage. Then, the multi-scale fusion is employed to improve the quality of the resultant high-resolution depth maps, and finally, the depth field statistic (DFS) is used to further refine the resultant high-resolution depth maps.
Our method can also be extended to handle flexible up-sampling factors such as , , which can be recognized as generating depth maps of virtual cameras from 9 or 25 positions from a low-resolution depth map.
3.6 Depth Field Statistics
In the above end-to-end learning framework for DSR, the learned depth maps could be biased by different depth map statistics and we cannot expect that the network could learn the high-frequency information or edge information.
The distribution of a natural depth map can often be modeled as a generalized Laplace distribution , where the distribution of gradient magnitude of depth images can be well approximated with Laplacian distribution. Therefore, we propose to minimize the total variation of the depth map, i.e., where the total variation could be expressed in matrix form:
where consists of as its elements that works as the gradient operator, denotes the vectorization operator that transforming a matrix to a vector.
3.7 An Energy Minimization Formulation
To further refine the depth super-resolution results, we integrate the depth super-resolution cue from the deeply supervised DCNN and the depth field statistics (DFS), and reach the following energy minimization formulation:
where is the depth super-resolution result from the deep neural network (novel view synthesis strategy + MFS), and is the final depth map we want to generate. is set as empirically in our experiments. This is a convex optimization problem where a global optimal solution exists. We propose to use the iterative reweighted least squares (IRLS)   to efficiently solve the problem. Given the depth map estimation result in the -th iteration, the optimization for the -th iteration can be expressed as:
which could be equivalently expressed as:
Denote , i.e., the row-wise reweighted version of and , we have:
The above least squares problem owns a closed-form solution.
Park et al. 
|Yang et al. ||2.4214||1.8941||5.6312||1.2368||5.1390||4.0660||13.1748||2.7559||5.1390||4.0660||13.1748||2.7559|
Ferstl et al. 
Yang et al. 
|Zeyde et al. ||1.9539||1.5013||4.5276||0.9305||3.2232||2.3527||7.3003||1.4751||4.8945||3.5670||11.9758||2.2879|
|Aodha et al. ||4.3185||3.2828||9.1089||2.2098||12.6938||4.1113||12.6938||2.6497||N/A||N/A||N/A||N/A|
|Hornácek et al. ||3.7512||3.1395||8.8070||2.0383||5.4898||5.0212||11.1101||3.5833||N/A||N/A||N/A||N/A|
|Huang et al. ||4.6273||3.4293||10.0766||2.1653||6.2723||4.8346||13.7645||3.0606||6.1629||6.6235||10.6618||4.1399|
|Schultery et al. ||1.9199||1.5545||4.2400||1.0185||2.9859||2.3793||6.8026||1.5477||N/A||N/A||N/A||N/A|
|Ferstl et al. ||2.2139||1.7205||5.3252||1.1230||3.5680||2.6474||7.5356||1.7771||N/A||N/A||N/A||N/A|
|Xie et al. ||2.7338||2.4911||6.3534||1.6390||4.4087||3.2768||9.7765||2.3714||N/A||N/A||N/A||N/A|
|Wang et al. ||1.8895||1.4074||3.8789||0.8935||2.9263||2.0638||6.0356||1.2640||4.8933||3.0437||9.8942||1.8618|
Song et al. 
|Park et al. ||0.9452||0.9610||0.9052||0.9811||0.9321||0.9510||0.8756||0.9799||0.9231||0.9426||0.8409||0.9792|
|Yang et al. ||0.9833||0.9850||0.9721||0.9946||0.9629||0.9697||0.9322||0.9882||0.9370||0.9488||0.8633||0.9773|
|Ferstl et al. ||0.9755||0.9795||0.9576||0.9938||0.9625||0.9707||0.9245||0.9901||N/A||N/A||N/A||N/A|
Yang et al. 
|Zeyde et al. ||0.9655||0.9717||0.9438||0.9886||0.9604||0.9628||0.9147||0.9884||0.9385||0.9503||0.8718||0.9816|
|Aodha et al. ||0.9606||0.9690||0.9364||0.9874||0.9392||0.9520||0.9080||0.9822||N/A||N/A||N/A||N/A|
|Hornácek et al. ||0.9696||0.9719||0.9461||0.9895||0.9501||0.9503||0.9137||0.9789||N/A||N/A||N/A||N/A|
|Huang et al. ||0.9582||0.9673||0.9301||0.9875||0.9360||0.9425||0.8821||0.9784||0.9280||0.9254||0.9027||0.9712|
|Ferstl et al. ||0.9866||0.9884||0.9766||0.9963||0.9645||0.9716||0.9413||0.9893||N/A||N/A||N/A||N/A|
|Xie et al. ||0.8932||0.9012||0.9053||0.9300||0.8885||0.8927||0.8405||0.9175||N/A||N/A||N/A||N/A|
|Wang et al. ||0.9891||0.9907||0.9866||0.9966||0.9758||0.9798||0.9640||0.9937||0.9475||0.9618||0.9132||0.9878|
Song.et al 
3.8 Implementation Details
DCNN unit: In this paper, we use the VDSR-Net  with 10 convolutional layers as the DCNN unit. Each convolutional filter has the size of for the novel view synthesis sub-task and for the multi-scale fusion strategy, and each hidden layer of the network has feature maps. For each DCNN unit, the learning rate varies from to , and the momentum is chosen as . The stepwise decrease (4 steps with learning rate multiplier
) as the learning policy, and adjustable gradient clipping strategy is used.
|Park et al. ||3.7906||4.1823||2.2619||3.7799||4.1322||2.2075||3.0175||3.1835||N/A||N/A||N/A|
|Zeyde et al. ||3.0823||2.9615||1.8113||2.8241||3.8967||1.7817||2.9839||2.9304||2.3399||2.3371||3.3357|
|Yang et al. ||3.1787||3.0837||1.8748||2.9407||3.8967||1.7817||2.9839||2.9304||2.2071||2.1705||3.7650|
|Aodha et al. ||N/A||N/A||N/A||N/A||3.8967||1.7817||2.9839||2.9304||2.3156||2.1225||4.0006|
|Ferstl et al. ||3.6394||3.8577||2.2640||3.4801||4.9346||4.5651||6.9055||4.6487||N/A||N/A||N/A|
|Huang et al. ||3.6394||3.8577||2.2640||3.4801||3.7537||3.1474||3.4566||3.5258||2.6827||2.7017||3.2517|
|Xie et al. ||N/A||N/A||N/A||N/A||3.7995||2.0742||2.6162||2.9992||2.3233||2.2411||4.0130|
|Wang et al. ||3.3520||3.3191||2.8827||3.0703||3.5072||2.7229||3.0895||3.1753||2.1460||2.1314||3.0442|
|Park et al. ||0.9258||0.9762||0.9859||0.9824||0.9551||0.9717||0.9661||0.9679||N/A||N/A||N/A|
|Zeyde et al. ||0.9321||0.9799||0.9865||0.9839||0.9337||0.9408||0.9361||0.9378||0.9592||0.9592||0.9506|
|Yang et al. ||0.9311||0.9801||0.9866||0.9845||0.9486||0.9656||0.9597||0.9607||0.9725||0.9732||0.9599|
|Aodha et al. ||N/A||N/A||N/A||N/A||0.9714||0.9869||0.9764||0.9828||0.9857||0.9873||0.9828|
|Ferstl et al. ||0.9237||0.9704||0.9796||0.9760||0.9631||0.9663||0.9365||0.9733||N/A||N/A||N/A|
|Huang et al. ||0.8834||0.9078||0.9063||0.9170||0.9074||0.9077||0.9014||0.9047||0.9359||0.9350||0.9301|
|Xie et al. ||N/A||N/A||N/A||N/A||0.9567||0.9706||0.9710||0.9685||0.9718||0.9716||0.9643|
|Wang et al. ||0.9077||0.9349||0.9333||0.9455||0.9320||0.9353||0.9288||0.9318||0.9655||0.9657||0.9582|
|Park et al. ||6.5770||8.2173||4.5044||6.3069||9.1964||3.7853||6.4360||6.0934||N/A||N/A||N/A|
|Zeyde et al. ||5.3660||5.9962||3.5302||5.5972||6.6357||3.4477||4.6541||5.0603||4.4270||4.0878||7.2210|
|Yang et al. ||5.5559||6.2265||3.6571||5.8026||6.6079||2.9578||4.3285||4.8519||4.3007||3.8619||7.2310|
|Wang et al. ||5.2034||5.6034||3.8110||4.6530||6.2276||3.3479||4.4003||4.7777||3.4242||3.1981||5.4845|
Training Data: depth maps from the Middlebury stereo dataset  (25 images), the Sintel dataset  (60 images) and the synthetic New Tsukuba dataset  (30 images) are collected to construct our dataset, depth maps are used for training and depth maps are used for validation. Using these depth maps as ground-truth high-resolution depth maps , the input low-resolution depth maps are generated by , where is the down-sampling factor and bicubic interpolation is used as the down-sampling strategy. Note that to accelerate the training time of larger up-sampling factors, such as , and , parameters of (model of deeply supervised novel view synthesis part) are used to initialize the parameters of the first stage of these larger up-sampling factors. For larger up-sampling factors, in each DCNN unit of the first stage, the learning rate varies from 0.01 to 0.0001, and the stepwise decrease (3 steps with learning rate multiplier ) as the learning policy. Using a Titan X GPU (Pascal), training for the task of depth map super-resolution under up-sampling factors , , and roughly takes 3 hours, 4 hours, 5 hours and 6 hours, respectively.
4 Experimental Results
In this section, we present an extensive experimental evaluation of our proposed method. Both quantitative and qualitative results on noise-free and noisy benchmark datasets are provided. , , and , as noisy-free depth maps, are extracted from the Middlebury 2001 and 2003 datasets . , , and as noisy depth maps, are collected from the Middlebury 2005 dataset . Furthermore, to further evaluate our proposed method, , , and , as noisy depth maps, are extracted from the Middlebury 2014 dataset . Additionally, we also demonstrate test results of the Laserscan dataset (, and ) provided by Aodha et al. .
Baseline Methods: We compare our methods with the following five categories of methods:
Single color image super resolution approaches: Zeyde et al., Yang et al. and Timofte et al. , including two kinds of methods: Global Regression () and Anchored Neighborhood Regression (), the neighborhood embedding methods proposed by Bevilacqua et al. , including , and , Huang et al. , Schulter et al.  and Lai et al. ;
Standard interpolate approaches: Bicubic and Nearest Neighbour ();
State-of-the-art deep convolutional neural networks: Wang et al. , VDSR-Net 111For a fair comparison, results of VDSR-Net are obtained using the same number of layers with ours, which means VDSR-Net contains 10 layers for up-sampling factor , contains 20 layers for up-sampling factor , contains 30 layers for up-sampling factor , and contains 40 layers for up-sampling factor ., Song et al. , Hui et al.  (MS-Net) and Riegler et al.  (ATGV-Net).
Note that for these deep learning based methods, we retrained the model using depth maps dataset either with the source code provided by the authors or implement those methods by ourself.
Error metrics: In this paper, for quantitative comparison, we use two kinds of error metrics to evaluate the results obtained by our method and other state-of-the-art methods, including: 1) Root Mean Squared Error (); 2). Structure Similarity of Index (). As a frequently used measure,
represents the sample standard deviation of the differences between the obtained HR depth maps and the ground truth. Specifically,, where and are the obtained HR depth map and ground truth respectively, is the number of pixels in the HR depth map. Meanwhile, is an error metric which evaluates the perceived quality between the obtained HR depth map and the ground truth depth map. We have also computed the percent of errors (percent of pixels that have errors larger than 1 pixel in the disparity map) and the results of percent errors are pretty consistent with the error metrics of and reported in our paper.
4.1 Clean Input
We evaluate the performance of our method on clean depth maps (there is no noise added in the process of down-sampling). The commonly used , , and from the Middlebury stereo dataset are employed. The quantitative results in terms of and of up-sampling factors of , and are reported in Table I and Table II. We can clearly see that all the DCNN based methods achieve significant performance leap over other methods under all the up-sampling factors, even though methods  utilize auxiliary intensity information. Additionally, our proposed method outperforms other DCNN based methods for almost all the up-sampling factors. Note that the VDSR-Net  utilizes up-sampled (bicubic) depth map as input, and its structure is the same as our DCNN unit. Additionally, for a fair comparison, results of VDSR-Net  are obtained using the same number of layers with ours, which means VDSR-Net contains 10 layers for up-sampling factor , contains 20 layers for up-sampling factor , and contains 30 layers for up-sampling factor . Obviously, our proposed method outperforms VDSR-Net, which demonstrates the excellent performance of our deeply supervised novel view synthesis and multi-scale fusion strategy. Meanwhile, for , as shown in Table I and Table II, we trained two kinds of models, namely (up-sampling factors of , and are trained together) and (up-sampling factors of , and are trained separately), we can see that results obtained by our method are better than  under up-sampling factors of , and , which further proves the effectiveness of our method.
As indicated in Table I and Table II, our deeply supervised novel view synthesis strategy () outperforms others for most of the up-sampling factors, and the multi-scale fusion strategy () can handle blocking effect and further improve the results for most up-scale factors, though results of are slightly lower than for an up-sampling factor of . Lastly, the depth field statistic prior () can further improve the results.
Qualitative results are illustrated in Fig. 5 (part of and ) for an up-sampling factor . Obviously, our method produces more visually appealing results. Boundaries generated by our method are sharper and accurate, which show that our method can effectively recover the structure of high-resolution depth maps.
|Park et al. ||0.9125||0.9613||0.9782||0.9727||0.9401||0.9792||0.9632||0.9721||N/A||N/A||N/A|
|Zeyde et al. ||0.9067||0.9575||0.9741||0.9651||0.9034||0.9792||0.9632||0.9721||0.9482||0.9644||0.9277|
|Yang et al. ||0.9080||0.9596||0.9750||0.9671||0.9356||0.9804||0.9666||0.9664||0.9675||0.9722||0.9426|
|Wang et al. ||0.9083||0.9578||0.9708||0.9699||0.9362||0.9742||0.9609||0.9643||0.9746||0.9771||0.9577|
4.2 Noisy Input
We also evaluate our proposed method on the noisy Middlebury dataset  and the Laserscan dataset . To simulate the acquisition process of a Time-of-Flight sensor, depth dependent Gaussian noise is added to the Middlebury dataset (, , of the Middlebury 2005 dataset and , , , of the Middlebury 2014 dataset) and the Laserscan dataset (, and ). Following , we add depth dependent Gaussian noise to our low-resolution training data in the form , where and denotes the depth value of each pixel in .
Table III and Table IV report the quantitative results in terms of and for the up-sampling factor of with noisy input, from which, we can clearly see that our proposed method outperforms others. We can observe that our method can well eliminate the influence of noise, thus depth maps with smaller and larger can be obtained. Note that results of VDSR-Net  are obtained using the same number of layers with ours for noisy input.
Besides, Fig. 7 illustrates the qualitative results of our method () for an up-sampling factor . As shown in Fig. 7, contains complex textures and luxuriant details, which is hard to recover an HR depth map from an LR depth map. Obviously, our method produces more visually appealing results. Boundaries generated by our method are sharper and accurate, which show that our method can effectively recover the structure of high-resolution depth maps.
4.3 Large Up-sampling Factor
Most of the existing work such as  could only handle depth super-resolution problem under up-sampling factors , and . In our paper, due to the proposed deeply supervised learning network structure, our method can handle large up-sampling factors effectively. Meanwhile, the multi-scale fusion and depth field statistics can further refine the obtained results. In this section, we demonstrate the qualitative results under large up-sampling factors (, and ). It is a very challenging super-resolution task, i.e., on average inferring depth values from depth value for .
Fig. 8 shows the qualitative comparison results of of from the Middlebury 2014 dataset , which demonstrates the ability of our method in handling general up-sampling factors besides (). Fig. 6 ( of the Laserscan dataset) shows the qualitative comparison results for an up-sampling factor . Clean low-resolution depth maps are used as input of Fig. 8 and Fig. 6. As these figures illustrate, depth maps generated by other competing methods all suffer from heavy edge blur, while our method can obtain results with sharp boundaries and accurate depth edges.
Meanwhile, we also illustrate the qualitative results under large up-sampling factor in Fig. 9 ( of the Middlebury dataset). Note that Gaussian noisy described in Section 4.2 is added in the depth input of Fig. 9. Obviously, our method can well eliminate the influence of noise and accurately recover the high-resolution depth maps with sharper boundaries, even under a very large up-sampling factor of .
In addition, Table V and Table VI demonstrate the quantitative results of our method on and , respectively. As illustrated in these tables, our method outperforms all competing methods with a margin by achieving smaller and larger on all the testing datasets. Note that same depth dependent Gaussian noisy is added to the depth maps as described in Section 4.2.
4.4 Ablation Analysis
As shown in Table I, Table II, Table III, Table IV, Table V and Table VI, compared with other state-of-the-art methods, our deeply supervised novel view synthesis strategy () can obtain better results. To handle the blocking effect introduced in the resultant depth maps, the multi-scale fusion strategy ( + ) is used, and as shown in these tables, the multi-scale fusion strategy () can improve the and under most of the testing depth maps. Besides, since all DCNN based methods can only learn the generality of depth super-resolution, we also use depth filed statistic () information (features of each depth map) to further refine the results. As shown in these tables, better and results can be obtained for most of the testing depth maps by using depth field statistic (). Therefore, we can conclude that the three components, namely, deeply supervised novel view synthesis strategy, multi-scale fusion strategy and depth field statistics all contribute positively toward the final success of our approach.
4.5 Influence of the number of layers
In this section, we analyze the influence of the number of layers of DCNN unit in novel view synthesis sub-task. Fig. 10 shows the of our method on the Middlebury dataset, where the number of layers of DCNN unit varies from 3 to 10. We can observe that decreases with the increase of the number of layers of DCNN unit, however, the difference between 3 layers and 5 layers is larger than the difference between 5 layers and 10 layers. Thus, we can conclude that deeper DCNN generally results in better performance. However, as the net goes deeper, the improvement becomes marginal. Hence, we fix the number of layers of DCNN unit as 10 in our experiments.
4.6 Running time
We summarize the computation time for up-sampling different LR depth maps to their full resolution () in Table VII. Up-sampling is performed in MATLAB with a TITAN X GPU (Pascal) using MatConvNet . Note that the running time of MS-Net  is obtained from their paper which is calculated using a TITAN X GPU.
Besides, the running time of the depth field statistics (DFS) varies a lot depending on different size of input depth maps. For examples, the depth field statistic strategy costs 2.845 seconds on the depth map (370 450 resolution ), while it costs 10.321 seconds on the depth map (1376 1104 resolution).
4.7 Generalization ability
To evaluate the generalization ability of our network, we use a new training dataset (60 depth maps from the Sintel dataset and 30 depth maps from the synthetic New Tsukuba dataset) to retrain our network, and then test the network on the Middlebury dataset. results () are shown in TableVIII (_). of TableVIII shows the results obtained by training data of section 3.8. We can see that and results are pretty similar to the results reported in Table I, which demonstrates the excellent generalization ability of our network.
To further demonstrate our generalization ability on raw depth maps, we show the average results of the NYU dataset  in Table IX. The NYU dataset comprises of video sequences of indoor scenes captured by a Kinect. For each set, depth maps are selected randomly. Note that we use the model trained by training data (115 depth maps from , and ) described in our paper to test the NYU dataset. As shown in Table IX, our network model can handle wild and raw depth maps. Note that results shown here are obtained by our deeply supervised novel view synthesis strategy.
In this paper, we propose to represent depth map super-resolution as a series of novel view synthesis sub-tasks, where each sub-task can be efficiently solved in an end-to-end learning manner. The training stage can be performed in parallel, while neither deconvolution nor pre-processing of input depth map is needed. Furthermore, a deeply supervised learning framework is proposed to handle large up-sampling factors (), where strong supervisions are directly applied in different stages. To further exploit the feature maps at different stages, a multi-scale fusion strategy has also been introduced. Our framework can handle DSR efficiently without the need for high-resolution intensity images, and both qualitative and quantitative results demonstrate the outstanding performance of our method compared with the state-of-the-art DSR methods. In the future, we plan to investigate how to exploit the color images under the same framework and how to learn a natural and realistic depth map statistics through GAN (Generative adversarial network).
This work is supported by National Natural Science Foundation of China (Nos.61672326, 61671387, 61420106007), National 1000 Young Talents Plan of China, ARC Grants (DE140100180, LP100100588), and China Scholarship Council.
-  J. Zhang, Y. Cao, Z.-J. Zha, Z. Zheng, C. W. Chen, Z. Wang: A Unified Scheme for Super-Resolution and Depth Estimation From Asymmetric Stereoscopic Video. IEEE Trans. Circuits Syst. Video Techn. 26(3): 479-493, 2016.
-  X. Song, Y. Dai, X. Qin: Deep Depth Super-Resolution: Learning Depth Super-Resolution Using Deep Convolutional Neural Network. In Proc. Asian Conf. Comp. Vis. pp. 360-376, 2016.
-  F. Zhou, S.-T. Xia, Qi. Liao: Nonlocal Pixel Selection for Multisurface Fitting-Based Super-Resolution. IEEE Trans. Circuits Syst. Video Techn. 24(12): 2013-2017, 2014.
-  L.-J. Deng, W. Guo, T.-Z. Huang: Single-Image Super-Resolution via an Iterative Reproducing Kernel Hilbert Space Method. IEEE Trans. Circuits Syst. Video Techn. 26(11): 2001-2014, 2016.
-  J. Sun, W. Cao, Z. Xu, J. Ponce: Learning a convolutional neural network for non-uniform motion blur removal. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 769-777, 2015.
-  L. Xu, J. S. J. Ren, C. Liu, J. Jia: Deep Convolutional Neural Network for Image Deconvolution. In Proc. Adv. Neural Inf. Process. Syst. pp. 1790-1798, 2014.
-  V. Jain, H. S. Seung: Natural Image Denoising with Convolutional Networks. In Proc. Adv. Neural Inf. Process. Syst. pp. 769-776, 2008.
-  J. Xie, L. Xu, E. Chen: Image Denoising and Inpainting with Deep Neural Networks. In Proc. Adv. Neural Inf. Process. Syst. pp. 350-358, 2012.
-  C. Dong, C. C. Loy, K. He, X. Tang: Learning a Deep Convolutional Network for Image Super-Resolution. In Proc. Eur. Conf. Comp. Vis. pp. 184-199, 2014.
-  C. Dong, C. C. Loy, X. Tang: Accelerating the Super-Resolution Convolutional Neural Network. In Proc. Eur. Conf. Comp. Vis. pp. 391-407, 2016.
-  J. Kim, J. K. Lee, K. M. Lee: Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp., 1646-1654, 2016.
-  J. Kim, J. K. Lee, K. M. Lee: Deeply-Recursive Convolutional Network for Image Super-Resolution. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1637-1645, 2016.
-  T.-W. Hui, C. C. Loy, X. Tang: Depth Map Super-Resolution by Deep Multi-Scale Guidance. In Proc. Eur. Conf. Comp. Vis. pp. 353-369, 2016.
-  G. Riegler, D. Ferstl, M. Rüther, H. Bischof: A Deep Primal-Dual Network for Guided Depth Super-Resolution. In Proc. Brit. Mach. Vis. Conf. 2016.
-  G. Riegler, M. Rüther, H. Bischof: ATGV-Net: Accurate Depth Super-Resolution. In Proc. Eur. Conf. Comp. Vis. pp. 268-284, 2016.
-  J. Xie, R. S. Feris, M.-T. Sun: Edge-Guided Single Depth Image Super Resolution. IEEE Trans. Image Proc. 25(1): 428-438, 2016.
-  W. T. Freeman, T. R. Jones, E. C. Pasztor: Example-Based Super-Resolution. IEEE Computer Graphics and Applications 22(2): 56-65, 2002.
-  O. M. Aodha, N. D. F. Campbell, A. Nair, G. J. Brostow: Patch Based Synthesis for Single Depth Image Super-Resolution. In Proc. Eur. Conf. Comp. Vis. pp. 71-84, 2012.
-  M. Hornacek, C. Rhemann, M. Gelautz, C. Rother: Depth Super Resolution by Rigid Body Self-Similarity in 3D. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1123-1130, 2013.
-  D. Ferstl, M. R ther, H. Bischof: Variational Depth Superresolution Using Example-Based Edge Representations. In Proc. IEEE Int. Conf. Comp. Vis. pp. 513-521, 2015,
-  J. Park, H. Kim, Y.-W. Tai, M. S. Brown, I.-S. Kweon: High quality depth map upsampling for 3D-TOF cameras. In Proc. IEEE Int. Conf. Comp. Vis. pp. 1623-1630, 2011.
-  J. Yang, X. Ye, K. Li, C. Hou: Depth Recovery Using an Adaptive Color-Guided Auto-Regressive Model. In Proc. Eur. Conf. Comp. Vis. pp. 158-171, 2012.
-  D. Ferstl, C. Reinbacher, R. Ranftl, M. R ther, H. Bischof: Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation. In Proc. IEEE Int. Conf. Comp. Vis. pp. 993-1000, 2013.
-  M. Kiechle, S. Hawe, M. Kleinsteuber: A Joint Intensity and Depth Co-sparse Analysis Model for Depth Map Super-resolution. In Proc. IEEE Int. Conf. Comp. Vis. pp. 1545-1552, 2013.
-  K. Matsuo, Y. Aoki: Depth image enhancement using local tangent plane approximations. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 3574-3583, 2015.
-  J. Lu, D. A. Forsyth: Sparse depth super resolution. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 2245-2253, 2015.
-  J. Zhang, T. Zhang, Y. Dai, M. Harandi, R. Hartley: Deep Unsupervised Saliency Detection: A Multiple Noisy Labeling Perspective. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 9029-9038, 2018.
-  A. Borji, M. Cheng, H. Jiang, and J. Li. Salient object detection: A benchmark. IEEE Trans. Image Proc., 24(12):5706 C 5722, 2015.
-  W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, Z. Wang: Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1874-1883, 2016.
-  C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. P. Aitken, Al. Tejani, J. Totz, Z. Wang, W. Shi: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 105-114, 2017.
-  Y. Tai, J. Yang, X. Liu: Image Super-Resolution via Deep Recursive Residual Network. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 2790-2798, 2017.
-  W.-S. Lai, J.-B. Huang, N. Ahuja, M.-H. Yang: Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 5835-5843, 2017.
-  M. S. M. Sajjadi, B. Schölkopf, M. Hirsch: EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis. In Proc. IEEE Int. Conf. Comp. Vis. pp. 4501-4510, 2017.
-  T. Tong, G. Li, X. Liu, Q. Gao: Image Super-Resolution Using Dense Skip Connections. In Proc. IEEE Int. Conf. Comp. Vis. pp. 4809-4817, 2017.
-  Y. Huang, L. Shao, A. F. Frangi: Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images Using Weakly-Supervised Joint Convolutional Sparse Coding. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 5787-5796, 2017.
-  R. Zeyde, M. Elad, M. Protter: On Single Image Scale-Up Using Sparse-Representations. In International Conference on Curves and Surfaces pp. 711-730, 2010.
-  R. Timofte, V. D. Smet, L. J. Van Gool: Anchored Neighborhood Regression for Fast Example-Based Super-Resolution. In Proc. IEEE Int. Conf. Comp. Vis. pp. 1920-1927, 2013.
-  R. Chartrand, W. Yin: Iteratively reweighted algorithms for compressive sensing. In Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Process. pp. 3869-3872, 2008.
-  T. Ajanthan, R. I. Hartley, M. Salzmann, H. Li: Iteratively reweighted graph cut for multi-label MRFs with non-convex priors. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 5144-5152, 2015.
-  J. Yang, J. Wright, T. S. Huang, Y. Ma: Image Super-Resolution Via Sparse Representation. IEEE Trans. Image Process. 19(11): 2861-2873, 2010.
-  J.-B. Huang, A. Singh, N. Ahuja: Single image super-resolution from transformed self-exemplars. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 5197-5206, 2015.
-  S. Schulter, C. Leistner, H. Bischof: Fast and accurate image upscaling with super-resolution forests. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 3791-3799, 2015.
-  Z. Wang, D. Liu, J. Yang, W. Han, T. S. Huang: Deep Networks for Image Super-Resolution with Sparse Prior. In Proc. IEEE Int. Conf. Comp. Vis. pp. 370-378, 2015.
-  H. Hirschmüller, D. Scharstein: Evaluation of Cost Functions for Stereo Matching. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1-8. 2007.
-  D. Scharstein, C. Pal: Learning Conditional Random Fields for Stereo. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1-8. 2007.
B. Li, C. Shen, Y. Dai, A. van den Hengel, M. He: Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. InProc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 1119-1127, 2015.
-  D. Scharstein, R. Szeliski: A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. International Journal of Computer Vision 47(1-3): 7-42, 2002.
-  D. J. Butler, J. Wulff, G. B. Stanley, M. J. Black: A Naturalistic Open Source Movie for Optical Flow Evaluation. In Proc. Eur. Conf. Comp. Vis. pp. 611-625, 2012.
-  S. Martull, M. Peris, K. Fukui. Realistic CG Stereo Image Dataset with Ground Truth Disparity Maps. IEICE Technical Report, 2012, 111(430):117-118.
-  D. Scharstein, R. Szeliski: High-Accuracy Stereo Depth Maps Using Structured Light. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 195-202, 2003.
-  M. Bevilacqua, A. Roumy, C. Guillemot, M.-L. Alberi-Morel: Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In Proc. Brit. Mach. Vis. Conf. pp. 1-10, 2012.
D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nesic, X. Wang, P. Westling: High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth. In
Proc. German Conference on Pattern Recognitionpp. 31-42, 2014.
-  A. Vedaldi, K. Lenc: MatConvNet: Convolutional Neural Networks for MATLAB. In ACM international conference on Multimedia pp. 689-692, 2015.
-  N. Silberman, D. Hoiem, P. Kohli, R. Fergus: Indoor Segmentation and Support Inference from RGBD Images. In Proc. Eur. Conf. Comp. Vis. pp.746-760, 2012.
-  N. Silberman, R. Fergus: Indoor scene segmentation using a structured light sensor. In ICCV Workshops pp. 601-608, 2011.
-  B. Li, Y. Dai, M. He: Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference. Pattern Recognition 83(11): 328-339, 2018.