1 Introduction
Recent learningbased multiview stereo (MVS) methods [ji2017surfacenet_an1, yao2018mvsnet_depth1, huang2018deepmvs_learning] have shown great success compared with their traditional counterparts as learningbased approaches are able to learn to take advantage of scene global semantic information, including object materials, specularity, and environmental illumination, to get more robust matching and more complete reconstruction. All these approaches apply dense multiscale 3D CNNs to predict the depth map or voxel occupancy. However, 3D CNNs require memory cubic to the model resolution, which can be potentially prohibitive to achieving optimal performance. While Maxim et al. [tatarchenko2017octreegen] addressed this problem by progressively generating an Octree structure, the quantization artifacts brought by grid partitioning still remain, and errors may accumulate since the tree is generated layer by layer.
In this work, we propose a novel point cloud multiview stereo network, where the target scene is directly processed as a point cloud, a more efficient representation, particularly when the 3D resolution is high. Our framework is composed of two steps: first, in order to carve out the approximate object surface from the whole scene, an initial coarse depth map is generated by a relatively small 3D cost volume and then converted to a point cloud. Subsequently, our novel PointFlow module is applied to iteratively regress accurate and dense point clouds from the initial point cloud. Similar to ResNet [he2016resnet], we explicitly formulate the PointFlow to predict the residual between the depth of the current iteration and that of the ground truth. The 3D flow is estimated based on geometry priors inferred from the predicted point cloud and the 2D image appearance cues dynamically fetched from multiview input images (Figure 1).
We find that our Pointbased Multiview Stereo Network (PointMVSNet) framework enjoys advantages in accuracy, efficiency, and flexibility when it is compared with previous MVS methods that are built upon a predefined 3D volume with the fixed resolution to aggregate information from views. Our method adaptively samples potential surface points in the 3D space. It keeps the continuity of the surface structure naturally, which is necessary for high precision reconstruction. Furthermore, because our network only processes valid information near the object surface instead of the whole 3D space as is the case in 3D CNNs, the computation is much more efficient. Lastly, the adaptive refinement scheme allows us to first peek at the scene at coarse resolution and then densify the reconstructed point cloud only in the region of interest. For scenarios such as interactionoriented robot vision, this flexibility would result in saving of computational power.
Our method achieves stateoftheart performance on standard multiview stereo benchmarks among learningbased methods, including DTU [aanaes2016LargeScaleDataMultipleView] and Tanks and Temples [knapitsch2017tanks_and]. Compared with previous stateoftheart, our method produces better results in terms of both completeness and overall quality. Besides, we show potential applications of our proposed method, such as foveated depth inference.
2 Related work
Multiview Stereo Reconstruction
MVS is a classical problem that had been extensively studied before the rise of deep learning. A number of 3D representations are adopted, including volumes
[vogiatzis2007multiview, hornung2006hierarchical], deformation models [esteban2004silhouette, zaharescu2007transformesh], and patches [furukawa2010accurate_dense], which are iteratively updated through multiview photometric consistency and regularization optimization. Our iterative refinement procedure shares a similar idea with these classical solutions by updating the depth map iteratively. However, our learningbased algorithm achieves improved robustness to input image corruption and avoids the tedious manual hyperparameters tuning.Learningbased MVS
Inspired by the recent success of deep learning in image recognition tasks, researchers began to apply learning techniques to stereo reconstruction tasks for better patch representation and matching [han2015matchnet, seki2017sgm, knobelreiter2017cnncrf]. Although these methods in which only 2D networks are used have made a great improvement on stereo tasks, it is difficult to extend them to multiview stereo tasks, and their performance is limited in challenging scenes due to the lack of contextual geometry knowledge. Concurrently, 3D cost volume regularization approaches have been proposed [kendall2017gcnet, ji2017surfacenet_an1, kar2017lsm], where a 3D cost volume is built either in the camera frustum or the scene. Next, the 2D image features of multiviews are warped in the cost volume, so that 3D CNNs can be applied to it. The key advantage of 3D cost volume is that the 3D geometry of the scene can be captured by the network explicitly, and the photometric matching can be performed in 3D space, alleviating the influence of image distortion caused by perspective transformation and potential occlusions, which makes these methods achieve better results than 2D learning based methods. Instead of using voxel grids, in this paper we propose to use a pointbased network for MVS tasks to take advantage of 3D geometry learning without being buredened by the inefficiency found in 3D CNN computation.
HighResolution MVS
Highresolution MVS is critical to real applications such as robot manipulation and augmented reality. Traditional methods [lhuillier2005quasi, furukawa2010accurate_dense, owens2013shape_anchors] generate dense 3D patches by expanding from confident matching keypoints repeatedly, which is potentially timeconsuming. These methods are also sensitive to noise and change of viewpoint owing to the usage of handcrafted features. Recent learning methods try to ease memory consumption by advanced space partitioning [riegler2017octnet, Wang2017OCNN, tatarchenko2017octreegen]. However, most of these methods construct a fixed cost volume representation for the whole scene, lacking flexibility. In our work, we use point clouds as the representation of the scene, which is more flexible and enables us to approach the accurate position progressively.
Pointbased 3D Learning
Recently, a new type of deep network architecture has been proposed in [qi2016PointNetDeepLearning, qi2017PointNetDeepHierarchical], which is able to process point clouds directly without converting them to volumetric grids. Compared with voxelbased methods, this kind of architecture concentrates on the point cloud data and saves unnecessary computation. Also, the continuity of space is preserved during the process. While PointNets have shown significant performance and efficiency improvement in various 3D understanding tasks, such as object classification and detection [qi2017PointNetDeepHierarchical], it is under exploration how this architecture can be used for MVS task, where the 3D scene is unknown to the network. In this paper, we propose PointFlow module, which estimates the 3D flow based on joint 2D3D features of point hypotheses.
3 Method
This section describes the detailed network architecture of PointMVSNet (Figure 2). Our method can be divided into two steps, coarse depth prediction, and iterative depth refinement. Let denote the reference image and denote a set of its neighbouring source images. We first generate a coarse depth map for (subsection 3.1). Since the resolution is low, the existing volumetric MVS method has sufficient efficiency and can be used. Second we introduce the 2D3D feature lifting (subsection 3.2), which associates the 2D image information with 3D geometry priors. Then we propose our novel PointFlow module (subsection 3.3) to iteratively refine the input depth map to higher resolution with improved accuracy.
3.1 Coarse depth prediction
Recently, learningbased MVS [ji2017surfacenet_an1, yao2018mvsnet_depth1, im2018dpsnet_endtoend] achieves stateoftheart performance using multiscale 3D CNNs on cost volume regularization. However, this step could be extremely memory expensive as the memory requirement is increasing cubically as the cost volume resolution grows. Taking memory and time into consideration, we use the recently proposed MVSNet [yao2018mvsnet_depth1] to predict a relatively lowresolution cost volume.
Given the images and corresponding camera parameters, MVSNet [yao2018mvsnet_depth1] builds a 3D cost volume upon the reference camera frustum. Then the initial depth map for reference view is regressed through multiscale 3D CNNs and the soft argmin [knapitsch2017tanks_and] operation. In MVSNet, feature maps are downsampled to of the original input image in each dimension and the number of virtual depth planes are for both training and evaluation. On the other hand, in our coarse depth estimation network, the cost volume is constructed with feature maps of the size of the reference image, containing or virtual depth planes for training and evaluation, respectively. Therefore, our memory usage of this 3D feature volume is about 1/20 of that in MVSNet.
3.2 2D3D feature lifting
Image Feature Pyramid
Learningbased image features are demonstrated to be critical to boosting up dense pixel correspondence quality [yao2018mvsnet_depth1, tang2018BANetDenseBundle]
. In order to endow points with a larger receptive field of contextual information at multiple scales, we construct a 3scale feature pyramid. 2D convolutional networks with stride
are applied to downsample the feature map, and each last layer before the downsampling is extracted to construct the final feature pyramid for image . Similar to common MVS methods[yao2018mvsnet_depth1, im2018dpsnet_endtoend], feature pyramids are shared among all input images.Dynamic Feature Fetching
The point feature used in our network is compromised of the fetched multiview image feature variance with the normalized 3D coordinates in world space
. We will introduce them separately.Image appearance features for each 3D point can be fetched from the multiview feature maps using a differentiable unprojection given corresponding camera parameters. Note that features are at different image resolutions, thus the camera intrinsic matrix should be scaled at each level of the feature maps for correct feature warping. Similar to MVSNet [yao2018mvsnet_depth1], we keep a variancebased cost metric, i.e. the feature variance among different views, to aggregate features warped from an arbitrary number of views. For pyramid feature at level , the variance metric for views is defined as below:
(1) 
To form the features residing at each 3D point, we do a concatenation of the fetched image feature and the normalized point coordinates:
(2) 
This feature augmented point is the input to our PointFlow module.
As shall be seen in the next section, since we are predicting the depth residual iteratively, we update the point position after each iteration and fetch the point feature from the image feature pyramid, an operation we name as dynamic feature fetching. Note that this step is distinct from costvolumebased methods, by which the fetched features at each voxel are determined by the fixed space partition of the scene. In contrast, our method can fetch features from different areas of images dynamically according to the updated point position. Therefore, we can concentrate on the regions of interest in the feature maps, instead of treating them uniformly.
3.3 PointFlow
Depth maps generated from subsection 3.1 have limited accuracy due to the low spatial resolution of 3D cost volume. We propose PointFlow, our novel approach to iteratively refine the depth map.
With known camera parameters, we first unproject the depth map to be a 3D point cloud. For each point, we aim to estimate its displacement to the ground truth surface along the reference camera direction by observing its neighboring points from all views, so as to push the points to flow to the target surface. Next, we discuss the components of our module in detail.
Point Hypotheses Generation
It is nontrivial to regress depth displacement of each point from the extracted image feature maps. Due to perspective transformation, the spatial context embedded in 2D feature maps cannot reflect the proximity in 3D Euclidean space.
In order to facilitate the modeling of network, we propose to generate a sequence of point hypotheses with different displacement along the reference camera direction as shown in Figure 3. Let denote the normalized reference camera direction, and denote the displacement step size. For an unprojected point , its hypothesized point set is generated by
(3) 
These point hypotheses are critical for the network to infer the displacement, for the necessary neighbourhood image feature information at different depth are gathered in these points along with spatial geometry relationship.
Edge Convolution
Classical MVS methods have demonstrated that local neighborhood is important for robust depth prediction. Similarly, we take the strategy of recent work DGCNN [wang2018DynamicGraphCNN] to enrich feature aggregation between neighboring points. As shown in Figure 3, a directed graph is constructed on the point set using nearest neighbors (kNN), such that local geometric structure information could be used for the feature propagation of points.
Denote the feature augmented point cloud by , then edge convolution is defined as:
(4) 
where is a learnable nonlinear function parameterized by , and
is a channelwise symmetric aggregation operation. There are multiple options for the symmetry operation, including max pooling, average pooling, and weighted sum. We compared max pooling and average pooling and observed similar performance after tuning hyperparameters carefully.
Flow Prediction
The network architecture for flow prediction is shown in Figure 4
. The input is a feature augmented point cloud, and the output is a depth residual map. We use three EdgeConv layers to aggregate point features at different scales of the neighborhood. Shortcut connections are used to include all the EdgeConv outputs as local point features. Finally, a shared multilayer perceptron (MLP) is used to transform the pointwise features, which outputs a probability scalar with softmax among hypothesized points of each unprojected point. The displacement of the unprojected points are predicted as the probabilistic weighted sum of the displacement among all predicted point hypotheses:
(5) 
Note that this operation is differentiable. The output depth residual map is obtained by projecting the displacement back, which will be added to the initial input depth map for depth refinement.
Iterative Refinement with Upsampling
Because of the flexibility of our pointbased network architecture, the flow prediction can be performed iteratively, which is much harder for 3D costvolumebased methods, because the space partitioning is fixed after the construction of cost volume. For depth map from coarse prediction or former residual prediction, we can first upsample it using nearest neighbor to higher spatial resolution and then perform the flow prediction to obtain . Moreover, we decrease the depth interval between the unprojected points and hypothesized points at each iteration, so that more accurate displacement can be predicted by capturing more detailed features from closer point hypotheses.
3.4 Training loss
Similar to most deep MVS networks, we treat this problem as a regression task and train the network with the loss, which measures the absolute difference between the predicted depth map and the groundtruth depth map. Losses for the initial depth map and iteratively refined ones are all considered:
(6) 
where represents the valid groundtruth pixel set and is the iteration number. The weight is set to 1.0 in training.
Acc. (mm)  Comp. (mm)  Overall (mm)  
Camp [campbell2008using_multiple]  0.835  0.554  0.695 
Furu [furukawa2010accurate_dense]  0.613  0.941  0.777 
Tola [tola2012EfficientLargescaleMultiview]  0.342  1.190  0.766 
Gipuma [galliani2016gipuma]  0.283  0.873  0.578 
SurfaceNet [ji2017surfacenet_an1]  0.450  1.040  0.745 
MVSNet [yao2018mvsnet_depth1]  0.396  0.527  0.462 
Ours  0.361  0.421  0.391 
OursHiRes  0.342  0.411  0.376 
4 Experiments
4.1 DTU dataset
The DTU dataset [aanaes2016LargeScaleDataMultipleView] is a largescale MVS dataset, which consists of 124 different scenes scanned in 7 different lighting conditions at 49 or 64 positions. The data for each scan is composed of an RGB image and corresponding camera parameters. The dataset is split into training, validation, and evaluation sets.
4.2 Implementation details
Training
We train PointMVSNet on the DTU training dataset. For data preprocessing, we follow MVSNet [yao2018mvsnet_depth1] to generate depth maps from the given groundtruth point clouds. During training, we set input image resolution to , and number of views to . The input view set is chosen with the same view selection strategy as in MVSNet (Section 4.1). For coarse prediction, we construct a 3D cost volume with virtual depth planes, which are uniformly sampled from mm to mm. For the depth refinement step, we set flow iterations , with depth intervals being mm and mm, respectively. The number of nearest neighbor points
. We use RMSProp of initial learning rate
which is decreased by 0.9 for every 2 epochs. The coarse prediction network is trained alone for 4 epochs, and then, the model is trained endtoend for another 12 epochs. Batch size is set to 4 on 4 NVIDIA GTX 1080Ti graphics cards.
Evaluation
We use depth layers for initial depth prediction and set flow iterations for depth refinement. We predict the reference view depth map for each view set. Then we fuse all depth maps to point clouds using same postprocessing provided by [yao2018mvsnet_depth1]. We evaluate our method in two different input image resolutions: (“Ours”), and (“OursHiRes”).
MVSNet [yao2018mvsnet_depth1]  Ours  Ground Truth 
9 of DTU dataset. Top: Whole point cloud. Bottom: Visualization of normals in zoomed local area. Our PointMVSNet generates detailed point clouds with more highfrequency component than MVSNet. For fair comparison, the depth maps predicted by MVSNet are interpolated to the same resolution as our method.
4.3 Benchmarking on DTU dataset
We evaluate the proposed method on the DTU evaluation dataset. Quantitative results are shown in Table 1 and Figure 6, where the accuracy and completeness are computed using the official code from the DTU dataset, and the fscore is calculated as mentioned in [knapitsch2017tanks_and] as the measure of overall performance of accuracy and completeness. While Gipuma [galliani2016gipuma] performs the best in terms of accuracy, our PointMVSNet outperforms startoftheart in both completeness and overall quality. Qualitative results are shown in Figure 5. PointMVSNet generates a more detailed point cloud compared with MVSNet. Especially in those edgy areas, our method can capture highfrequency geometric features.
4.4 PointFlow iteration
Initial  Iter1  Iter2  Iter3 
Iter.  Acc. (mm)  Comp. (mm)  Overall (mm)  0.5mm fscore  Depth Map Res.  Depth Interval (mm)  GPU Mem. (MB)  Runtime (s) 

  0.693  0.758  0.726  47.95  160120  5.30  7219  0.34 
1  0.674  0.750  0.712  48.63  160120  5.30  7221  0.61 
2  0.448  0.487  0.468  76.08  320240  4.00  7235  1.14 
3  0.361  0.421  0.391  84.27  640480  0.80  8731  3.35 
MVSNet[yao2018mvsnet_depth1]  0.456  0.646  0.551  71.60  288216  2.65  10805  1.05 
Because of the continuity and flexibility of point representation, the refinement and densification can be performed iteratively on former predictions to give denser and more accurate predictions. While the model is trained using iterations, we test the model using iteration ranging from 0 to 3. For each iteration, we upsample the point cloud and decrease the depth interval of point hypotheses simultaneously, enabling the network to capture more detailed features. We compare the reconstruction quality, depth map resolution, GPU memory consumption and runtime at different iterations, along with performance reported by stateoftheart methods in Table 2. The reconstruction quality improves significantly with multiple iterations, which verifies the effectiveness of our methods. Note that our method already outperforms the stateoftheart after the second iteration. Qualitative results are shown in Figure 7.
4.5 Ablation study
In this section we provide ablation experiments and quantitative analysis to evaluate the strengths and limitations of the key components in our framework. For all the following studies, experiments are performed and evaluated on the DTU dataset, and both accuracy and completeness are used to measure the reconstruction quality. We set the iteration number to , and all other experiment settings are the same as subsection 4.2.
Edge Convolution
By replacing the edge convolution ( 4) with geometryunaware feature aggregation:
(7) 
where the features of neighbor points are treated equally with no regard for their geometric relationship to the centroid point, the reconstruction quality drops significantly as shown in Table 3, which illustrates the importance of local neighborhood relationship information (captured by ) for feature aggregation.
Euclidean Nearest Neighbour
In this part, we construct the directed graph using points belonging to adjacent pixels in the reference image, instead of searching the NN points, which leads to decreased reconstruction quality. The reason is that, for images of 3D scenes, nearby pixels may correspond to distant objects due to occlusion. Therefore, using neighboring points in the image space may aggregate irrelevant features for depth residual prediction, leading to descending performance.
Feature Pyramid
In this part, point cloud only fetches features from the last layer of the feature map, instead of from the whole feature pyramid. As shown in Table 3, in contrast to the relatively stable performance for changing edge convolution strategies as discussed above, the drop will be significant in the absence of the other two components, which demonstrates the effectiveness of the leveraging context information at different scales for feature fetching.
4.6 Reliance on initial depth maps
Our method uses stateoftheart approaches to get a coarse depth map prediction, which is then iteratively refined by predicting depth residuals. We found that our approach is robust to noisy initial depth estimation in a certain range through the following experiments. We added Gaussian noise of different scales to the initial depth map and evaluated the reconstruction error. Figure 8 shows that the error increases slowly and is smaller than MVSNet within 6mm noise.
EDGE  EUCNN  PYR  Acc. (mm)  Comp. (mm) 

✓  ✓  ✓  0.448  0.487 
✓  ✓  ✕  0.455  0.489 
✓  ✕  ✓  0.455  0.492 
✕  ✓  ✓  0.501  0.518 
✓  ✕  ✕  0.475  0.504 
✕  ✓  ✕  0.574  0.565 
✕  ✕  ✓  0.529  0.532 
Acc. (mm)  Comp. (mm)  Overall (mm)  

PUNet [yu2018punet]  1.220  0.667  0.943 
Ours  0.361  0.421  0.391 
4.7 Comparison to point cloud upsampling
Our work can also be considered as a datadriven point cloud upsampling method with assisting information from reference views. Therefore, we compare our method with PUNet [yu2018punet], where multilevel features are extracted from the coarse point cloud to reconstruct an upsampled point cloud.
We use the same coarse depth prediction network as in our model, and train PUNet to upsample the coarse point cloud. We use the same joint loss as mentioned in their paper, which consists of two losses — the Earth Mover’s distance (EMD) [fan2017point] loss between the predicted point cloud and the reference groundtruth point cloud and a repulsion loss. For evaluation, the PUNet is applied on the coarse predicted point cloud twice to generate a denser point cloud with times more points. Quantitative result is shown in Table 4. Our PointMVSNet can generate a more accurate point cloud from the coarse one by inducing flow for each point from observation of context information in multiview images.
4.8 Foveated depth inference
The pointbased network architecture enables us to process an arbitrary number of points. Therefore, instead of upsampling and refining the whole depth map, we can choose to only infer the depth in the region of interest (ROI) based on the input image or the predicted coarse depth map. As shown in Figure 9, we generate a point cloud of three different density levels by only upsampling and refining the ROI in the previous stage.
4.9 Generalizability of the PointFlow Module
In order to evaluate the generalizability of our PointFlow module, we test it on the Tanks and Temples intermediate dataset [knapitsch2017tanks_and], which is a large outdoor dataset captured in complex environments. We first generate coarse depth maps using MVSNet [yao2018mvsnet_depth1], and then apply our PointFlow module to refine them. The fscore increases from to (larger is better) and the rank rises from to (lower is better, date: Mar. 22, 2019). Reconstructed point clouds are shown in supplementary materials.
5 Conclusion
We present a novel pointbased architecture for highresolution multiview stereo reconstruction. Instead of building a highresolution cost volume, our proposed PointMVSNet processes the scene as a point cloud directly, which reduces unnecessary computation and preserves the spatial continuity. Experiments show that PointMVSNet is able to produce highquality reconstruction point clouds on benchmarks. Additionally, PointMVSNet is applicable to foveated depth inference to greatly reducing computation, which cannot be easily implemented for costvolumebased methods.
Acknowledgement
The authors gratefully acknowledge the support of an NSF grant IIS1764078, gifts from Qualcomm, Adobe and support from DMAI corporations.
References
6 Additional Ablation Study
6.1 Number of Point Hypotheses
In this section, we conduct an ablation study to verify the influence of the number of point hypotheses. In the main paper, we choose for both the training and evaluation. We change to and , and conduct the evaluation on the DTU evaluation set [aanaes2016LargeScaleDataMultipleView]. Table 5 shows the comparison result. Our proposed algorithm achieves best reconstruction quality in terms of completeness and overall quality when the number of point hypotheses is .
Point Hypotheses  Acc.(mm)  Comp.(mm)  Overall(mm) 

1  0.442  0.515  0.479 
2  0.448  0.487  0.468 
3  0.468  0.499  0.484 
6.2 Number of Views
In this section, we study the influence of the number of input views . Utilizing a variancebased cost metric, our PointMVSNet can process an arbitrary number of input views. Although the model is trained using , we can evaluate the model using either on the DTU evaluation set [aanaes2016LargeScaleDataMultipleView]. Table 6 demonstrates that the reconstruction quality improves with an increasing number of input views, which is consistent with common knowledge of MVS reconstruction.
Number of Views  Acc.(mm)  Comp. (mm)  Overall(mm) 

2  0.462  0.604  0.533 
3  0.448  0.507  0.478 
5  0.448  0.487  0.468 
7 Memory, runtime and overhead of kNN
Table 7 compares our memory usage and running speed against MVSNet. Our method is able to predict different resolutions of depth maps at different speed by changing the iterations. Naïve kNN of point cloud of points can be memoryconsuming with complexity. However, we notice the kNN of a point tend to come from its nearby 2D pixels in the depth map. By leveraging this fact and taking the hypothetical points into consideration, we restrict the kNN search of each point from the whole point cloud to its neighborhood. Furthermore, we parallel the distance computation by using a fixed weight 3D kernel.
Iter. 

Resolution 




0  0.726  160120  7219  0.34  
1  0.712  160120  7221  0.61  
2  0.468  320240  7235  1.14  
2^{†}  0.474  320240  7233  0.97  
3  0.391  640480  8731  3.35  
MVSNet  0.551  288216  10805  1.05 
8 Postprocessing
In this section, we describe the postprocessing procedure in details. Similar to MVSNet [yao2018mvsnet_depth1], our postprocessing is composed of three steps: photometric filtering, geometric consistency filtering, and depth fusion.
For photometric filtering, we use predicted probability of the most likely depth layer as the confidence metric and filter out points whose confidence is below a threshold. The filtering threshold is set to 0.5 and 0.2 for coarse and our PointFlow stage, respectively. For geometric consistency, we calculate the discrepancy of predicted depths among multiview predictions through reverseprojection. Points with discrepancy larger than 0.12mm are discarded. For depth fusion, we take average value of all reprojected depths of each point in visible views as the final depth prediction and produce the 3D point cloud.