Semantically Informed Multiview Surface Refinement

by   Maros Blaha, et al.

We present a method to jointly refine the geometry and semantic segmentation of 3D surface meshes. Our method alternates between updating the shape and the semantic labels. In the geometry refinement step, the mesh is deformed with variational energy minimization, such that it simultaneously maximizes photo-consistency and the compatibility of the semantic segmentations across a set of calibrated images. Label-specific shape priors account for interactions between the geometry and the semantic labels in 3D. In the semantic segmentation step, the labels on the mesh are updated with MRF inference, such that they are compatible with the semantic segmentations in the input images. Also, this step includes prior assumptions about the surface shape of different semantic classes. The priors induce a tight coupling, where semantic information influences the shape update and vice versa. Specifically, we introduce priors that favor (i) adaptive smoothing, depending on the class label; (ii) straightness of class boundaries; and (iii) semantic labels that are consistent with the surface orientation. The novel mesh-based reconstruction is evaluated in a series of experiments with real and synthetic data. We compare both to state-of-the-art, voxel-based semantic 3D reconstruction, and to purely geometric mesh refinement, and demonstrate that the proposed scheme yields improved 3D geometry as well as an improved semantic segmentation.


page 1

page 6

page 8


Active Label Refinement for Semantic Segmentation of Satellite Images

Remote sensing through semantic segmentation of satellite images contrib...

Improving Semantic Image Segmentation via Label Fusion in Semantically Textured Meshes

Models for semantic segmentation require a large amount of hand-labeled ...

Virtual Multi-view Fusion for 3D Semantic Segmentation

Semantic segmentation of 3D meshes is an important problem for 3D scene ...

A Classification Refinement Strategy for Semantic Segmentation

Based on the observation that semantic segmentation errors are partially...

Compensation Learning in Semantic Segmentation

Label noise and ambiguities between similar classes are challenging prob...

A Data-driven Prior on Facet Orientation for Semantic Mesh Labeling

Mesh labeling is the key problem of classifying the facets of a 3D mesh ...

Semantic Validation in Structure from Motion

The Structure from Motion (SfM) challenge in computer vision is the proc...

Please sign up or login with your details

Forgot password? Click here to reset