Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction

06/25/2019
by   Nikolay Savinov, et al.
2

Dense semantic 3D reconstruction is typically formulated as a discrete or continuous problem over label assignments in a voxel grid, combining semantic and depth likelihoods in a Markov Random Field framework. The depth and semantic information is incorporated as a unary potential, smoothed by a pairwise regularizer. However, modelling likelihoods as a unary potential does not model the problem correctly leading to various undesirable visibility artifacts. We propose to formulate an optimization problem that directly optimizes the reprojection error of the 3D model with respect to the image estimates, which corresponds to the optimization over rays, where the cost function depends on the semantic class and depth of the first occupied voxel along the ray. The 2-label formulation is made feasible by transforming it into a graph-representable form under QPBO relaxation, solvable using graph cut. The multi-label problem is solved by applying alpha-expansion using the same relaxation in each expansion move. Our method was indeed shown to be feasible in practice, running comparably fast to the competing methods, while not suffering from ray potential approximation artifacts.

READ FULL TEXT

page 7

page 8

research
04/11/2016

Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

We propose an approach for dense semantic 3D reconstruction which uses a...
research
03/23/2023

Semantic Ray: Learning a Generalizable Semantic Field with Cross-Reprojection Attention

In this paper, we aim to learn a semantic radiance field from multiple s...
research
03/28/2017

Solving Non-parametric Inverse Problem in Continuous Markov Random Field using Loopy Belief Propagation

In this paper, we address the inverse problem, or the statistical machin...
research
06/24/2015

Kernel Cuts: MRF meets Kernel & Spectral Clustering

We propose a new segmentation model combining common regularization ener...
research
09/13/2018

Efficient Graph Cut Optimization for Full CRFs with Quantized Edges

Fully connected pairwise Conditional Random Fields (Full-CRF) with Gauss...
research
12/06/2020

Depth Completion using Piecewise Planar Model

A depth map can be represented by a set of learned bases and can be effi...
research
12/03/2022

Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field

Ultra-fine entity typing (UFET) aims to predict a wide range of type phr...

Please sign up or login with your details

Forgot password? Click here to reset