SIMstack: A Generative Shape and Instance Model for Unordered Object Stacks

by   Zoe Landgraf, et al.

By estimating 3D shape and instances from a single view, we can capture information about an environment quickly, without the need for comprehensive scanning and multi-view fusion. Solving this task for composite scenes (such as object stacks) is challenging: occluded areas are not only ambiguous in shape but also in instance segmentation; multiple decompositions could be valid. We observe that physics constrains decomposition as well as shape in occluded regions and hypothesise that a latent space learned from scenes built under physics simulation can serve as a prior to better predict shape and instances in occluded regions. To this end we propose SIMstack, a depth-conditioned Variational Auto-Encoder (VAE), trained on a dataset of objects stacked under physics simulation. We formulate instance segmentation as a centre voting task which allows for class-agnostic detection and doesn't require setting the maximum number of objects in the scene. At test time, our model can generate 3D shape and instance segmentation from a single depth view, probabilistically sampling proposals for the occluded region from the learned latent space. Our method has practical applications in providing robots some of the ability humans have to make rapid intuitive inferences of partially observed scenes. We demonstrate an application for precise (non-disruptive) object grasping of unknown objects from a single depth view.



There are no comments yet.


page 1

page 5

page 7

page 8

page 13

page 15

page 16


Amodal Segmentation Based on Visible Region Segmentation and Shape Prior

Almost all existing amodal segmentation methods make the inferences of o...

Descriptor-Free Multi-View Region Matching for Instance-Wise 3D Reconstruction

This paper proposes a multi-view extension of instance segmentation with...

Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects

We present a robotic system for picking a target from a pile of objects ...

Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation

Despite rapid progress in scene segmentation in recent years, 3D segment...

Occluded Video Instance Segmentation

Can our video understanding systems perceive objects when a heavy occlus...

Diverse Plausible Shape Completions from Ambiguous Depth Images

We propose PSSNet, a network architecture for generating diverse plausib...

A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities

Medical imaging only indirectly measures the molecular identity of the t...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.