Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset

by   Sagi Eppel, et al.

We present TransProteus, a dataset, and methods for predicting the 3D structure, masks, and properties of materials, liquids, and objects inside transparent vessels from a single image without prior knowledge of the image source and camera parameters. Manipulating materials in transparent containers is essential in many fields and depends heavily on vision. This work supplies a new procedurally generated dataset consisting of 50k images of liquids and solid objects inside transparent containers. The image annotations include 3D models, material properties (color/transparency/roughness...), and segmentation masks for the vessel and its content. The synthetic (CGI) part of the dataset was procedurally generated using 13k different objects, 500 different environments (HDRI), and 1450 material textures (PBR) combined with simulated liquids and procedurally generated vessels. In addition, we supply 104 real-world images of objects inside transparent vessels with depth maps of both the vessel and its content. We propose a camera agnostic method that predicts 3D models from an image as an XYZ map. This allows the trained net to predict the 3D model as a map with XYZ coordinates per pixel without prior knowledge of the image source. To calculate the training loss, we use the distance between pairs of points inside the 3D model instead of the absolute XYZ coordinates. This makes the loss function translation invariant. We use this to predict 3D models of vessels and their content from a single image. Finally, we demonstrate a net that uses a single image to predict the material properties of the vessel content and surface.


page 1

page 3

page 4

page 9

page 11

page 14


Colored Transparent Object Matting from a Single Image Using Deep Learning

This paper proposes a deep learning based method for colored transparent...

One-shot recognition of any material anywhere using contrastive learning with physics-based rendering

We present MatSim: a synthetic dataset, a benchmark, and a method for co...

What Is Around The Camera?

How much does a single image reveal about the environment it was taken i...

Self-supervised Transparent Liquid Segmentation for Robotic Pouring

Liquid state estimation is important for robotics tasks such as pouring;...

TOM-Net: Learning Transparent Object Matting from a Single Image

This paper addresses the problem of transparent object matting. Existing...

Tracing liquid level and material boundaries in transparent vessels using the graph cut computer vision approach

Detection of boundaries of materials stored in transparent vessels is es...

Please sign up or login with your details

Forgot password? Click here to reset