Cycle-Consistent Generative Rendering for 2D-3D Modality Translation

by   Tristan Aumentado-Armstrong, et al.

For humans, visual understanding is inherently generative: given a 3D shape, we can postulate how it would look in the world; given a 2D image, we can infer the 3D structure that likely gave rise to it. We can thus translate between the 2D visual and 3D structural modalities of a given object. In the context of computer vision, this corresponds to a learnable module that serves two purposes: (i) generate a realistic rendering of a 3D object (shape-to-image translation) and (ii) infer a realistic 3D shape from an image (image-to-shape translation). In this paper, we learn such a module while being conscious of the difficulties in obtaining large paired 2D-3D datasets. By leveraging generative domain translation methods, we are able to define a learning algorithm that requires only weak supervision, with unpaired data. The resulting model is not only able to perform 3D shape, pose, and texture inference from 2D images, but can also generate novel textured 3D shapes and renders, similar to a graphics pipeline. More specifically, our method (i) infers an explicit 3D mesh representation, (ii) utilizes example shapes to regularize inference, (iii) requires only an image mask (no keypoints or camera extrinsics), and (iv) has generative capabilities. While prior work explores subsets of these properties, their combination is novel. We demonstrate the utility of our learned representation, as well as its performance on image generation and unpaired 3D shape inference tasks.



There are no comments yet.


page 17

page 18

page 19

page 20

page 21

page 22


Visual Object Networks: Image Generation with Disentangled 3D Representation

Recent progress in deep generative models has led to tremendous breakthr...

Cartoon-to-real: An Approach to Translate Cartoon to Realistic Images using GAN

We propose a method to translate cartoon images to real world images usi...

Textured Neural Avatars

We present a system for learning full-body neural avatars, i.e. deep net...

Self-Supervised 2D Image to 3D Shape Translation with Disentangled Representations

We present a framework to translate between 2D image views and 3D object...

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

The advancement of generative radiance fields has pushed the boundary of...

SiCloPe: Silhouette-Based Clothed People

We introduce a new silhouette-based representation for modeling clothed ...

Unpaired Pose Guided Human Image Generation

This paper studies the task of full generative modelling of realistic im...

Code Repositories


Implementation of 2D-3D Cyclic Generative Renderer (3DV-2020).

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.