ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer

by   Ruohan Gao, et al.

Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rather unrealistic. ObjectFolder 1.0 is a recent dataset that introduces 100 virtualized objects with visual, acoustic, and tactile sensory data. However, the dataset is small in scale and the multisensory data is of limited quality, hampering generalization to real-world scenarios. We present ObjectFolder 2.0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1.0 in three aspects. First, our dataset is 10 times larger in the amount of objects and orders of magnitude faster in rendering time. Second, we significantly improve the multisensory rendering quality for all three modalities. Third, we show that models learned from virtual objects in our dataset successfully transfer to their real-world counterparts in three challenging tasks: object scale estimation, contact localization, and shape reconstruction. ObjectFolder 2.0 offers a new path and testbed for multisensory learning in computer vision and robotics. The dataset is available at https://github.com/rhgao/ObjectFolder.


page 5

page 6


Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

Traditional approaches for learning 3D object categories have been predo...

ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations

Multisensory object-centric perception, reasoning, and interaction have ...

Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer

Domain randomisation is a very popular method for visual sim-to-real tra...

ABO: Dataset and Benchmarks for Real-World 3D Object Understanding

We introduce Amazon-Berkeley Objects (ABO), a new large-scale dataset of...

A System for General In-Hand Object Re-Orientation

In-hand object reorientation has been a challenging problem in robotics ...

ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes

The two popular datasets ScanRefer [16] and ReferIt3D [3] connect natura...

ShapeGlot: Learning Language for Shape Differentiation

In this work we explore how fine-grained differences between the shapes ...

Code Repositories


ObjectFolder Dataset

view repo

Please sign up or login with your details

Forgot password? Click here to reset