RoboTHOR: An Open Simulation-to-Real Embodied AI Platform

by   Matt Deitke, et al.

Visual recognition ecosystems (e.g. ImageNet, Pascal, COCO) have undeniably played a prevailing role in the evolution of modern computer vision. We argue that interactive and embodied visual AI has reached a stage of development similar to visual recognition prior to the advent of these ecosystems. Recently, various synthetic environments have been introduced to facilitate research in embodied AI. Notwithstanding this progress, the crucial question of how well models trained in simulation generalize to reality has remained largely unanswered. The creation of a comparable ecosystem for simulation-to-real embodied AI presents many challenges: (1) the inherently interactive nature of the problem, (2) the need for tight alignments between real and simulated worlds, (3) the difficulty of replicating physical conditions for repeatable experiments, (4) and the associated cost. In this paper, we introduce RoboTHOR to democratize research in interactive and embodied visual AI. RoboTHOR offers a framework of simulated environments paired with physical counterparts to systematically explore and overcome the challenges of simulation-to-real transfer, and a platform where researchers across the globe can remotely test their embodied models in the physical world. As a first benchmark, our experiments show there exists a significant gap between the performance of models trained in simulation when they are tested in both simulations and their carefully constructed physical analogs. We hope that RoboTHOR will spur the next stage of evolution in embodied computer vision. RoboTHOR can be accessed at the following link:


page 1

page 4

page 5

page 8


AllenAct: A Framework for Embodied AI Research

The domain of Embodied AI, in which agents learn to complete tasks throu...

Are We Making Real Progress in Simulated Environments? Measuring the Sim2Real Gap in Embodied Visual Navigation

Does progress in simulation translate to progress in robotics? Specifica...

Bridging the Domain Gap between Synthetic and Real-World Data for Autonomous Driving

Modern autonomous systems require extensive testing to ensure reliabilit...

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

Massive datasets and high-capacity models have driven many recent advanc...

Alexa Arena: A User-Centric Interactive Platform for Embodied AI

We introduce Alexa Arena, a user-centric simulation platform for Embodie...

Out of the Box: Embodied Navigation in the Real World

The research field of Embodied AI has witnessed substantial progress in ...

Dynamic Object Comprehension: A Framework For Evaluating Artificial Visual Perception

Augmented and Mixed Reality are emerging as likely successors to the mob...

Please sign up or login with your details

Forgot password? Click here to reset