BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

by   Sanjana Srivastava, et al.

We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation. These activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world. Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation. BEHAVIOR addresses these with three innovations. First, we propose an object-centric, predicate logic-based description language for expressing an activity's initial and goal conditions, enabling generation of diverse instances for any activity. Second, we identify the simulator-agnostic features required by an underlying environment to support BEHAVIOR, and demonstrate its realization in one such simulator. Third, we introduce a set of metrics to measure task progress and efficiency, absolute and relative to human demonstrators. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth. Our experiments demonstrate that even state of the art embodied AI solutions struggle with the level of realism, diversity, and complexity imposed by the activities in our benchmark. We make BEHAVIOR publicly available at to facilitate and calibrate the development of new embodied AI solutions.


page 5

page 15

page 16

page 24

page 27

page 28

page 30

page 31


iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks

Recent research in embodied AI has been boosted by the use of simulation...

Towards understanding and modelling office daily life

Measuring and modeling human behavior is a very complex task. In this pa...

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

One of the main challenges of advancing task-oriented learning such as v...

Automated acquisition of structured, semantic models of manipulation activities from human VR demonstration

In this paper we present a system capable of collecting and annotating, ...

Development of a wheelchair simulator for children with multiple disabilities

Virtual reality allows to create situations which can be experimented un...

Rearrangement: A Challenge for Embodied AI

We describe a framework for research and evaluation in Embodied AI. Our ...

Shaping embodied agent behavior with activity-context priors from egocentric video

Complex physical tasks entail a sequence of object interactions, each wi...