Shaping embodied agent behavior with activity-context priors from egocentric video

10/14/2021
by   Tushar Nagarajan, et al.
0

Complex physical tasks entail a sequence of object interactions, each with its own preconditions – which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity-context priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning. Project page: http://vision.cs.utexas.edu/projects/ego-rewards/

READ FULL TEXT

page 2

page 5

page 6

page 8

page 14

page 15

research
08/21/2020

Learning Affordance Landscapes forInteraction Exploration in 3D Environments

Embodied agents operating in human spaces must be able to master how the...
research
02/01/2022

DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

Dexterous multi-fingered robotic hands have a formidable action space, y...
research
11/24/2020

Foundations of the Socio-physical Model of Activities (SOMA) for Autonomous Robotic Agents

In this paper, we present foundations of the Socio-physical Model of Act...
research
09/03/2020

Dexterous Robotic Grasping with Object-Centric Visual Affordances

Dexterous robotic hands are appealing for their agility and human-like m...
research
06/17/2019

An IoT Based Framework For Activity Recognition Using Deep Learning Technique

Activity recognition is the ability to identify and recognize the action...
research
12/30/2019

Learning Predictive Models From Observation and Interaction

Learning predictive models from interaction with the world allows an age...

Please sign up or login with your details

Forgot password? Click here to reset