DeepAI AI Chat
Log In Sign Up

Learning to Act Properly: Predicting and Explaining Affordances from Images

by   Ching-Yao Chuang, et al.

We address the problem of affordance reasoning in diverse scenes that appear in the real world. Affordances relate the agent's actions to their effects when taken on the surrounding objects. In our work, we take the egocentric view of the scene, and aim to reason about action-object affordances that respect both the physical world as well as the social norms imposed by the society. We also aim to teach artificial agents why some actions should not be taken in certain situations, and what would likely happen if these actions would be taken. We collect a new dataset that builds upon ADE20k, referred to as ADE-Affordance, which contains annotations enabling such rich visual reasoning. We propose a model that exploits Graph Neural Networks to propagate contextual information from the scene in order to perform detailed affordance reasoning about each object. Our model is showcased through various ablation studies, pointing to successes and challenges in this complex task.


page 1

page 4

page 5

page 7

page 8


Object-oriented state editing for HRL

We introduce agents that use object-oriented reasoning to consider alter...

Visual Reaction: Learning to Play Catch with Your Drone

In this paper we address the problem of visual reaction: the task of int...

The "something something" video database for learning and evaluating visual common sense

Neural networks trained on datasets such as ImageNet have led to major a...

MS-LaTTE: A Dataset of Where and When To-do Tasks are Completed

Tasks are a fundamental unit of work in the daily lives of people, who a...

Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences

In social settings, much of human behavior is governed by unspoken rules...

Improving Object Permanence using Agent Actions and Reasoning

Object permanence in psychology means knowing that objects still exist e...

Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text

Our goal is to better comprehend procedural text, e.g., a paragraph abou...