DeepAI AI Chat
Log In Sign Up

Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos

by   Zecheng Yu, et al.
The University of Tokyo

Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, the definition of affordance in existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordance labels and introducing the concept of mechanical action to represent the action possibilities between two objects. We provide new annotations by applying this scheme to the EPIC-KITCHENS dataset and test our annotation with tasks such as affordance recognition, hand-object interaction hotspots prediction, and cross-domain evaluation of affordance. The results show that models trained with our annotation can distinguish affordance from other concepts, predict fine-grained interaction possibilities on objects, and generalize through different domains.


page 3

page 6

page 7


Precise Affordance Annotation for Egocentric Action Video Datasets

Object affordance is an important concept in human-object interaction, p...

Classifying Object Manipulation Actions based on Grasp-types and Motion-Constraints

In this work, we address a challenging problem of fine-grained and coars...

Forecasting Human Object Interaction: Joint Prediction of Motor Attention and Egocentric Activity

We address the challenging task of anticipating human-object interaction...

Weakly Supervised Fine-Grained Image Categorization

In this paper, we categorize fine-grained images without using any objec...

Object Properties Inferring from and Transfer for Human Interaction Motions

Humans regularly interact with their surrounding objects. Such interacti...

FootApp: an AI-Powered System for Football Match Annotation

In the last years, scientific and industrial research has experienced a ...

Actionable Email Intent Modeling with Reparametrized RNNs

Emails in the workplace are often intentional calls to action for its re...