Fine-grained Affordance Annotation for Egocentric Hand-Object Interaction Videos

02/07/2023
by   Zecheng Yu, et al.
0

Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, the definition of affordance in existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordance labels and introducing the concept of mechanical action to represent the action possibilities between two objects. We provide new annotations by applying this scheme to the EPIC-KITCHENS dataset and test our annotation with tasks such as affordance recognition, hand-object interaction hotspots prediction, and cross-domain evaluation of affordance. The results show that models trained with our annotation can distinguish affordance from other concepts, predict fine-grained interaction possibilities on objects, and generalize through different domains.

READ FULL TEXT

page 3

page 6

page 7

research
06/11/2022

Precise Affordance Annotation for Egocentric Action Video Datasets

Object affordance is an important concept in human-object interaction, p...
research
06/20/2018

Classifying Object Manipulation Actions based on Grasp-types and Motion-Constraints

In this work, we address a challenging problem of fine-grained and coars...
research
11/25/2019

Forecasting Human Object Interaction: Joint Prediction of Motor Attention and Egocentric Activity

We address the challenging task of anticipating human-object interaction...
research
09/16/2023

AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose

How human interact with objects depends on the functional roles of the t...
research
08/20/2020

Object Properties Inferring from and Transfer for Human Interaction Motions

Humans regularly interact with their surrounding objects. Such interacti...
research
03/04/2021

FootApp: an AI-Powered System for Football Match Annotation

In the last years, scientific and industrial research has experienced a ...
research
09/30/2017

Fine-grained Event Learning of Human-Object Interaction with LSTM-CRF

Event learning is one of the most important problems in AI. However, not...

Please sign up or login with your details

Forgot password? Click here to reset