Affordance Transfer Learning for Human-Object Interaction Detection

by   Zhi Hou, et al.

Reasoning the human-object interactions (HOI) is essential for deeper scene understanding, while object affordances (or functionalities) are of great importance for human to discover unseen HOIs with novel objects. Inspired by this, we introduce an affordance transfer learning approach to jointly detect HOIs with novel objects and recognize affordances. Specifically, HOI representations can be decoupled into a combination of affordance and object representations, making it possible to compose novel interactions by combining affordance representations and novel object representations from additional images, i.e. transferring the affordance to novel objects. With the proposed affordance transfer learning, the model is also capable of inferring the affordances of novel objects from known affordance representations. The proposed method can thus be used to 1) improve the performance of HOI detection, especially for the HOIs with unseen objects; and 2) infer the affordances of novel objects. Experimental results on two datasets, HICO-DET and HOI-COCO (from V-COCO), demonstrate significant improvements over recent state-of-the-art methods for HOI detection and object affordance detection. Code is available at


page 1

page 3

page 8

page 10

page 12


Detecting Human-Object Interaction via Fabricated Compositional Learning

Human-Object Interaction (HOI) detection, inferring the relationships be...

Exploiting Scene Graphs for Human-Object Interaction Detection

Human-Object Interaction (HOI) detection is a fundamental visual task ai...

Visual Compositional Learning for Human-Object Interaction Detection

Human-Object interaction (HOI) detection aims to localize and infer rela...

Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection

Detecting Human-Object Interaction (HOI) in images is an important step ...

OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction

Learning how humans manipulate objects requires machines to acquire know...

Effective Actor-centric Human-object Interaction Detection

While Human-Object Interaction(HOI) Detection has achieved tremendous ad...

Deep Transfer Learning for Multiple Class Novelty Detection

We propose a transfer learning-based solution for the problem of multipl...

Code Repositories