KINet: Keypoint Interaction Networks for Unsupervised Forward Modeling

02/18/2022
by   Alireza Rezazadeh, et al.
0

Object-centric representation is an essential abstraction for physical reasoning and forward prediction. Most existing approaches learn this representation through extensive supervision (e.g., object class and bounding box) although such ground-truth information is not readily accessible in reality. To address this, we introduce KINet (Keypoint Interaction Network) – an end-to-end unsupervised framework to reason about object interactions in complex systems based on a keypoint representation. Using visual observations, our model learns to associate objects with keypoint coordinates and discovers a graph representation of the system as a set of keypoint embeddings and their relations. It then learns an action-conditioned forward model using contrastive estimation to predict future keypoint states. By learning to perform physical reasoning in the keypoint space, our model automatically generalizes to scenarios with a different number of objects, and novel object geometries. Experiments demonstrate the effectiveness of our model to accurately perform forward prediction and learn plannable object-centric representations which can also be used in downstream model-based control tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/28/2018

Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

Object-based factorizations provide a useful level of abstraction for in...
research
06/19/2019

Unsupervised Learning of Object Keypoints for Perception and Control

The study of object representations in computer vision has primarily foc...
research
06/15/2021

End-to-End Learning of Keypoint Representations for Continuous Control from Images

In many control problems that include vision, optimal controls can be in...
research
12/09/2021

Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings

Dense object tracking, the ability to localize specific object points wi...
research
01/26/2023

Unsupervised Volumetric Animation

We propose a novel approach for unsupervised 3D animation of non-rigid d...
research
09/11/2023

Learning Geometric Representations of Objects via Interaction

We address the problem of learning representations from observations of ...
research
10/28/2019

Entity Abstraction in Visual Model-Based Reinforcement Learning

This paper tests the hypothesis that modeling a scene in terms of entiti...

Please sign up or login with your details

Forgot password? Click here to reset