Action sequencing using visual permutations

08/03/2020
by   Michael Burke, et al.
0

Humans can easily reason about the sequence of high level actions needed to complete tasks, but it is particularly difficult to instil this ability in robots trained from relatively few examples. This work considers the task of neural action sequencing conditioned on a single reference visual state. This task is extremely challenging as it is not only subject to the significant combinatorial complexity that arises from large action sets, but also requires a model that can perform some form of symbol grounding, mapping high dimensional input data to actions, while reasoning about action relationships. Drawing on human cognitive abilities to rearrange objects in scenes to create new configurations, we take a permutation perspective and argue that action sequencing benefits from the ability to reason about both permutations and ordering concepts. Empirical analysis shows that neural models trained with latent permutations outperform standard neural architectures in constrained action sequencing tasks. Results also show that action sequencing using visual permutations is an effective mechanism to initialise and speed up traditional planning techniques and successfully scales to far greater action set sizes than models considered previously.

READ FULL TEXT

page 7

page 11

page 12

research
09/29/2021

Grounding Predicates through Actions

Symbols representing abstract states such as "dish in dishwasher" or "cu...
research
05/02/2021

Learning Visually Guided Latent Actions for Assistive Teleoperation

It is challenging for humans – particularly those living with physical d...
research
10/07/2022

See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction

Cognitive planning is the structural decomposition of complex tasks into...
research
03/30/2018

Visual Robot Task Planning

Prospection, the act of predicting the consequences of many possible fut...
research
12/21/2022

Deep set conditioned latent representations for action recognition

In recent years multi-label, multi-class video action recognition has ga...
research
08/30/2020

Action similarity judgment based on kinematic primitives

Understanding which features humans rely on – in visually recognizing ac...
research
02/07/2023

Learning Manner of Execution from Partial Corrections

Some actions must be executed in different ways depending on the context...

Please sign up or login with your details

Forgot password? Click here to reset