How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning

06/04/2021
by   Namiko Saito, et al.
0

Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.

READ FULL TEXT

page 5

page 6

page 7

research
09/23/2018

Detecting Features of Tools, Objects, and Actions from Effects in a Robot using Deep Learning

We propose a tool-use model that can detect the features of tools, targe...
research
10/01/2015

Multimodal Hierarchical Dirichlet Process-based Active Perception

In this paper, we propose an active perception method for recognizing ob...
research
03/07/2023

Cross-Tool and Cross-Behavior Perceptual Knowledge Transfer for Grounded Object Recognition

Humans learn about objects via interaction and using multiple perception...
research
06/29/2022

Deep Active Visual Attention for Real-time Robot Motion Generation: Emergence of Tool-body Assimilation and Adaptive Tool-use

Sufficiently perceiving the environment is a critical factor in robot mo...
research
10/13/2017

Transfer of Tool Affordance and Manipulation Cues with 3D Vision Data

Future service robots working in human environments, such as kitchens, w...
research
06/18/2022

ToolTango: Common sense Generalization in Predicting Sequential Tool Interactions for Robot Plan Synthesis

Robots assisting us in environments such as factories or homes must lear...
research
02/24/2017

Sequence-based Multimodal Apprenticeship Learning For Robot Perception and Decision Making

Apprenticeship learning has recently attracted a wide attention due to i...

Please sign up or login with your details

Forgot password? Click here to reset