Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding

11/26/2022
by   Priya Sundaresan, et al.
0

Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71 material, datasets, code, and videos are available on our website: https://sites.google.com/view/hapticvisualnet-corl22/home

READ FULL TEXT

page 2

page 5

page 11

page 12

page 13

research
08/19/2019

Adaptive Robot-Assisted Feeding: An Online Learning Framework for Acquiring Previously-Unseen Food Items

Successful robot-assisted feeding requires bite acquisition of a wide va...
research
11/22/2021

Balancing Efficiency and Comfort in Robot-Assisted Bite Transfer

Robot-assisted feeding in household environments is challenging because ...
research
01/06/2021

Playing with Food: Learning Food Item Representations through Interactive Exploration

A key challenge in robotic food manipulation is modeling the material pr...
research
09/27/2019

Leveraging Multimodal Haptic Sensory Data for Robust Cutting

Cutting is a common form of manipulation when working with divisible obj...
research
11/23/2022

In-Mouth Robotic Bite Transfer with Visual and Haptic Sensing

Assistance during eating is essential for those with severe mobility iss...
research
11/05/2020

Leveraging Post Hoc Context for Faster Learning in Bandit Settings with Applications in Robot-Assisted Feeding

Autonomous robot-assisted feeding requires the ability to acquire a wide...
research
08/19/2019

Learning from failures in robot-assisted feeding: Using online learning to develop manipulation strategies for bite acquisition

Successful robot-assisted feeding requires bite acquisition of a wide va...

Please sign up or login with your details

Forgot password? Click here to reset