Real-time Soft Robot 3D Proprioception via Deep Vision-based Sensing

by   Ruoyu Wang, et al.

The soft robots are welcomed in many robotic applications because of their high flexibility, which also poses a long-standing challenge on their proprioception, or measuring the real-time 3D shapes of the soft robots from internal sensors. The challenge exists in both the sensor design and robot modeling. In this paper, we propose a framework to measure the real-time high-resolution 3D shapes of soft robots. The framework is based on an embedded camera to capture the inside/outside patterns of the robots under different loading conditions, and a CNN to produce a latent code representing the robot state, which can then be used to reconstruct the 3D shape using a neural network improved from FoldingNet. We tested the framework on four different soft actuators with various kinds of deformations, and achieved real-time computation (<2ms/frame) for robust shape estimation of high precision (<5 relative error for 2025 points) at an arbitrary resolution. We believe the method could be widely applied to different designs of soft robots for proprioception, and enabling people to better control them under complicated environments. Our code is available at


page 1

page 5

page 6

page 7

page 8


Exoskeleton-covered soft finger with vision-based proprioception and exteroception

Soft robots offer significant advantages in adaptability, safety, and de...

Sensing and Reconstruction of 3D Deformation on Pneumatic Soft Robots

Real-time proprioception is a challenging problem for soft robots, which...

Model-Based Control of Planar Piezoelectric Inchworm Soft Robot for Crawling in Constrained Environments

Soft robots have drawn significant attention recently for their ability ...

Toward Zero-Shot Sim-to-Real Transfer Learning for Pneumatic Soft Robot 3D Proprioceptive Sensing

Pneumatic soft robots present many advantages in manipulation tasks. Not...

MoSS: Monocular Shape Sensing for Continuum Robots

Continuum robots are promising candidates for interactive tasks in vario...

Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering

State estimation from measured data is crucial for robotic applications ...

ViSE: Vision-Based 3D Real-Time Shape Estimation of Continuously Deformable Robots

The precise control of soft and continuum robots requires knowledge of t...

Please sign up or login with your details

Forgot password? Click here to reset