Intermittent Visual Servoing: Efficiently Learning Policies Robust to Instrument Changes for High-precision Surgical Manipulation

by   Samuel Paradis, et al.

Automation of surgical tasks using cable-driven robots is challenging due to backlash, hysteresis, and cable tension, and these issues are exacerbated as surgical instruments must often be changed during an operation. In this work, we propose a framework for automation of high-precision surgical tasks by learning sample efficient, accurate, closed-loop policies that operate directly on visual feedback instead of robot encoder estimates. This framework, which we call intermittent visual servoing (IVS), intermittently switches to a learned visual servo policy for high-precision segments of repetitive surgical tasks while relying on a coarse open-loop policy for the segments where precision is not necessary. To compensate for cable-related effects, we apply imitation learning to rapidly train a policy that maps images of the workspace and instrument from a top-down RGB camera to small corrective motions. We train the policy using only 180 human demonstrations that are roughly 2 seconds each. Results on a da Vinci Research Kit suggest that combining the coarse policy with half a second of corrections from the learned policy during each high-precision segment improves the success rate on the Fundamentals of Laparoscopic Surgery peg transfer task from 72.9 47.2 contexts we studied, IVS attains the highest published success rates for automated surgical peg transfer and is significantly more reliable than previous techniques when instruments are changed. Supplementary material is available at



page 1

page 2

page 4

page 5


Superhuman Surgical Peg Transfer Using Depth-Sensing and Deep Recurrent Neural Networks

We consider the automation of the well-known peg-transfer task from the ...

Learning to Localize, Grasp, and Hand Over Unmodified Surgical Needles

Robotic Surgical Assistants (RSAs) are commonly used to perform minimall...

VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation

Robotic fabric manipulation has applications in cloth and cable manageme...

Bayesian hierarchical nonlinear modelling of intra-abdominal volume during pneumoperitoneum for laparoscopic surgery

Laparoscopy is an operation carried out in the abdomen or pelvis through...

Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot

Recent advances in depth-sensing have significantly increased accuracy, ...

VisuoSpatial Foresight for Physical Sequential Fabric Manipulation

Robotic fabric manipulation has applications in home robotics, textiles,...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.