Log In Sign Up

daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

by   Yidan Qin, et al.

This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85 long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.


page 2

page 14

page 15


Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Many tasks in robot-assisted surgeries (RAS) can be represented by finit...

Organ Shape Sensing using Pneumatically Attachable Flexible Rails in Robotic-Assisted Laparoscopic Surgery

In robotic-assisted partial nephrectomy, surgeons remove a part of a kid...

Unsupervised identification of surgical robotic actions from small non homogeneous datasets

Robot-assisted surgery is an established clinical practice. The automati...

Unsupervised Learning for Surgical Motion by Learning to Predict the Future

We show that it is possible to learn meaningful representations of surgi...

Robotic-assisted Ultrasound for Fetal Imaging: Evolution from Single-arm to Dual-arm System

The development of robotic-assisted extracorporeal ultrasound systems ha...