Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

by   Yidan Qin, et al.

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4 state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.


page 2

page 9


daVinciNet: Joint Prediction of Motion and Surgical State in Robot-Assisted Surgery

This paper presents a technique to concurrently and jointly predict the ...

Learning Invariant Representation of Tasks for Robust Surgical State Estimation

Surgical state estimators in robot-assisted surgery (RAS) - especially t...

Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification

Recognition of surgical gesture is crucial for surgical skill assessment...

Unsupervised identification of surgical robotic actions from small non homogeneous datasets

Robot-assisted surgery is an established clinical practice. The automati...

Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference

Surgical context inference has recently garnered significant attention i...

Offline identification of surgical deviations in laparoscopic rectopexy

Objective: A median of 14.4 during surgery and a third of them are preve...

Surgery Scene Restoration for Robot Assisted Minimally Invasive Surgery

Minimally invasive surgery (MIS) offers several advantages including min...

Please sign up or login with your details

Forgot password? Click here to reset