Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video

by   Shan Lin, et al.

Deep learning-based methods have achieved promising results on surgical instrument segmentation. However, the high computation cost may limit the applications of deep models to time-sensitive tasks such as online surgical video analysis for robotic-assisted surgery. Also, current performance may still suffer from challenging conditions in surgical images such as various lighting conditions and the presence of blood. We propose a novel Multi-frame Feature Aggregation (MFFA) module that leverages information of neighboring frames for segmentation while reducing the influence of spatial misalignment between frames. The MFFA module also further aggregates features spatially based on the spatial self-attention mechanism. Neighboring frames usually have similar appearances, so we consider feature aggregation over a frame sequence as an iterative feature aggregation procedure. By distributing the computational workload of deep feature extraction over each frame in a sequence, we can use a lightweight encoder to reduce the computation costs. Moreover, public surgical videos usually are not labeled by frame, so we develop a method that can randomly synthesize a surgical frame sequence from a labeled frame to assist network training. We demonstrate that our approach achieves superior performance to corresponding deeper segmentation models on a public endoscopic sinus surgery dataset.



There are no comments yet.


page 1

page 4

page 6


Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video

Performing a real-time and accurate instrument segmentation from videos ...

U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument

Conventional therapy approaches limit surgeons' dexterity control due to...

One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video

Surgical instrument segmentation in robot-assisted surgery (RAS) - espec...

Towards Better Surgical Instrument Segmentation in Endoscopic Vision: Multi-Angle Feature Aggregation and Contour Supervision

Accurate and real-time surgical instrument segmentation is important in ...

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Real-time segmentation of surgical instruments plays a crucial role in r...

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Automatic instrument segmentation in video is an essentially fundamental...

ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools

Real-time tool segmentation from endoscopic videos is an essential part ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.