Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance

by   Dominik Rivoir, et al.

Intra-operative anticipation of instrument usage is a necessary component for context-aware assistance in surgery, e.g. for instrument preparation or semi-automation of robotic tasks. However, the sparsity of instrument occurrences in long videos poses a challenge. Current approaches are limited as they assume knowledge on the timing of future actions or require dense temporal segmentations during training and inference. We propose a novel learning task for anticipation of instrument usage in laparoscopic videos that overcomes these limitations. During training, only sparse instrument annotations are required and inference is done solely on image data. We train a probabilistic model to address the uncertainty associated with future events. Our approach outperforms several baselines and is competitive to a variant using richer annotations. We demonstrate the model's ability to quantify task-relevant uncertainties. To the best of our knowledge, we are the first to propose a method for anticipating instruments in surgery.


page 1

page 2

page 3

page 4


Autonomous Coordinated Control of the Light Guide for Positioning in Vitreoretinal Surgery

Vitreoretinal surgery is challenging even for expert surgeons owing to t...

Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion

Surgical instrument segmentation is a key component in developing contex...

Real-time image-based instrument classification for laparoscopic surgery

During laparoscopic surgery, context-aware assistance systems aim to all...

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Recognition of surgical activity is an essential component to develop co...

ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery

Representation learning of the task-oriented attention while tracking in...

Designing Intelligent Instruments

Remote science operations require automated systems that can both act an...

Code Repositories



view repo

1 Introduction

Figure 1: Predicted remaining time until occurrence of the scissors within a horizon of three minutes on an example surgery. In cholecystectomies, the clipper is an indicator for future use of the scissors. Error and uncertainty decrease when the clipper appears.

Anticipating the usage of surgical instruments before they appear is a highly useful task for various applications in computer-assisted surgery. It represents a large step towards understanding surgical workflow to provide context-aware assistance. For instance, it enables more efficient instrument preparation [maier2017surgical]. For semi-autonomous robot assistance, instrument anticipation can facilitate the identification of events that trigger the usage of certain tools and can eventually help a robotic system to decide when to intervene. Anticipating the use of certain instruments such as the irrigator can further enable early detection and anticipation of complications like bleedings.

The proposed applications require continuous anticipation estimates in long surgeries. Many instruments occur only rarely and briefly, i.e. only sparse annotations are available. Nevertheless, a useful anticipation framework should only react to these sparse occurrences and remain idle otherwise. Our approach addresses these requirements and is thus applicable to real-world scenarios. We train a neural network to regress the remaining time until the occurrence of a specific instrument within a given future horizon. Our uncertainty-quantification framework addresses the uncertainty associated with future events. This enables the identification of trigger events for instruments by measuring decreases in uncertainty associated with anticipation estimates (Fig.


Various works have investigated short-horizon anticipation of human actions [damen2018scaling, gao2017red, jain2016recurrent, vondrick2016anticipating]. However, these methods are only designed to predict actions in the immediate future based on a single frame [damen2018scaling, vondrick2016anticipating] or short sequences of frames [gao2017red]. Most importantly, they are trained and evaluated on artificially constructed scenarios where a new action is assumed to occur within typically one second and only its correct category is unknown. In surgery however, the challenge rather lies in correctly timing predictions given the video stream of a whole surgical procedure. Our task definition considers long video sequences with sparse instrument usage and encourages models to only react to relevant cues. Jain et al. [jain2016recurrent] address this by adding the default activity ’drive straight’ for anticipation of driver activities. For our task, we propose a similar category when no instrument is used within a defined horizon.

While methods for long-horizon anticipation exist, they require dense, rich action segmentations of the observed sequence during inference and mostly do not use visual features for anticipation [abu2019uncertainty, abu2018will, du2016recurrent, ke2019time, mehrasa2019variational]. This is not applicable to our task, since the information required for anticipating the usage of some instruments relies heavily on visual information. For instance, the usage of the irrigator is often triggered by bleedings and cannot be predicted solely from instrument signals. Some methods utilize visual features but nevertheless require dense action labels [mahmud2017joint, zhong2018time]. However, these labels are tedious to define and annotate and therefore not applicable to many real-world scenarios, especially surgical applications. In contrast, we propose to predict the remaining time until the occurrence of sparse events rather than dense action segmentations. During training, only sparse instrument annotations are required and inference is done solely on image data. This aids data annotation since instrument occurrence does not require complex definitions or expert knowledge [maier2014can].

The uncertainty associated with future events has been addressed by some approaches [abu2019uncertainty, vondrick2016anticipating]. Similar to Farha et al. [abu2019uncertainty], we do so by learning a probabilistic prediction model. Bayesian Deep Learning through Monte-Carlo Dropout provides a framework for estimating uncertainties in deep neural networks [gal2016dropout]. Kendall et al. [kendall2017uncertainties]

identify the model and data as two relevant sources of uncertainty in machine learning. Several approaches have been proposed for estimating these quantities 

[gal2017deep, kwon2020uncertainty, shridhar2018uncertainty, wang2019aleatoric]. We evaluate our model’s ability to quantify task-relevant uncertainties using these insights. The contributions of this work can be summarized as follows:

  1. To the best of our knowledge, we are the first to propose a method for anticipating the use of surgical instruments for context-aware assistance.

  2. We reformulate the general task of action anticipation to alleviate limitations of current approaches regarding their applicability to real-world problems.

  3. Our model outperforms several baseline methods and gives competitive results by learning only from sparse annotations.

  4. We evaluate our model’s ability to quantify uncertainties relevant to the task and show that we can improve performance by filtering uncertain predictions.

  5. We demonstrate the potential of our model to identify trigger events for surgical instruments through uncertainty quantification.

2 Methods

2.1 Anticipation Task

2.1.1 Regression:

We define surgical instrument anticipation as a regression task of predicting the remaining time until the occurrence of one of surgical instruments within a future horizon of minutes. Given frame from a set of recorded surgeries and an instrument , the ground truth for the regression task is defined as


where is the true remaining time in minutes until the occurrence of with for frame where is present. The target value is truncated at minutes since we assume that instruments cannot be anticipated accurately for arbitrarily long intervals. This design choice encourages the network only to react when the usage of an instrument in the foreseeable future is likely and predict a constant otherwise. Opposed to current definitions for anticipation tasks, we do not assume an imminent action or rely on dense action segmentations.

2.1.2 Classification:

For regularization, we add a similar classification objective to predict one of three categories , which correspond to an instrument appearing within the next minutes, being present and a background category when neither is the case. In Section 3.2, we discuss the benefits of this regularization task for uncertainty quantification.

2.2 Bayesian Deep Learning

Due to the inherent ambiguity of future events, anticipation tasks are challenging and benefit from estimating uncertainty scores alongside model predictions. Bayesian neural networks enable uncertainty quantification by estimating likelihoods for predictions rather than point estimates [kendall2017uncertainties]. Given data with labels , Bayesian neural networks place a prior distribution over parameters which results in the posterior . Since the integration over the parameter space makes learning intractable, variational inference [graves2011practical] is often used to approximate

by minimizing the Kullback-Leibler divergence

to a tractable distribution . Gal et al. [gal2016dropout] have shown that this is equivalent to training a network with dropout if is in the family of binomial dropout distributions.

During inference, we draw =10 parameter samples to approximate the predictive expectation for regression variables and the predictive posterior for classification variables (Eq. 2 & 3), where and are the network’s regression and softmax outputs parametrized by  [kendall2017uncertainties].


We estimate uncertainties through the predictive variance

. Kendall et al. [kendall2017uncertainties] argue that predictive uncertainty can be divided into aleatoric (data) uncertainty, which originates from missing information in the data, and epistemic (model) uncertainty, which is caused by the model’s lack of knowledge about the data. Intuitively, uncertainty regarding future events corresponds best to aleatoric uncertainty but presumably affects model variations (epistemic) as well.

For regression, we follow Kendall et al.’s approach to estimate epistemic uncertainty (Eq. 4). Computing the predictive variance over parameter variation captures noise in the model. We omit aleatoric uncertainty for regression, since it was not effective for our task. For classification variables, we follow Kwon et al. [kwon2020uncertainty] (Eq. 5). The epistemic term captures noise in the model by estimating variance over parameter samples, while the variance over the multinomial softmax distribution in the aleatoric term captures inherent sample-independent noise. In the classification case, uncertainties are averaged over all classes.


2.3 Model, Data & Training

The model (suppl. material) consists of a Bayesian AlexNet-style Convolutional Network [krizhevsky2012imagenet]

and a Bayesian Long Short-Term Memory network (LSTM) 

[hochreiter1997long]. We sample dropout masks with a dropout rate of once per video and per sample and reuse the same masks at each time step as proposed by Gal et al. [gal2016theoretically] for recurrent architectures. The AlexNet backbone has proven effective in a similar setting [bodenstedt2019active] and empirically gave the best results. State-of-the-art architectures such as ResNet [he2016deep]

performed poorly as they appeared to learn from future frames through batch statistics in batch-normalization layers 

[ioffe2015batch]. Further, the AlexNet can be trained from scratch, which is beneficial for introducing dropout layers. The code is published on

We train on the Cholec80 dataset [twinanda2016endonet] of 80 recorded cholecystectomies and anticipate sparsely used instruments which are associated with specific tasks in the surgical workflow, i.e. bipolar (appears in of frames), scissors (), clipper (), irrigator (), and specimen bag (). Grasper and hook are dropped as they are used almost constantly during procedures and hence, are not of interest for anticipation. We extract frames at fps, resize to width and height of , process batches of

sequential frames and accumulate gradients over three batches. We use 60 videos for training and 20 for testing. We train for 100 epochs using the Adam optimizer (learning rate

). The loss (Eq. 6) is composed of smooth L1 [twinanda2018rsdnet] for the primary regression task, cross entropy (CE) for the regularizing classification task and L2-regularization, where are parameter estimates of the approximate distribution . We set and .

= 2 min. = 3 min. = 5 min. = 7 min.
MeanHist 0.56 0.93 0.85 1.34 1.41 2.14 1.97 2.79
OracleHist (offline) (0.49) (0.83) (0.71) (1.18) (1.11) (1.73) (1.48) (2.23)
Ours non-Bayes. 0.44 0.67 0.64 0.93 1.13 1.64 1.58 2.21
Ours+Phase 0.42 0.64 0.67 0.94 1.07 1.49 1.61 2.16
Ours 0.44 0.65 0.64 0.92 1.09 1.53 1.58 2.16
Table 1: Comparison to baselines. Results for OracleHist are parenthesized since it is an offline approach. We report the mean over instrument types in minutes per metric.
Bipolar Scissors Clipper Irrigator Specimen Bag
MeanH. 0.85 1.36 0.81 1.42 0.80 1.29 0.89 1.40 0.89 1.24
OracleH. (0.79) (1.17) (0.76) (1.32) (0.80) (1.39) (0.73) (1.11) (0.50) (0.95)
O. non-B. 0.75 0.92 0.49 0.78 0.74 1.01 0.71 1.05 0.54 0.92
O.+Ph. 0.79 0.96 0.53 0.82 0.77 1.01 0.72 1.05 0.56 0.88
Ours 0.76 0.96 0.51 0.76 0.71 0.90 0.70 1.03 0.55 0.93
Table 2: Instrument-wise error in minutes for a horizon of 3 minutes.

3 Evaluation

3.1 Anticipation Results

3.1.1 Evaluation Metrics:

We evaluate frame-wise based on a weighted mean absolute error (wMAE). We average the MAE of ’anticipating’ frames () and ’background’ frames () to compensate for the imbalance in the data. As instruments are not always predictable, a low recall does not necessarily indicate poor performance, making precision metrics popular for anticipation [gao2017red, vondrick2016anticipating]. We capture the idea of precision in the pMAE as the MAE of predictions with when the model is anticipating . Factors are chosen for robustness against variations during ’background’ predictions.

3.1.2 Baselines:

Since our task is not comparable to current anticipation methods, we compare to two histogram-based baselines. For instrument , the bin accumulates the occurrences of within the segments of all training video. If the bin count exceeds a learned threshold, we assume that occurs regularly in the video segment and generate anticipation values according to Eq. 1. Using 1000 bins, the thresholds are optimized to achieve best training performance w.r.t our main metric wMAE. For MeanHist, segments are expanded to the mean video duration. For OracleHist, we expand the segments to the real video duration at train and test time. This is a strong baseline as instrument usage correlates strongly with the progress of surgery, which is not known beforehand. See Fig. 4 for a visual overview of the baseline construction.

Additionally, we compare our model to a variant simultaneously trained on dense surgical phase segmentations [twinanda2016endonet] to investigate whether our model achieves competitive results using only sparse instrument labels. Surgical phases strongly correlate with instrument usage [twinanda2016endonet] and have shown to be beneficial for the related tasks of predicting the remaining surgery duration [twinanda2018rsdnet]. Finally, we compare to a non-Bayesian variant of our model without dropout to show that the Bayesian setting does not lead to a decline in performance.

Figure 2: Frame-wise (orange) and median (black) uncertainties for anticipating predictions of scissors depending on the clipper’s presence. Since the clipper often proceeds scissors we expect lower uncertainty when the clipper is present.

3.1.3 Discussion:

We train models on horizons of 2, 3, 5 and 7 minutes and repeat runs for Ours and Ours non-Bayes. four times and Ours+Phase twice. In all settings, the our methods outperform MeanHist by a large margin (Table 1). Compared to the offline baseline OracleHist, we achieve lower pMAE and comparable wMAE errors, even though knowledge about the duration of a procedure provides strong information regarding the occurrence of instruments. Further, there is no visible difference in performance with and without surgical phase regularization. This suggests that our approach performs well by learning only from sparse instrument labels. The Bayesian setting also does not lead to a drop in performance while adding the advantage of uncertainty estimation. For , we outperform MeanHist for all instruments and OracleHist for all except the specimen bag (Table 2). However, this instrument is easy to anticipate when the procedure duration is known since it is always used toward the end. For instrument-wise errors of other horizons, see the supplementary material.

3.2 Uncertainty Quantification Results

We analyze the model’s ability to quantify uncertainties using a model for . For all experiments, we consider predictions which indicate that the model is anticipating an instrument. We define anticipating predictions as where , and with .

Figure 3: Left: pMAE as a plot of percentiles of filtered predictions w.r.t. . Center: Frame-wise error-uncertainty plot per instrument with Pearson Correlation Coefficient (PCC) and linear fit (black). Right: Frame-wise (colored) and median (black) uncertainty of true positives (TP) vs. false positives (FP) for class anticipating.

Uncertainty quantification enables identification of events which trigger the usage of instruments. We evaluate the model using the known event-trigger relationship of scissors and clipper in cholecystectomies, where the cystic duct is first clipped and subsequently cut. Hence, we expect lower uncertainty for anticipating scissors when a clipper is visible. Fig. 2 supports this hypothesis for epistemic uncertainty during regression. However, the difference is marginal and most likely not sufficient for identifying trigger events. Even though clipper occurrence makes usage of the scissors foreseeable, predicting the exact time of occurrence is challenging and contains noise. Uncertainties for classification are more discriminative. The classification objective eliminates the need for exact timing and enables high-certainty class predictions. Both epistemic and aleatoric estimates are promising while the latter seems to be more discriminative. This is consistent with our hypothesis that uncertainty regarding future events corresponds best to aleatoric uncertainty but induces epistemic uncertainty as well.

We assess the quality of uncertainty estimates through correlations with erroneous predictions as high uncertainty should result in higher errors. For regression (Fig. 3, center), we observe the highest Pearson Correlation Coefficients (PCC) for scissors, clipper and specimen bag. These instruments are presumably the most predictable since they empirically yield the best results (Table 2) and are correlated with specific surgical phases (’Clipping & Cutting’ and ’Gallbladder Packaging’) [twinanda2016endonet]. Irrigator and Bipolar yield less reliable predictions as they are used more dynamically. For classification (Fig. 3, right), the median aleatoric uncertainty of true positive predictions for the ’anticipating’ class is almost consistently lower than for false positives. Scissors and specimen bag show the largest margin. We can reduce precision errors (pMAE) by filtering uncertain predictions, shown in Fig. 3 (left) for epistemic regression uncertainty. As expected, the decrease is steaper for instruments with higher PCC.

4 Conclusion

We propose a novel task and model for uncertainty-aware anticipation of intra-operative instrument usage. Limitations of current anticipation methods are addressed by enabling anticipation of sparse events in long videos. Our approach outperforms several baselines and matches a model variant using richer annotations, which indicates that sparse annotations suffice for this task. Since uncertainty estimation is useful for both anticipation tasks and intra-operative applications, we employ a probabilistic model. We demonstrate the model’s ability to quantify task-relevant uncertainties by investigating error-uncertainty correlations and show that we can reduce errors by filtering uncertain predictions. Using a known example, we illustrate the model’s potential for identifying trigger events for instruments, which could be useful for robotic applications. Future work could investigate more effective methods for uncertainty quantification where aleatoric uncertainty is especially interesting due to its link to future events.

Figure 4: Overview of the baseline construction (example: irrigator). The temporal histogram of instrument occurences is thresholded to obtain an estimated instrument presence. The instrument presence signal is used to generate anticipation values according to Eq. 1. For each instrument, we find the optimal threshold w.r.t. wMAE on the train set. For the online method MeanHist, predictions are expanded to the mean duration of procedures in the train set. For the offline method OracleHist, we assume the length of the target procedure to be known and expand predictions accordingly.

Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance (Supplementary Material) D. Rivoir et al.

Figure 5: Network architecture for instruments: Dropout masks (Drop

) are sampled once per video and reused at each time step. Notation: Conv (kernels, kernel size, stride), Drop (drop rate), MaxPool (kernel size, stride), AdaptiveAvgPool (output size), FC

(fully-connected) (size), LSTM (size). In the LSTM, dropout is applied to both the input and the hidden state of the previous time point.
Bipolar Scissors Clipper Irrigator Specimen Bag
= 2 min.
MeanH. 0.55 0.90 0.55 1.01 0.54 0.90 0.58 0.99 0.60 0.83
OracleH. (0.54) (0.89) (0.51) (0.91) (0.53) (0.79) (0.50) (0.78) (0.39) (0.72)
O. non-b. 0.51 0.71 0.33 0.57 0.50 0.67 0.47 0.76 0.38 0.65
O.+Ph. 0.51 0.68 0.33 0.56 0.47 0.62 0.46 0.70 0.36 0.63
Ours 0.51 0.66 0.36 0.61 0.50 0.65 0.47 0.72 0.37 0.61
= 5 min.
MeanH. 1.44 2.26 1.33 2.18 1.34 2.12 1.50 2.15 1.46 1.98
OracleH. (1.25) (1.67) (1.31) (2.20) (1.23) (1.93) (1.14) (1.63) (0.64) (1.28)
O. non-b. 1.36 1.70 0.93 1.41 1.24 1.75 1.25 1.80 0.90 1.54
O.+Ph. 1.32 1.61 0.88 1.33 1.09 1.55 1.22 1.63 0.85 1.34
Ours 1.33 1.67 0.89 1.32 1.12 1.64 1.25 1.64 0.85 1.40
= 7 min.
MeanH. 2.08 2.90 1.82 2.76 1.86 2.93 2.14 2.78 1.95 2.59
OracleH. (1.71) (2.18) (1.78) (2.90) (1.64) (2.39) (1.54) (2.16) (0.74) (1.50)
O. non-b. 2.00 2.34 1.37 2.02 1.56 2.30 1.82 2.31 1.18 2.07
O.+Ph. 2.14 2.43 1.31 1.98 1.54 2.14 1.89 2.26 1.22 2.01
Ours 2.04 2.39 1.26 1.89 1.51 2.20 1.86 2.30 1.23 2.04
Table 3: Instrument-wise errors in minutes for horizons of 2, 5 and 7 min. The offline method OracleHist is parenthesized.
Figure 6: Histogram of instrument occurrences used for learning the baselines MeanHist and OracleHist. Horizontal lines indicate learnt thresholds for baseline OracleHist and a horizon of 3 min.
Figure 7: Anticipation results for example surgery with . Grey areas highlight anticipating frames (). The best anticipation results are achieved for scissors and specimen bag. Our model anticipates the clipper when used directly before scissors but fails otherwise as it is used outside of the typical workflow of cholecystectomies.