With the emergence of imaging devices in the operating room, the automated analysis of videos recorded during the surgery is becoming a hot research topic. In particular, videos can be used to monitor the surgery, for instance by recognizing
which surgical tools are being used at every moment.An immediate application of surgery monitoring is report generation. If automatic reports are available for many surgeries, then the automatic analysis of these reports can help optimize the surgical workflow or evaluate surgical skills. Additionally, if we are able to generate such a report in real-time, during a surgery, then we could compare it with previous reports to generate warnings, if we recognize patterns often leading to complications, or recommendations, to help younger surgeons emulate more experienced colleagues based on their surgical reports (Quellec et al., 2015). With adequate image analysis techniques, tool usage could be monitored reliably in tool-interaction videos, such as endoscopic videos (in laparoscopic or retinal surgeries) or microscopic videos (in anterior eye segment surgeries). In the simplest scenario, we can consider that a tool is being used if it is visible in these videos. In a more advanced scenario, we can consider that it is in use if it is in contact with the tissue (as opposed to approaching the tissue, waiting to be used, etc.). Therefore, several tool detection techniques for tool-interaction videos have been proposed in recent years (Bouget et al., 2017). To compare these techniques, two tool detection challenges were organized recently. A first challenge, organized at the M2CAI 2016 workshop,111http://camma.u-strasbg.fr/m2cai2016/index.php/tool-presence-detection-challenge-results/ relied on endoscopic videos of cholecystectomy operations performed laparoscopically. We organized a second challenge for cataract surgery, the most common surgical procedure worldwide (Trikha et al., 2013).222https://cataracts.grand-challenge.org/ It relied on videos recorded through a surgical microscope. Following the trend in medical image and video analysis (Shen et al., 2017), the best solutions of both challenges all relied on convolutional neural networks (CNNs) (Raju et al., 2016; Sahu et al., 2016; Twinanda et al., 2017; Zia et al., 2016; Roychowdhury et al., 2017; Hu and Heng, 2017; Maršalkaitė et al., 2017).
Compared to other computer vision tasks, surgical tool usage annotation has several specificities. First, as opposed to many computer vision tasks, including the popular ImageNet visual recognition challenges,333http://www.image-net.org/challenges/LSVRC/2017/index.php the problem at hand is not multiclass classification (one correct label per image among multiple classes), but rather multilabel classification (multiple correct labels per image): the number of tools being used in each image varies (from zero to three in cataract surgery for instance). Therefore, multilabel CNNs should be used. Second, taking the temporal sequencing into account is important: knowing which tools have already been used since the beginning of the surgery greatly helps recognize which tools are currently being used. Therefore, multilabel recurrent neural networks (RNNs) (Hochreiter and Schmidhuber, 1997)
may also be used advantageously. In fact, recent machine learning competitions clearly show that ensembles of CNNs outperform single CNNs(Russakovsky et al., 2015)
: multiple CNNs with different architectures are generally trained independently and their outputs are combined afterward using standard machine learning algorithms (decision trees, random forests, multilayer perceptrons, etc.). However, this simple strategy is suboptimal since difficult samples may be misclassified by all CNNs. And there are many difficult samples to classify in surgery videos: in particular, many tools resemble one other (e.g. two types of cannulae in cataract surgery). Building the ensemble of CNNs using a boosting meta-algorithm(Freund and Schapire, 1997) can theoretically design CNNs focusing specifically on challenging samples. Boosting an ensemble of RNNs would also make sense as there are difficult samples along the time dimension as well: in particular, some tools or tool usage sequences are very rare and temporal sequencing algorithms tend to misclassify those rare cases. Therefore, we propose to jointly boost an ensemble of CNNs and an ensemble of RNNs for automatic tool usage annotation in surgery videos. In the same way as CNN boosting (or RNN boosting) allows various CNNs (or RNNs) to be complementary, this general boosting solution allows CNNs to be complementary with RNNs. In that sense, it approximates the end-to-end training of a “CNN+RNN” network, which is theoretically ideal but not computationally tractable.
The remainder of this paper is organized as follows. Section 2 reviews the state of the art of video analysis, and surgery video analysis in particular. Sections 3 and 4 describe the proposed solution. Section 5 presents the video datasets and section 6 reports the experiments performed on that dataset. We end with a discussion and conclusions in section 7.
2 State of the Art
2.1 Deep Learning for Video Analysis
The automatic analysis of dynamic scenes through deep learning has become a very hot research topic
The automatic analysis of dynamic scenes through deep learning has become a very hot research topic(Simonyan and Zisserman, 2014; Wang et al., 2017; Donahue et al., 2017). Different strategies have been proposed for this task. A first strategy is to regard videos or video portions as 3-D images and therefore analyze them with 3-D CNNs (Ji et al., 2013), which is computationally expansive. A second strategy is to analyze 2-D images as well as the optical flow between consecutive images (Simonyan and Zisserman, 2014), with the disadvantage of only modeling short-term relationships between images. A third strategy is to combine a CNN, analyzing 2-D images, with a RNN analyzing the temporal sequencing (Donahue et al., 2017). The main advantage of this “CNN+RNN” approach, which is now the leading video analysis solution, is that long-term relationships between events can be taken into account efficiently. One application of “CNN+RNN” models, which is particularly relevant for our study, is video labeling: the goal is to assign one class label to each frame inside a video (Singh et al., 2016; Khorrami et al., 2016). Medical applications of this research, ranging from gait analysis (Feng et al., 2016) to surgery monitoring (Bodenstedt et al., 2017; Twinanda et al., 2016), are starting to emerge.
2.2 Temporal Analysis of Surgery Videos
In the context of surgical workflow analysis, solutions have been proposed to recognize surgical phases in surgery videos (Lalys and Jannin, 2014; Charrière et al., 2017). In Primus et al. (2018), phases are recognized using one CNN processing the visual content of one frame plus the relative timestamp of that frame.
However, most solutions rely on statistical models, such as Hidden Markov Models (HMMs)(Cadène et al., 2016), Hidden semi-Markov Models (Dergachyova et al., 2016; Tran et al., 2017), Hierarchical HMMs (Twinanda et al., 2017), Linear Dynamical Systems (Zappella et al., 2013; Tran et al., 2017) or Conditional Random Fields (Tao et al., 2013; Quellec et al., 2014; Lea et al., 2016a). Recently, solutions based on RNNs have also been proposed (Jin et al., 2016; Bodenstedt et al., 2017; Twinanda et al., 2016)
. Following the state-of-the-art video analysis strategy, these RNNs process instant visual features extracted by a CNN from images. In particular,Jin et al. (2016) applied a “CNN+RNN” network to a small sliding window of three images. Bodenstedt et al. (2017) applied a “CNN+RNN” network to larger sliding windows and copy the internal state of the network between consecutive window locations. As for Twinanda et al. (2016), they applied a “CNN+RNN” network to full videos. Interestingly, the CNN proposed by Twinanda et al. (2016), namely EndoNet, detects tools as an intermediate step. A challenge on surgical workflow analysis was also organized at M2CAI 2016:444http://camma.u-strasbg.fr/m2cai2016/index.php/workflow-challenge-results/ two of the top three solutions relied on RNNs (Jin et al., 2016; Twinanda et al., 2016). It should be noted that successful works on the analysis of kinematics surgery data have also been reported, using a RNN (Dipietro et al., 2016) or a CNN along the temporal dimension (Lea et al., 2016b). In all these works, statistical models or RNNs were used to label surgical activities and phases. Given the strong correlation between surgical activities and tool usage, they can be expected to improve tool recognition as well.
2.3 Deep Learning for Surgical Tool Detection
As evidenced by the M2CAI 2016 and CATARACTS 2017 challenges, the state-of-the-art algorithms for tool detection in surgery videos are CNNs. The best solutions of these challenges
rely on a transfer learning strategy: well-known CNNs trained to classify still images in the ImageNet dataset were fine-tuned on images extracted from surgery videos.For M2CAI 2016, Sahu et al. (2016) and Twinanda et al. (2017) fine-tuned AlexNet (Krizhevsky et al., 2012), Raju et al. (2016) fine-tuned GoogleNet (Szegedy et al., 2015a) and VGG-16 (Simonyan and Zisserman, 2015), and Zia et al. (2016) fine-tuned AlexNet, VGG-16 and Inception-v3 (Szegedy et al., 2015b). For CATARACTS, Roychowdhury et al. (2017) fine-tuned Inception-v4 (Szegedy et al., 2017), ResNet-50 (He et al., 2016a) and two NASNet-A instances (Zoph et al., 2017), Hu and Heng (2017) fine-tuned ResNet-101 and DenseNet-169 (Huang et al., 2017), and Maršalkaitė et al. (2017) fine-tuned four ResNet-50 instances. Training a CNN proved challenging due to highly frequent tool co-occurrences: a solution based on label-set sampling has been proposed by Sahu et al. (2017) to reduce this bias. Note that temporal information is not exploited in these solutions, with a few exceptions presented hereafter (Sahu et al., 2017; Maršalkaitė et al., 2017; Al Hajj et al., 2017; Mishra et al., 2017; Roychowdhury et al., 2017). In Sahu et al. (2017) and Maršalkaitė et al. (2017), a linear filter is used to smooth CNN predictions from consecutive frames. In Al Hajj et al. (2017), a CNN processes short sequences of consecutive images, using the optical flow to register and combine local features from consecutive images. In Mishra et al. (2017), one RNN processes the outputs of a frame-level CNN inside short sequences of consecutive frames. Note that long-term relationships between images are not exploited neither in these four solutions: the goal is to combine slightly different views on a tool, some of which being affected by motion blur or occlusion. In Roychowdhury et al. (2017), on the other hand, long-term relationships between images are exploited through a Markov Random Field (MRF) modeling long sequences of approximately 20,000 frames. The drawback is that online video analysis is not possible.
2.4 Proposed Solution
In this paper, we propose to design “CNN+RNN” networks, the state-of-the-art video analysis framework, for the task of automatic tool usage annotation. Due to the specific challenges of this task, namely the similarity between some tools and the rarity of some tool usages, we propose to apply the boosting principle to both the CNN part and the RNN part of the network, in a novel and unified manner. Besides addressing the previously mentioned difficult cases, the proposed framework has multiple advantages: 1) it can be used to select the network architectures automatically, an open problem in deep learning, and 2) it can improve the complementarity of CNNs and RNNs, an unsolved problem in “CNN+RNN” models for which end-to-end learning is not tractable (see Fig. 1). Section 3 briefly describes the networks considered in this paper and the related challenges. Section 4 describes the boosting algorithm proposed to address those challenges. The proposed solution has several novelties. First, the use of CNN boosting and RNN boosting for medical images or videos is novel. Second, the data-driven design of a CNN or CNN ensemble to be used as input for an RNN or RNN ensemble (through boosting — see section 4.5) has never been studied before.
3 “CNN+RNN” Networks
Let denote a set of surgical tools whose usage should be monitored in videos. Let denote a collection of training videos and let denote the -th frame in video . Let denote the binary label assigned to frame for tool : this label indicates whether or not tool is being used in frame . We are addressing a multilabel classification problem, so . In contrast, in a multiclass classification problem.
Neural networks considered in this paper consist of one or several CNNs working in parallel: this set of CNNs is referred to as the “CNN block”. Let denote the instant predictions computed by the CNN block for frame . Some of the neural networks considered in this paper also contain one or several RNNs working in parallel: this set of RNNs is referred to as the “RNN block”. Let denote the context-aware predictions computed by the RNN block for frame .
3.2 RNNs Processing CNN Predictions
A recurrent neural network (RNN) is a neural network that takes a sequence of observations at the input and produces a sequence of predictions at the output (Hochreiter and Schmidhuber, 1997). In this paper, the input sequence is , i.e. the predictions of the CNN block for each frame in a video. The output sequence is . The network is structured in such a way that the prediction vector to a group of neurons . Weights are shared across all cells. The most popular cells are Long Short-Term Memory (LSTM) cells : they include a “forgetting” mechanism preventing backpropagated errors from vanishing or exploding in long sequences. More recently, Gated Recurrent Units (GRU) were proposed by
. The network is structured in such a way that the prediction vectordepends on feature vector , but also on all previous feature vectors , . This behavior is achieved by 1) connecting each input element
to a group of neuronscalled “cell”, 2) connecting to the output element and 3) connecting to the next cell
. Weights are shared across all cells. The most popular cells are Long Short-Term Memory (LSTM) cells(Hochreiter and Schmidhuber, 1997)
: they include a “forgetting” mechanism preventing backpropagated errors from vanishing or exploding in long sequences. More recently, Gated Recurrent Units (GRU) were proposed byCho et al. (2014): the labeling performance of these lower-complexity cells is often comparable with LSTM.
A multi-layer extension was proposed for RNNs. In this extension, each timestamp is associated with multiple cells , where is the layer index. At each timestamp , is connected to , is connected to for , and is connected to . In each layer , is connected to . Weights are shared across all cells in the same layer. A bidirectional extension was also proposed for RNNs (Schuster and Paliwal, 1997). In this extension, illustrated in Fig. 2, two independent RNNs are defined: in one of them, information flows from timestamp to timestamp ; in the other one, information flows from timestamp to timestamp . Their outputs are concatenated and connected to the output sequence. The performance of bidirectional RNNs, which take advantage of past and future information, is generally higher. The drawback is of course that online video labeling is not possible.
3.3 RNNs on Long Video Sequences
In the literature, RNNs are generally trained using video sequences consisting of a few dozen frames at most (Chen et al., 2017; Gammulle et al., 2017; Mishra et al., 2017). In contrast, analyzing all frames of full surgery videos requires the analysis of much longer sequences: for instance, there are at least 10,000 frames per video sequence in our cataract surgery videos (see section 5.1). Training long-term relationships with RNNs is more computationally intensive using long sequences, so we propose to analyze shorter sequences. In that purpose, subsampled versions of each original sequence , denoted by , , are generated as follows:
During training, this results in a novel kind of data augmentation (Shen et al., 2017): the number of training sequences increases artificially. For simplicity, is denoted by in the remainder of this paper. During testing, each of the subsequences of are analyzed independently and the final prediction sequence for is obtained by interleaving the resulting prediction sequences. The resulting prediction sequence is further processed by median filters to blend subsequences: a filter of radius is used for each tool-specific channel of the sequence.
3.4 Training Complexity for “CNN+RNN” Networks
Because CNNs and RNNs are integrated into the same network, it would make sense to train the entire network from end to end, so that features extracted by the CNNs are as relevant as possible to the RNNs that process them further. However, as illustrated in Fig. 1, the complexity of the learning process is very high. The error measured for each prediction is backpropagated to but also to all (such as , in unidirectional networks). Errors computed for each are backpropagated further towards .
The vast majority of weights in a “CNN+RNN” network are in the CNNs. Therefore, the cost of backpropagating an error measured for one timestamp to all frames in the video sequence (such as , in unidirectional networks) is very high. As a consequence, a two-step training process is always preferred in the literature (see section 2.1). A CNN is trained first: errors measured for one timestamp are only backpropagated to . Then, a RNN is trained: errors measured for one timestamp are backpropagated to all (such as , in unidirectional networks) without affecting the CNN weights. Given the number of weights in a RNN, this process is tractable. We propose a solution based on boosting that is able to improve the CNN block after or while training the RNN block, in order to achieve the desirable properties of end-to-end training, but at a reasonable computational cost.
4 Boosted “CNN+RNN” Networks (see Fig. 3)
Recent boosting algorithms, such as AnyBoost (Mason et al., 1999) and Friedman (2001) ’s Gradient Boosting Machines (GBM), are formulated as a gradient descent optimization, which integrates nicely with the way neural networks are trained. When CNNs or RNNs are used as weak learners, the boosting meta-algorithm controls the loss function used to train these learners. Typically, training samples with large classification errors are assigned a larger weight in the updated loss function. A few authors thus used CNNs as weak learners for AnyBoost
’s Gradient Boosting Machines (GBM), are formulated as a gradient descent optimization, which integrates nicely with the way neural networks are trained. When CNNs or RNNs are used as weak learners, the boosting meta-algorithm controls the loss function used to train these learners. Typically, training samples with large classification errors are assigned a larger weight in the updated loss function. A few authors thus used CNNs as weak learners for AnyBoost(Moghimi et al., 2016) or GBM (Zhang et al., 2016; Walach and Wolf, 2016). A boosting algorithm based on GBM (Friedman, 2001) is proposed in this section to design either a CNN block or an RNN block. The same algorithm is used for CNN boosting in RNN-free networks and for RNN boosting in “CNN+RNN” networks. To ensure the complementarity of the CNN and RNN blocks in “CNN+RNN” networks, an improved criterion is proposed for CNN boosting in such networks (see section 4.5). How to design an adequate neural network architecture for a given classification problem remains an open question. So, generalizing Gao et al. (2016), multiple architectures of neural networks (CNNs or RNNs) are considered in this study; let denote the set of (CNN or RNN) architectures.
4.2 Gradient Boosting Machine
The purpose of GBM is to build a strong learner by linearly combining multiple weak learners , l=1..L, with weights . Let denote the predictions of for some input . The predictions of the strong learner for are given by:
: in CNN boosting, in RNN boosting. Weak learners are added sequentially in order to minimize the negative log-likelihood (Friedman, 2001):
where is the binary label assigned to for tool (see section 3.1). At each boosting iteration , all weak learners are trained as detailed in sections 4.3 to 4.4. Then, the weak learner minimizing , is added to the strong classifier:
Boosting stops when stops decreasing.
4.3 Loss Function for Boosting Neural Networks
As noted by Friedman (2001), the weak learner selected at boosting iteration should ideally return values proportional to :
where the coefficients, called sample weights, are given by:
With that property, the strong learner’s loss function would decrease directly towards zero. Neural networks can be trained to solve Eq. (5) in the least square sense, using without loss of generality. Therefore, the following quadratic loss function can be used for (Moghimi et al., 2016):
4.4 Efficiently Training Neural Networks as Weak Learners
The proposed solution for training weak learners can be summarized as follows. At iteration 1 (), each weak learner is trained to minimize , the negative log likelihood [see Eq. (3)]. CNN weights are fine-tuned from a model trained on ImageNet; RNN weights are initialized at random. At iterations , , each weak learner is trained to minimize , the quadratic loss function [see Eq. (8)]. Following Moghimi et al. (2016), the neuron weights of are fine-tuned from neuron weights obtained at the previous boosting iteration. This strategy saves time and also improves performance. Indeed, more and more samples receive marginal weights at each boosting iteration, as the classification error decreases [see Eq. (7)]. Therefore, the training set somehow becomes smaller and smaller. The proposed strategy can be regarded as transfer learning from a larger dataset, which is known to be beneficial.
4.5 Boosting CNNs inside a “CNN+RNN” Network
The boosting solution described in previous sections is suboptimal for CNN boosting in a “CNN+RNN” network. Let us assume that one image in a video sequence is wrongly classified by the firstly selected CNN . Based on the temporal context, the RNN block might be able to correct this classification error. Therefore, building a second CNN for correcting that error specifically might be useless. Instead, CNNs should be trained to maximize the performance of the “CNN+RNN” network as a whole.
Throughout the rest of this paper, let , , and denote respectively the strong learner, the weak learners, their weights and their number in the RNN block, in order to avoid confusion with their counterparts in the CNN block. To achieve the desired behavior, the sample weights should be defined based on , the outputs of the RNN block, rather than , the outputs of the CNN block: the goal should be to minimize . In this scenario, , the weight assigned to frame and label , does not depend solely on instant quantities, namely and . In bidirectional networks (for offline processing), it depends on all pairs, . In unidirectional networks, it depends on all pairs such that . For , sample weights become:
If a unidirectional RNN network is used, then the partial derivatives equal zero for all . In all other cases, they can be computed automatically by the backpropagation algorithm. Note that the backpropagation algorithm does not compute each term individually, where
denotes an input tensor whose influence on the output tensorshould be computed. Instead, it computes:
given a tensor weighting each coefficient of the output tensor. However, Eq. (9) can be computed setting:
or depending on .
Proof for Eq. (9)
In this scenario, the partial derivative of the negative log-likelihood function [see Eq. (3)], with respect to , is given by:
Each term in this sum can be decomposed according to the chain rule of derivation, usingthe following equations:
The second factor on the right hand side of Eq. (14) can be decomposed using the derivative of the sigmoid function:
Similarly, the first factor on the right hand side of Eq. (14) can be decomposed as follows:
where and is a function of all values.
The sample weights we have defined for CNN boosting inside a “CNN+RNN” network are more complex than the general case [see Eq. 7]. However, they are only computed once per boosting iteration. Therefore, they do not make the optimization problem significantly less tractable, as opposed to the end-to-end training of a “CNN+RNN” network. But, like end-to-end training, they ensure a good complementarity between the CNN and RNN blocks.
4.6 Joint CNN and RNN Boosting
Two strategies are proposed below to define the order in which CNNs and RNNs are trained to design data-driven “CNN+RNN” architectures.
The most straightforward solution is to boost the CNN block while decreases, and then to boost the RNN block while decreases. Besides the use of boosting, this is the standard approach for designing “CNN+RNN” networks (see section 2.1). However, this solution suffers from the limitation described in the previous section, namely the lack of complementarity between the CNN and RNN blocks.
To overcome this limitation, we propose to design the CNN and RNN blocks inside a single boosting loop, using a single strong learner’s loss function, namely . At each boosting iteration, all CNN architectures and all RNN architectures are trained (or re-trained) and only one CNN or one RNN is added to the network: the one minimizing
Of course, in the first boosting iteration, only CNN architectures are considered: RNNs need at least one feature extractor to operate. Eq. (9) is used to define the sample weights for CNN boosting as soon as .
5 Surgery Video Datasets
The proposed approach is applied to tool usage annotation in two surgical video datasets: CATARACTS and Cholec80.
% of training frames
|Charleux cannula||0.963||1.79 %|
|hydrodissection cannula||0.982||2.43 %|
|Rycroft cannula||0.919||3.18 %|
|viscoelastic cannula||0.975||2.54 %|
|capsulorhexis cystotome||0.995||4.42 %|
|Bonn forceps||0.798||1.10 %|
|capsulorhexis forceps||0.849||1.62 %|
|Troutman forceps||0.764||0.258 %|
|needle holder||0.630||0.0817 %|
|phacoemulsifier handpiece||0.997||15.3 %|
|vitrectomy handpiece||0.998||2.76 %|
|implant injector||0.980||1.41 %|
|primary incision knife||0.961||0.700 %|
|secondary incision knife||0.852||0.522 %|
|suture needle||0.893||0.219 %|
|Mendez ring||0.953||0.100 %|
|Vannas scissors||0.823||0.0443 %|
|specimen bag||n/a||6.35 %|
5.1 CATARACTS Dataset
The CATARACTS dataset contains 50 videos of cataract surgeries performed in Brest University Hospital.555https://cataracts.grand-challenge.org The purpose of cataract surgeries is to remove a clouded natural lens and replace it with an artificial lens. The entire procedure can be performed with small incisions only. Surgeries were monitored through an OPMI Lumera T microscope (Carl Zeiss Meditec, Jena, Germany). Videos were recorded with a 180I camera (Toshiba, Tokyo, Japan) and a MediCap USB200 recorder (MediCapture, Plymouth Meeting, USA). The frame definition was 1920x1080 pixels and the frame rate was approximately 30 frames per second (fps). Videos had a duration of 10 minutes and 56 s on average (minimum: 6 minutes 23 s, maximum: 40 minutes 34 s). In total, more than nine hours of surgery have been video recorded. A list of 21 tools visible in these videos was compiled by a surgeon (see Fig 4). Then, the usage of each tool in videos was annotated independently by two non-clinical experts, after an initial training by a surgeon. A tool was considered to be in use whenever it was in contact with the eyeball. Therefore, both experts recorded a timestamp whenever one tool started or stopped touching the eyeball. Tool-tissue contacts can be detected well: they imply deformations of the eye surface, which are well visible thanks to specular reflections of light. Finally, annotations from both experts were adjudicated: whenever experts disagreed about the label of one tool, they watched the video together and jointly determined the actual label. However, the precise timing of tool/eyeball contacts was not adjudicated. Inter-rater agreement after adjudication is reported in Table 1. The dataset was divided into a training set (25 videos) and a test set (25 videos). Division was made in such a way that each tool appears in the same number of videos from both subsets (plus or minus one). The classification performance for was assessed only in frames where experts agreed about the usage of . During training, some tool was considered to be in use if at least one expert said so.
5.2 Cholec80 Dataset
The Cholec80 dataset contains 80 videos of cholecystectomy surgeries (Twinanda et al., 2017). The purpose of cholecystectomy is to remove the gallbladder: this operation can be performed laparoscopically and monitored through an endoscope. Videos were recorded with a frame definition of 1920x1080 pixels and a frame rate of 25 fps. Videos had a duration of 38 minutes and 26 s on average (minimum: 12 minutes 19 s, maximum: 1 hour 39 minutes 55 s). They were downsampled to 1 fps for processing. In total, more than 51 hours of surgery have been video recorded (2 hours after downsampling). In Cholec80, a tool was considered to be in use if it was visible through the endoscope (if at least half of the tool tip was visible, precisely). The presence of seven tools was annotated in videos (see Fig 4): one binary label is provided per image and per tool. The dataset was divided into a training set (40 videos) and a test set (40 videos).
5.3 Training and Validation Subsets
For validation purposes, two training videos of CATARACTS (respectively four videos of Cholec80) were assigned to a validation subset; the remaining training videos were assigned to a learning subset used to optimize the CNN, RNN and boosting weights. In CATARACTS, the validation videos were chosen such that all tools appear in the learning subset: it was not possible to ensure this property for both subsets. In Cholec80, they were chosen at random.
Seven CNN architectures were used as weak classifiers in this paper:
VGG-16 and VGG-19 (Simonyan and Zisserman, 2015),
Inception-v4 and Inception-ResNet-v2 (Szegedy et al., 2017),
NASNet-A (Zoph et al., 2017).
The TensorFlow-Slim implementation of these CNNs was used, with weights pre-trained on ImageNet. The last layer of each CNN, which computes one logit prediction per class, was resized from 1000 neurons for ImageNet to 21 neurons for CATARACTS or 7 neurons for Cholec80; the weights of these neurons were initialized at random. The same input image size was used for ImageNet, CATARACTS and Cholec80: pixels and were then padded with zeros at the top and the bottom to obtain square images. All CNNs were trained using the RMSProp algorithm with a learning rate initialized to
The TensorFlow-Slim implementation666https://github.com/tensorflow/models/tree/master/research/slim
of these CNNs was used, with weights pre-trained on ImageNet. The last layer of each CNN, which computes one logit prediction per class, was resized from 1000 neurons for ImageNet to 21 neurons for CATARACTS or 7 neurons for Cholec80; the weights of these neurons were initialized at random. The same input image size was used for ImageNet, CATARACTS and Cholec80:pixels for VGG-16 and VGG-19, pixels for ResNet-101, ResNet-152, Inception-v4 and Inception-ResNet-v2, and pixels for NASNet-A. To preserve the aspect ratio, images from CATARACTS and Cholec80 were first resized to pixels, pixels or
pixels and were then padded with zeros at the top and the bottom to obtain square images.
All CNNs were trained using the RMSProp algorithm with a learning rate initialized toand decaying exponentially. In order to define a more challenging boosting problem, we conducted a secondary experiment involving the three worst performing CNNs only: this experiment is called “weaker CNNs”, while the primary experiment involving all CNNs is called “all CNNs”.
Regarding RNN boosting, two types of RNN cells were used: LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Cho et al., 2014). To limit complexity and computation times, the number of layers in RNNs was set to . Three different values were used for , the number of neurons per cell, in order to define six weak classifiers (three based on LSTM, three based on GRU): , , . In all RNN boosting experiments, a subsampling factor of and was used in CATARACTS and Cholec80, respectively: this number was found to be optimal in initial experiments on the validation subset (see Fig. 5). All RNNs were trained using the RMSProp algorithm with a constant learning rate of . As for the median filter radii , they were selected within to maximize the classification performance in the validation set; for rare tools absent from the validation set, the most frequently selected value was used.
RNNs were implemented using Keras version 2.0.8.
Inference times for CNNs, the most computationally intensive parts of the system, are reported in Table 2.
|CNN||single image||batch processing|
|VGG-16||7.50 ms / image||2.87 ms / image|
|VGG-19||8.50 ms / image||3.44 ms / image|
|ResNet-101||10.2 ms / image||3.16 ms / image|
|ResNet-152||13.2 ms / image||4.62 ms / image|
|Inception-v4||18.8 ms / image||6.09 ms / image|
|Inception-ResNet-v2||19.0 ms / image||6.34 ms / image|
|NASNet-A||24.6 ms / image||18.5 ms / image|
6.2 Performance of Boosted Video Labelers
The performance of the seven weak CNNs is reported in Tables 3 and 4. As expected, the best performing CNN, NASNet-A, is also the most recent. Surprisingly, VGG-19 and VGG-16 are also quite good, in spite of being older and less sophisticated than the others. The three worst performing CNNs (in the validation set and in the test set) are ResNet-101, ResNet-152 and Inception-ResNet-v2: they were used in the “weaker CNNs” experiment. The architecture of boosted bidirectional video labelers are reported in Fig 6 for the “all CNNs” and “weaker CNNs” experiments. Their performance is detailed in Tables 3 and 4 for the “all CNNs” experiment. In the largest dataset (CATARACTS), training the initial CNNs with early stopping took between 2h (ResNet-101) and 11h (Inception-ResNet-v2); training NASNet-A took 8h. In the following boosting iterations, fine-tuning the CNNs and training/fine-tuning the RNNs took 3h at most per CNN or RNN. At each boosting iterations, CNNs and RNNs were trained in parallel on a cluster of GeForce GTX 1080 Ti GPUs (RNNs were trained without GPU). Overall, if the process was fully-automated, boosting would have lasted approximately 29h. In practice, it took a few days, as the process involved manual interactions (for early stopping in particular). The end-to-end training of a NASNet-A + RNN network would have lasted more than 80,000 hours (9 years) for CATARACTS, which involves sequences of more than 10,000 frames.
Tables 3 and 4 show that “CNN+RNN” boosting improves performance compared to CNN boosting alone in both datasets. Median filtering also improves performance in the CATARACTS dataset but decreases it in Cholec80. For each tool , the least worst radius is for Cholec80 and the best radius is for CATARACTS. ROC curves and precision-recall curves for the best CNN, namely NASNet-A, and the best ensemble, namely joint “CNN+RNN” boosting (with median filtering for CATARACTS), are reported in Fig. 7 and 8. In terms of area under the ROC curve (), all tools were detected well by the best ensemble (). In terms of average precision (AP), rare tools are poorly detected before boosting ( in some cases). For rare tools, precision (and therefore AP) is indeed impacted strongly by the number of false alarms which, in the specificity criterion (and therefore ), is divided by the large number of negative samples. In fact, as shown in Table 4, AP is highly correlated with tool prevalence in the training set. However, the mean AP is greatly improved after boosting: from mAP to mAP in CATARACTS. Since there are no rare tools in Cholec80, mAP is much higher (up to mAP ).
smoothed boosted CNN+RNN
|primary incision knife||0.9881||0.9908||0.9819||0.9686||0.9959||0.9809||0.9814||0.9969||0.9994||0.9996|
|secondary incision knife||0.9924||0.9976||0.9977||0.9989||0.9982||0.9980||0.9972||0.9989||0.9996||0.9997|
|Corr. with prevalence||0.3246||0.2305||0.2793||0.2850||0.2955||0.2461||0.3922||0.2493||0.1309||0.1220|
|Corr. with prevalence||-0.4916||-0.4865||-0.4317||-0.3751||-0.4396||-0.4555||-0.5149||-0.5186||-0.5915||-0.6535|
smoothed boosted CNN+RNN
|primary incision knife||0.7944||0.8644||0.8121||0.8443||0.9081||0.7628||0.7823||0.9203||0.9696||0.9740|
|secondary incision knife||0.6524||0.8321||0.8434||0.9195||0.9120||0.7946||0.8903||0.9169||0.9615||0.9649|
|Corr. with prevalence||0.6474||0.5560||0.5974||0.5918||0.5600||0.6382||0.6166||0.5544||0.4443||0.4329|
|Corr. with prevalence||0.5477||0.5529||0.5899||0.5867||0.6371||0.5741||0.5261||0.4964||0.3838||0.3845|
Sequence labeling examples obtained with the best ensembles are illustrated and commented in Fig. 9. In summary, mistakes made by the best ensembles are mainly due to occlusions. To illustrate the problems that the proposed ensemble solves, Fig. 10 reports labeling sequences obtained at different ensemble complexity levels. This figure suggests that the same errors are made by all detectors, but these errors are progressively attenuated as the ensemble becomes more complex.
The label of image 1 (capsulorhexis forceps) has been correctly identified, but with a lower confidence level compared to previous images. The reason probably is that the forceps have remained closed for a long time and are therefore more difficult to recognize. In image 2, the capsulorhexis cystotome is detected as a hydrodissection cannula. The reason probably is that its distinctive claw-shaped tooltip is hidden in the incision and its distinctive elbow is out of the field of view. As soon as the elbow becomes visible (image 3), the correct label is assigned. In image 4, the phacoemulsifier handpiece is considered active, whereas it is not in contact with the eyeball yet. However, it touches the tear film, so the detector is almost correct. In image 5, one of the annotators indicated that the viscoelastic cannula is being used, although it is not actually visible: only indirect signs of presence (at the bottom) are visible; the detector was not able to recognize them. In image 6, the micromanipulator is partly mistaken for a Rycroft cannula: the explanation is similar for images 2 and 6. The reason why a hydrodissection cannula is detected in the former case and a Rycroft cannula in the latter probably comes from the RNN-based temporal modeling: hydrodissection cannulae are more likely at the beginning, Rycroft cannula are more likely at the end. In image 7, the grasper is not detected, probably because it is occluded by the hook. Finally, in image 8, a specimen bag is falsely detected, however the white string used for closing the bag is visible: the RNN-based temporal sequencer probably interpolated predictions from neighboring frames where the bag and the string are both visible.
6.3 Comparisons with Baseline Solutions
The proposed ensemble (obtained through “CNN+RNN boosting”, with median filtering for CATARACTS) is compared with various baseline methods in Table 5 . For each baseline, the statistical significance of the difference with the proposed solution is assessed using a paired sample t-test.
. For each baseline, the statistical significance of the difference with the proposed solution is assessed using a paired sample t-test.
|CATARACTS||Cholec80||p-value (paired sample t-test)|
|m||mAP||m||mAP||difference in m||difference in mAP|
|boosted CNN (weaker CNNs only)||0.9829||0.6192||0.9880||0.9501||2.479||0.003881|
|proposed ensemble (weaker CNNs only)||0.9900||0.6748||0.9917||0.9695||0.007271||0.01401|
|smoothed “sequentially” boosted CNN+RNN||0.9939||0.6956||0.9930||0.9741||0.002679||0.004047|
|proposed ensemble (unidirectional RNNs)||0.9957||0.7580||0.9936||0.9760||0.05397||0.07474|
|NASNet-A + 1 LSTM||0.9900||0.6949||0.9911||0.9723||0.001088||0.001972|
|NASNet-A features + 1 LSTM||0.9910||0.7264||0.9913||0.9755||0.02051||0.01078|
|boosted CNN + smoothing||0.9933||0.6735||0.9917||0.9703||0.001317||0.003009|
|linear-combination CNN ensemble||0.9913||0.6611||0.9917||0.9674||5.328||0.001204|
|smoothed linear-combination CNN+LSTM ensemble||0.9937||0.7010||0.9926||0.9733||0.002235||0.004602|
|EndoNet (Twinanda et al., 2017)||n/a||n/a||0.810||n/a||0.005394|
|DResSys (Roychowdhury et al., 2017)||0.9971||n/a||n/a||0.2936||n/a|
|CUMV (Hu and Heng, 2017)||0.9897||n/a||n/a||0.001682||n/a|
|TROLIS (Maršalkaitė et al., 2017)||0.9812||n/a||n/a||0.0204||n/a|
|proposed ensemble (union GT)||0.9938||0.7876||n/a||0.02266||0.2320|
|proposed ensemble (intersection GT)||0.9958||0.7585||0.001920||0.007045|
The first five baselines are variations on the proposed ensemble, as described above. All of these variations lead to decreased performance, with one exception: replacing bidirectional RNNs with unidirectional RNNs does not impact performance significantly. We note the good performance of ensembles obtained in the “weaker CNNs” experiment. In CATARACTS for instance, increases from 0.9663 for the best CNN (ResNet-152) to 0.9900 (), while in the “all CNNs” experiment, it increases from 0.9831 to 0.9961 (). On the downside, we also note that to achieve very high performance, good weak learners must be available.
The next six baselines were proposed to evaluate the relevance of each part of the proposed framework. The first two tests show that only using the best CNN (NASNet-A in the “all CNNs” experiment) or the best CNN and the best RNN (NASNet-A + 1 LSTM) is clearly suboptimal. Interestingly, the third experiment shows that LSTMs operating on NASNet-A features (the 4032 features of the next to last NASNet-A layer) are better than LSTMs operating on NASNet-A predictions (the outputs of the last layer). However, besides being more computationally intensive, RNNs operating on CNN features are not compatible with boosting across multiple CNN architectures: feature layers would not necessarily have compatible shapes and could therefore not be combined linearly. The fourth experiment shows that the RNN part of the proposed ensemble cannot simply be replaced with a median filter. The ensemble evaluated in the fifth experiment is similar to the proposed boosted CNN ensemble, in the sense that predictions from several CNNs are combined linearly inside a sigmoid function. The difference is that each CNN (the seven CNNs studied in this paper) is trained independently; the weight assigned to each CNN is trained through a gradient descent. This approach is similar to the ensemble method proposed by Roychowdhury et al. (2017). The result of this experiment is rather disappointing: the performance of the resulting ensemble is almost as good as the boosted CNN ensemble ( for , for AP). The only advantage of the proposed ensemble is that it is more compact: four CNNs (see Fig. 6) instead of seven. A similar ensemble is evaluated in the sixth experiment: one LSTM is trained independently on top of each of the seven CNNs and the predictions of these seven LSTMs is combined linearly inside a sigmoid function, again with weights obtained through a gradient descent. In CATARACTS, the ensemble predictions are then smoothed with a median filter. In that case, the proposed boosting approach is superior. We assume this superiority is mainly due to the proposed mechanism for boosting CNNs inside a “CNN+RNN” network (see section 4.5), since the performance of the linear-combination ensemble is close to that of the “sequentially” boosted “CNN+RNN”. Ideally, we would also compare the proposed solution with the end-to-end training of a “CNN+RNN” network, but the complexity of that model prevents any experimentation.
The next four baselines are recent solutions from the literature: EndoNet is from the original Cholec80 paper, the other three solutions are the top-ranking solutions of the CATARACTS challenge. We can see that the proposed solution is better than three of these solutions (EndoNet, CUMV and TROLIS) and not significantly worse than the other one (DResSys). One advantage of the proposed solution compared to DResSys is that it is more lightweight (less CNNs processing smaller images). The other advantage is that its unidirectional version, which is not significantly different from DResSys neither (), allows online video sequencing, while DResSys jointly analyzes batches of 20,000 frames.
The last two experiments reported in Table 5 evaluate the impact of the criterion chosen to define the ground truth in CATARACTS (exclusion of frames without a consensus). In those experiments, the ground truth is defined either as the union or the intersection of both expert interpretations, using all frames in the test videos. We can see that using those evaluation criteria decreases performance, in part because the most challenging frames (where experts disagree) are included, in part because the ground truth is of lower quality (more uncertain).
6.4 Sensitivity Analysis of the Boosted Video Labelers
To visualize what the CNNs have learned, one can rely on sensitivity analysis (Simonyan et al., 2014) and related metrics. Sensitivity is the gradient of the CNN predictions with respect to the pixel values: the pixel values influencing most the CNN predictions are highlighted. Recently, we proposed a variation on sensitivity called hue-constrained sensitivity (Quellec et al., 2017): the interpretation is similar, except that the three color components of a pixel are analyzed jointly rather than independently. Given a CNN and an input image with dimensions , the hue-constrained sensitivity heatmap of for is defined as:
where tensor is a matrix of ones with dimensions and where ’*’ denotes the element-wise tensor multiplication. It should be noted that and that all color components of a pixel in are multiplied by the same tensor element in , which ensures the desired hue preservation property (Quellec et al., 2017). Fig. 11 reports hue-constrained sensitivity heatmaps for all seven CNNs. It also reports heatmaps for , the second CNN (based on Inception-v4) added to the strong classifier in the “all CNNs” experiment (where is NASNet-A). This figure shows that, in CATARACTS, CNNs do not consider solely the tools, but also the anterior segment of the eye: the lens, which is modified by tools, the cornea, which is temporarily deformed by tools as they move, and the corneoscleral junction, where tools are inserted. One explanation is that each tool interacts differently with the eye and, therefore, analyzing the eye structures helps differentiating tools. Another explanation is that, in this dataset, the target labels are not related to tool presence, but rather to tool usage. So CNNs must be able to recognize whenever each tool is in contact with the eye. This hypothesis is backed up by the observation that responses from tissues are lower in Cholec80, where tool usage is simply defined as tool visibility. We notice, however, that the best CNNs (NASNet-A and VGG-19) have sparser heatmaps and that those heatmaps are more focused on the tools. Heatmaps obtained for (i.e NASNet-A) and have been analyzed jointly to assess their complementarity. Because the first image was already classified well by NASNet-A, the heatmap for is empty: the detections we see at the corner are just amplified noise (heatmap intensities have been normalized between 0 and 255). Similarly, in the second image, the phacoemulsifier handpiece at the center was detected well by NASNet-A, but not the forceps on the left: seems to focus on the forceps. In the third image, we note that NASNet-A did not focus primarily on the tool (it seemed disturbed by specular reflections) but does. In the last image, we also note a more focused heatmap for , compared to NASNet-A, although the grasper on the left (which was correctly detected by ) is not detected anymore. So, overall, and are indeed complementary. And, clearly, the heatmaps for Inception-v4 before and after a boosting step are very different.
Because our joint “CNN+RNN” boosting algorithm relies on the gradients of RNN predictions with respect to CNN predictions [see Eq. (9)], sensitivity analysis is also useful for RNNs in our case. These gradients are illustrated in a condensed form in Fig. 12: given an RNN , this figure shows , where:
For a lazy RNN, all coefficients outside the diagonal would be zero. Here, we observe that the diagonal is not even always dominant. This is particularly true for tools whose detection performance increases greatly after RNN boosting, such as needle holders, suture needles or Cholec80’s scissors (see Tables 3 and 4): the gradients of RNN predictions for those tools with respect to CNN predictions for other tools are very high. Clearly, RNNs are not lazy and quite useful for this task.
7 Discussion and Conclusions
A solution for labeling tool usage in cataract and cholecystectomy surgery videos has been presented. Following state-of-the-art video analysis solutions, it relies on convolutional neural networks (CNNs) for analyzing each frame in the video and on recurrent neural networks (RNNs) for analyzing the temporal sequencing throughout the entire surgery, based on the outputs of the CNNs. A novel framework for boosting a sequence labeler composed of CNNs and RNNs has been presented. The main motivation for this framework is the fact that “CNN+RNN” labelers cannot be trained from end to end, for complexity reasons. The framework allows to progressively improve the CNN and RNN parts of the system by adding weak classifiers (CNNs or RNNs) designed to improve the overall classification accuracy of the join system. In particular, like the theoretical end-to-end training solution, CNN training is supervised based on the outputs of the RNN block.
The proposed framework has several novelties. The main novelty lies in the boosting algorithm. CNN boosting had been proposed for multiclass classification problems (Moghimi et al., 2016). We adapted it for multilabel classification, showed its applicability to RNN boosting and, more importantly, introduced CNN boosting supervised based on the outputs of the RNN block. A second novelty lies in the proposed temporal sequence augmentation strategy: although very simple, it proved to be quite effective (see Fig. 5).
The proposed framework is quite general and is likely applicable outside the scope of surgery video analysis. However, it is of particular relevance for this application because many tools are very similar to one another (e.g. the cannulae or the forceps — see Fig. 4) but they are often used in a predefined order: using the temporal context (e.g. which tools have be used previously) is quite relevant for differentiating them. Therefore, it seems particularly useful to guide CNN training or boosting based on the temporal context. Experiments on two recent datasets (CATARACTS and Cholec80) for the task of tool usage annotation demonstrated its very good performance: the mean area under the ROC curve reaches up to m over a collection of 21 cataract surgery tools and up to m over a collection of 7 cholecystectomy tools.
If we look into the details of the proposed boosting solution, we first note that CNN boosting alone is disappointing: we found no significant difference between CNN boosting and a weighted sum of independently trained CNNs ( for , for AP), although the resulting architecture is more lightweight. The ability to boost CNNs based on the outputs of RNNs, on the other hand, leads to a significant improvement: joint “CNN+RNN” boosting is indeed significantly better than sequential “CNN+RNN” boosting ( for , for AP — see Table 5). Our explanation is that, when the CNN part is boosted independently of RNNs, much boosting effort is spent on trying to correct labeling errors, caused by previously selected CNNs, that RNNs could easily correct based on the temporal context: using temporally-filtered outputs to supervise boosting makes more sense. These observations support our hypothesis that CNNs should be trained to be complimentary to RNNs.
One advantage of the proposed approach is that its online version, which relies on unidirectional RNNs, does not perform significantly worse than its offline version, relying on bidirectional RNNs ( for , for AP — see Table 5). With slightly better performance, the offline version would be the preferred solution for report generation, surgical workflow optimization and surgical skill assessment. The online version, however, is the only valid solution for intraoperative warning or recommendation generation, provided that it is fast enough. Similarly to the bidirectional version (see Fig. 6), the online version relies on three weak CNNs: one based on NASNet-A, one based on Inception-v4 and one based on VGG-16. All three together, processing one frame takes 50.9 ms using one GeForce GTX 1080 Ti GPU by Nvidia (see Table 2). Videos of the CATARACTS dataset have a frame rate of 30 image per second (i.e. 33.3 ms per image). It means a faster GPU would be required for real-time video analysis. Alternatively, two GPUs can be used, as the CNN classifiers can be run in parallel (GPU 1: NASNet-A 24.6 ms per image, GPU 2: Inception-v4 and VGG-16 26.3 ms per image). Note that the use of median filters (with radii of 32 frames at most) delays predictions by one second. In Cholec80, the frame rate is 1 image per second, so computation times are not an issue.
The proposed framework compares favorably with state-of-the-art competing solutions (Twinanda et al., 2017; Hu and Heng, 2017; Maršalkaitė et al., 2017). In terms of , it does not differ significantly from the winner of the CATARACTS challenge (Roychowdhury et al., 2017). However, it has the advantage of being more lightweight and, more importantly, of allowing online video analysis.
This study has a few limitations. In particular, the same dataset was used to train CNNs and RNNs. Because CNN predictions are likely better in the learning set than in the validation and test sets, RNNs are trained under too favorable conditions, which could lead to overfitting. Because the number of learning videos is limited, we decided to use all of them for training CNNs and RNNs. We simply relied on early stopping to discard overfitted configurations. Another limitation is that we did not explore data rebalancing techniques (Sahu et al., 2017) or weighted cost functions to deal with multi-label imbalance, assuming that boosting can deal with it satisfactorily.
In conclusion, an accurate solution for labeling tool usage in surgery videos has been presented. In view of the good performance, automatic surgery monitoring can now be envisaged seriously (Charrière et al., 2017). We are currently exploring solutions to provide useful feedbacks to the surgeon, based on information collected during the surgery. Support to beginners is a particular relevant application, but many more can be envisioned for the near future.
- Al Hajj et al. (2017) Al Hajj, H., Lamard, M., Charrière, K., Cochener, B., Quellec, G., 2017. Surgical tool detection in cataract surgery videos through multi-image fusion inside a convolutional neural network. In: Proc IEEE EMBC. Jeju Island, Korea.
- Bodenstedt et al. (2017) Bodenstedt, S., Wagner, M., Katić, D., Mietkowski, P., Mayer, B., Kenngott, H., Müller-Stich, B., Dillmann, R., Speidel, S., Feb. 2017. Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis. Tech. Rep. arXiv:1702.03684 [cs], Karlsruhe Institute of Technology.
- Bouget et al. (2017) Bouget, D., Allan, M., Stoyanov, D., Jannin, P., 2017. Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35, 633–654.
- Cadène et al. (2016) Cadène, R., Robert, T., Thome, N., Cord, M., Oct. 2016. M2cai workflow challenge: convolutional neural networks with time smoothing and hidden Markov model for video frames classification. Tech. Rep. arXiv:1610.05541 [cs], Université de Pierre et Marie Curie.
- Charrière et al. (2017) Charrière, K., Quellec, G., Lamard, M., Martiano, D., Cazuguel, G., Coatrieux, G., Cochener, B., Nov. 2017. Real-time analysis of cataract surgery videos using statistical models. Multimed Tools Appl 76 (21), 22473–22491.
- Chen et al. (2017) Chen, H., Chen, J., Hu, R., Chen, C., Wang, Z., Mar. 2017. Action recognition with temporal scale-invariant deep learning framework. China Communications 14 (2), 163–172.
- Cho et al. (2014) Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y., Oct. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In: Proc SSST. Doha, Qatar, pp. 103–111, arXiv: 1409.1259.
- Dergachyova et al. (2016) Dergachyova, O., Bouget, D., Huaulmé, A., Morandi, X., Jannin, P., Jun. 2016. Automatic data-driven real-time segmentation and recognition of surgical workflow. Int J Comput Assist Radiol Surg 11 (6), 1081–1089.
- Dipietro et al. (2016) Dipietro, R., Lea, C., Malpani, A., Ahmidi, N., Vedula, S., Lee, G., Lee, M., Hager, G., Oct. 2016. Recognizing surgical activities with recurrent neural networks. In: Proc MICCAI. Athens, Greece, pp. 551–558.
- Donahue et al. (2017) Donahue, J., Hendricks, L., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., Darrell, T., Apr. 2017. Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans Pattern Anal Mach Intell 39 (4), 677–691.
- Feng et al. (2016) Feng, Y., Li, Y., Luo, J., Dec. 2016. Learning effective gait features using LSTM. In: Proc IEEE ICPR. Cancun, Mexico, pp. 325–330.
- Freund and Schapire (1997) Freund, Y., Schapire, R. E., Aug. 1997. A Decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55 (1), 119–139.
- Friedman (2001) Friedman, J. H., Oct. 2001. Greedy function approximation: A gradient boosting machine. Ann Stat 29 (5), 1189–1232.
- Gammulle et al. (2017) Gammulle, H., Denman, S., Sridharan, S., Fookes, C., Mar. 2017. Two stream LSTM: A deep fusion framework for human action recognition. In: Proc IEEE WACV. Santa Rosa, CA, USA, pp. 177–186.
Gao et al. (2016)
Gao, Y., Rong, W., Shen, Y., Xiong, Z., Jul. 2016. Convolutional neural network based sentiment analysis using Adaboost combination. In: Proc IEEE IJCNN. Vancouver, Canada, pp. 1333–1338.
- He et al. (2016a) He, K., Zhang, X., Ren, S., Sun, J., Jun. 2016a. Deep residual learning for image recognition. In: Proc CVPR. Las Vegas, NV, USA, pp. 770–778.
- He et al. (2016b) He, K., Zhang, X., Ren, S., Sun, J., Oct. 2016b. Identity mappings in deep residual networks. In: Proc ECCV. Lecture Notes in Computer Science. Springer, Cham, Amsterdam, The Netherlands, pp. 630–645.
- Hochreiter and Schmidhuber (1997) Hochreiter, S., Schmidhuber, J., Nov. 1997. Long short-term memory. Neural Comput 9 (8), 1735–1780.
- Hu and Heng (2017) Hu, X., Heng, P.-A., Nov. 2017. Surgical tool annotation in cataract surgery videos. Tech. rep., Chinese University of Hong Kong.
- Huang et al. (2017) Huang, G., Liu, Z., Maaten, L. v. d., Weinberger, K. Q., Jul. 2017. Densely connected convolutional networks. In: Proc IEEE CVPR. Honolulu, HI, USA, pp. 2261–2269.
- Ji et al. (2013) Ji, S., Xu, W., Yang, M., Yu, K., Jan. 2013. 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Mach Intell 35 (1), 221–231.
- Jin et al. (2016) Jin, Y., Dou, Q., Chen, H., Yu, L., Heng, P.-A., Oct. 2016. EndoRCN: recurrent convolutional networks for recognition of surgical workflow in cholecystectomy procedure video. Tech. rep., The Chinese University of Hong Kong.
- Khorrami et al. (2016) Khorrami, P., Le, P., Brady, K., Dagli, C., Huang, T., Sep. 2016. How deep neural networks can improve emotion recognition on video data. In: Proc IEEE ICIP. Phoenix, AZ, USA, pp. 619–623.
- Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G. E., Dec. 2012. ImageNet classification with deep convolutional neural networks. In: Proc NIPS. Vol. 25. Granada, Spain, pp. 1097–1105.
- Lalys and Jannin (2014) Lalys, F., Jannin, P., May 2014. Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9 (3), 495–511.
- Lea et al. (2016a) Lea, C., Vidal, R., Hager, G. D., May 2016a. Learning convolutional action primitives for fine-grained action recognition. In: Proc IEEE ICRA. Stockholm, Sweden, pp. 1642–1649.
- Lea et al. (2016b) Lea, C., Vidal, R., Reiter, A., Hager, G., Oct. 2016b. Temporal convolutional networks: a unified approach to action segmentation. In: Proc ECCV. Amsterdam, The Netherlands, pp. 47–54.
- Maršalkaitė et al. (2017) Maršalkaitė, G., Bialopetravičius, J., Armaitis, J., Nov. 2017. Towards robust tool identification for cataract surgery. Tech. rep., Oxipit, UAB.
- Mason et al. (1999) Mason, L., Baxter, J., Bartlett, P. L., Frean, M. R., Dec. 1999. Boosting algorithms as gradient descent. In: Proc NIPS. Vol. 12. Denver, CO, USA, pp. 512–518.
- Mishra et al. (2017) Mishra, K., Sathish, R., Sheet, D., Jul. 2017. Learning latent temporal connectionism of deep residual visual abstractions for identifying surgical tools in laparoscopy procedures. In: Proc IEEE CVPR Works. Honolulu, HI, USA, pp. 2233–2240.
- Moghimi et al. (2016) Moghimi, M., Saberian, M., Yang, J., Li, L.-J., Vasconcelos, N., Belongie, S., Sep. 2016. Boosted convolutional neural networks. In: Proc BMVC. York, UK.
- Primus et al. (2018) Primus, M., Putzgruber-Adamitsch, D., Taschwer, M., Münzer, B., El-Shabrawi, Y., Böszörmenyi, L., Schoeffmann, K., Feb. 2018. Frame-based classification of operation phases in cataract surgery videos. In: Proc MMM. Vol. 10704 LNCS. Bangkok, Thailand, pp. 241–253.
- Quellec et al. (2017) Quellec, G., Charrière, K., Boudi, Y., Cochener, B., Lamard, M., Jul. 2017. Deep image mining for diabetic retinopathy screening. Med Image Anal 39, 178–193.
- Quellec et al. (2014) Quellec, G., Lamard, M., Cochener, B., Cazuguel, G., Dec. 2014. Real-time segmentation and recognition of surgical tasks in cataract surgery videos. IEEE Trans Med Imaging 33 (12), 2352–2360.
- Quellec et al. (2015) Quellec, G., Lamard, M., Cochener, B., Cazuguel, G., Apr. 2015. Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials. IEEE Trans Med Imaging 34 (4), 877–887.
- Raju et al. (2016) Raju, A., Wang, S., Huang, J., Oct. 2016. M2CAI surgical tool detection challenge report. Tech. rep., University of Texas at Arlington.
- Roychowdhury et al. (2017) Roychowdhury, S., Bian, Z., Vahdat, A., William G., M., Nov. 2017. Identification of surgical tools using deep neural networks. Tech. rep., D-Wave Systems Inc.
- Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., Fei-Fei, L., Apr. 2015. ImageNet large scale visual recognition challenge. Int J Comput Vis 115 (3), 211–252.
- Sahu et al. (2016) Sahu, M., Mukhopadhyay, A., Szengel, A., Zachow, S., Oct. 2016. Tool and phase recognition using contextual CNN features. Tech. Rep. arXiv:1610.08854 [cs.CV], Zuse Institute Berlin.
- Sahu et al. (2017) Sahu, M., Mukhopadhyay, A., Szengel, A., Zachow, S., Jun. 2017. Addressing multi-label imbalance problem of surgical tool detection using CNN. Int J Comput Assist Radiol Surg 12 (6), 1013–1020.
Schuster and Paliwal (1997)
Schuster, M., Paliwal, K. K., Nov. 1997. Bidirectional recurrent neural networks. IEEE Trans Signal Process 45 (11), 2673–2681.
- Shen et al. (2017) Shen, D., Wu, G., Suk, H.-I., Jun. 2017. Deep learning in medical image analysis. Annu Rev Biomed Eng 19, 221–248.
- Simonyan et al. (2014) Simonyan, K., Vedaldi, A., Zisserman, A., Apr. 2014. Deep inside convolutional networks: visualising image classification models and saliency maps. In: ICLR Workshop. Calgary, Canada.
- Simonyan and Zisserman (2014) Simonyan, K., Zisserman, A., Dec. 2014. Two-stream convolutional networks for action recognition in videos. In: Proc NIPS. Vol. 27. Montreal, Canada, pp. 568–576.
- Simonyan and Zisserman (2015) Simonyan, K., Zisserman, A., May 2015. Very deep convolutional networks for large-scale image recognition. In: Proc ICLR. San Diego, CA, USA.
- Singh et al. (2016) Singh, B., Marks, T., Jones, M., Tuzel, O., Shao, M., Jan. 2016. A multi-stream bi-directional recurrent neural network for fine-grained action detection. In: Proc IEEE CVPR. Las Vegas, NV, USA, pp. 1961–1970.
Szegedy et al. (2017)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A., Feb. 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: Proc AAAI. San Francisco, CA, USA, pp. 4278–4284.
- Szegedy et al. (2015a) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Jun. 2015a. Going deeper with convolutions. In: Proc IEEE CVPR. Boston, MA, USA, pp. 1–9.
- Szegedy et al. (2015b) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., Dec. 2015b. Rethinking the Inception architecture for computer vision. Tech. Rep. arXiv:1512.00567 [cs], Google.
- Tao et al. (2013) Tao, L., Zappella, L., Hager, G. D., Vidal, R., Sep. 2013. Surgical gesture segmentation and recognition. In: Proc MICCAI. Nagoya, Japan, pp. 339–346.
- Tran et al. (2017) Tran, D., Sakurai, R., Yamazoe, H., Lee, J.-H., 2017. Phase segmentation methods for an automatic surgical workflow analysis. Int J Biomed Imaging 2017, 1985796.
- Trikha et al. (2013) Trikha, S., Turnbull, A. M. J., Morris, R. J., Anderson, D. F., Hossain, P., Apr. 2013. The journey to femtosecond laser-assisted cataract surgery: new beginnings or a false dawn? Eye (Lond) 27 (4), 461–473.
- Twinanda et al. (2016) Twinanda, A. P., Mutter, D., Marescaux, J., de Mathelin, M., Padoy, N., Oct. 2016. Single- and multi-task architectures for surgical workflow challenge at M2cai 2016. Tech. Rep. arXiv:1610.08844 [cs], University of Strasbourg.
- Twinanda et al. (2017) Twinanda, A. P., Shehata, S., Mutter, D., Marescaux, J., de Mathelin, M., Padoy, N., Jan. 2017. EndoNet: A deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36 (1), 86–97.
- Walach and Wolf (2016) Walach, E., Wolf, L., Oct. 2016. Learning to count with CNN boosting. In: Proc ECCV. Vol. 9906. Amsterdam, The Netherlands, pp. 660–676.
- Wang et al. (2017) Wang, X., Gao, L., Song, J., Shen, H., Apr. 2017. Beyond frame-level CNN: saliency-aware 3D CNN with LSTM for video action recognition. IEEE Signal Processing Letters 24 (4), 510–514.
- Zappella et al. (2013) Zappella, L., Béjar, B., Hager, G., Vidal, R., Oct. 2013. Surgical gesture classification from video and kinematic data. Med Image Anal 17 (7), 732–745.
Zhang et al. (2016)
Zhang, F., Du, B., Zhang, L., Mar. 2016. Scene classification via a gradient boosting random convolutional network framework. IEEE Trans Geosci Remote Sens 54 (3), 1793–1802.
- Zia et al. (2016) Zia, A., Castro, D., Essa, I., Oct. 2016. Fine-tuning deep architectures for surgical tool detection. Tech. rep., Georgia Institute of Technology.
- Zoph et al. (2017) Zoph, B., Vasudevan, V., Shlens, J., Le, Q. V., Jul. 2017. Learning transferable architectures for scalable image recognition. arXiv:1707.07012 [cs, stat].