Coronary artery disease (CAD) is the most common type of heart disease . CAD occurs when atherosclerotic plaque builds up in the wall of the coronary arteries. This may lead to stenosis, i.e. narrowing or occlusion of the coronary artery lumen, limiting blood supply to the myocardium and potentially leading to myocardial ischemia. Atherosclerotic plaque can be classified according to its composition into calcified plaque, non-calcified plaque, and mixed plaque, i.e plaque containing calcified and non-calcified components . Calcified plaque is considered stable and its amount in the coronary arteries is a strong predictor of cardiovascular events . Unlike calcified plaque, non-calcified and mixed plaque are considered unstable and more prone to rupture. Such rupture may lead to acute coronary syndrome and could result in irreversible damage to the myocardium, i.e. myocardial infarction [4, 5]. As different types of plaque and varying grades of stenosis lead to different patient management strategies, it is crucial to detect and characterize coronary artery plaque and stenosis [6, 7].
Coronary CT angiography (CCTA) is a well-established modality for identification, as well as for exclusion, of patients with suspected CAD. It allows for noninvasive detection and characterization of coronary artery plaque and grading of coronary artery stenosis . To day, these tasks are typically performed in the clinic by visual assessment , or semi-automatically by first utilizing lumen and arterial wall segmentation and thereafter, defining the presence of plaque or stenosis . However, the former suffers from substantial interobserver variability, even when performed by experienced experts , while the latter is dependent on coronary artery lumen and wall segmentation which is typically time-consuming and cumbersome, especially in images with extensive atherosclerotic plaque or imaging artefacts .
Given the importance of plaque detection, a number of methods for coronary artery plaque detection and quantification have been proposed. Thus far, (semi-)automatic methods have been designed to detect either calcified or non-calcified plaque. Several methods have been developed to automatically segment and quantify calcifications in the coronary arteries in non-contrast CT and CCTA scans (e.g. [3, 11, 12, 13]
). These methods employ machine learning to analyze axial reconstruction of CT scans. Typically, an excellent performance approaching the level of an expert is achieved. On the contrary, automatic detection and quantification of non-calcified coronary plaque using CCTA has been less investigated. While the visual detection of calcified plaque is straightforward in CT/CCTA due to its higher CT density, the detection of non-calcified and especially mixed plaque is more challenging because of low contrast with adjacent tissues. Therefore, unlike methods detecting calcifications, standard visual evaluation as well as (semi-)automatic approaches detecting non-calcified plaque typically analyze straightened multi-planar reformatted (MPR) images. MPR images allow better visualization of the arterial lumen and identification of difficult to delineate non-calcified plaque in the arterial wall. To detect and quantify non-calcified plaque, previously proposed methods have performed manual or semi-automatic thresholding on CT values in predefined regions of interest [14, 15]. Typically, these methods require substantial manual interactions by experts. Even though the presence of mixed plaque is usually reported in clinical visual assessment, to the best of our knowledge, automatic methods detecting and quantifying such plaque have not yet been presented.
As stenosis detection and grading is highly important, a number of methods have been developed to (semi-)automatically detect and grade coronary artery stenosis in CCTA . These methods either utilize machine learning approaches to analyze the vicinity of the coronary artery centerline (e.g. [16, 17, 18]), or segment arterial lumen (e.g. [19, 20, 21, 22, 23]). Algorithms that utilize lumen segmentation for stenosis detection first delineate coronary artery lumen and subsequently detect and quantify stenosis by analyzing local changes and anomalies in the lumen of the delineated artery. Shahzad et al. 
first extracted the centerline of the artery and subsequently employed a graph cut approach and robust kernel regression to segment the arterial lumen. Thereafter, to detect and grade coronary stenosis, the diameter of the segmented lumen was compared with the expected diameter of a healthy lumen. The expected diameter of the healthy lumen was estimated by regression on the diameters of the lumina along the coronary artery. Wang et al. employed a level-set model to separately segment the inner and the outer arterial walls. Thereafter, a comparison between these arterial wall profiles enabled the detection and grading of stenosis. Furthermore, algorithms that exploit machine learning for stenosis detection first compute a number of features along the centerline of an artery to describe local image intensities and arterial geometry. Subsequently, they use a supervised classifier to detect and quantify stenosis. For example, Zuluaga et al. 
formulated arterial stenosis as anomalous vascular cross-sections along the artery centerline. The shape of the vascular cross-sections and their intensity profiles were described by hand-crafted features, and then the abnormal vascular cross-sections were detected by a support vector machine. Sankaran et al.
first estimated healthy diameters of the coronary arteries using downstream and upstream properties of coronary tree vasculature as features for random forest regressors. Then, the degree of a stenosis was estimated based on the ratio of the local artery diameter, estimated using maximum inscribed spheres, to the estimated healthy diameter.
Here, we present a method to automatically detect and characterize the type of coronary artery plaque, as well as to detect and determine the anatomical significance of coronary artery stenosis in CCTA scans. To perform the automatic comprehensive analysis, a multi-task recurrent convolutional neural network (RCNN) is employed to analyze the vicinity along the extracted centerline in an MPR image and to simultaneously carry out two classification tasks. In the first task, the network detects and characterizes the type of the coronary artery plaque, i.e. no plaque, non-calcified, mixed or calcified plaque. In the second task, the network detects and determines the anatomical significance of the coronary artery stenosis, i.e. no stenosis, non-significant or significant stenosis. The RCNN analyzes the vicinity along the artery centerline, which is defined as a sequence of small volumes along the centerline. This definition enables the RCNN, built from a 3D convolutional neural network (CNN) and a recurrent neural network (RNN) connected in series 
, to extract image features from smaller volumes regardless of plaque length, and then to aggregate all features extracted along the plaque. Our contributions are fourfold. Firstly, we propose to jointly classify plaque and stenosis, while previously proposed methods have detected either plaque or stenosis. Secondly, unlike previous automatic methods, our method does not require segmentation of the coronary artery lumen and/or wall nor exploiting geometric information about the artery lumen. Instead, it only requires the coronary artery centerline. Thirdly, we are the first to use deep neural networks to approach the task of coronary artery plaque and stenosis analysis, or more specifically a3D-CNN paired with an RNN to analyze medical data. Finally, previous works for classifying coronary artery plaque require detailed reference annotations for each voxel affected by the plaque. This kind of manual annotations is extremely challenging and time consuming. In the presented work, we employ only weakly annotated reference (start- and end-points of a lesion with a single label for all voxels in that lesion) to detect and characterize the plaque and thereby substantially simplify the manual annotation procedure.
Ii-a Patient and image data
This study includes retrospectively collected clinically obtained CCTA scans of 163 patients (age: years, 126 males) acquired in our hospital between 2012 and 2016. The Institutional Ethical Review Board waived the need for informed consent. All CCTA scans were acquired using an ECG-triggered step-and-shoot protocol on a 256-detector row scanner (Philips Brilliance iCT, Philips Medical, Best, The Netherlands). A tube voltage of 120 kVp and tube current between 210 and 300 mAs were used. For patients kg contrast medium was injected using a flow rate of 6 mL/s for a total of 70 mL iopromide (Ultravist 300 mg I/mL, Bayer Healthcare, Berlin, Germany), followed by a 50 mL mixed contrast medium and saline (50:50) flush, and a 30 mL saline flush. For patients kg the flow rate was 6.7 mL/s and the volumes of the boluses were 80, 67 and 40 mL, respectively. Images were reconstructed to an in-plane resolution ranging from 0.38 to 0.56 mm, with 0.9 mm slice thickness and 0.45 mm slice increment.
In each CCTA image, centerlines of the coronary arteries were extracted using the method previously described by Wolterink et al. . The method requires manual placement of a single seed point in the artery of interest, after which the arterial centerline is extracted between the ostium and the most distal point as visualized in the CCTA image. Using the extracted centerlines, a 0.3 mm isotropic straightened MPR image was reconstructed for each artery and used for further analysis.
Ii-B Reference standard
To define a reference standard for atherosclerotic plaque and coronary artery stenosis, MPR images of coronary arteries were employed (Fig. 1). As only arteries with a diameter greater than 1.5 mm are clinically evaluated with CCTA , only those were annotated in this study. Plaque type and anatomical significance of the stenosis were manually annotated by an expert using custom-built software and following the guidelines of the Society of Cardiovascular Computed Tomography (SCCT) for reporting coronary artery disease . For each plaque, the expert marked its start- and end-points, its type (non-calcified, calcified, or mixed i.e. containing both non-calcified and calcified components), and the anatomical significance of the stenosis caused by the plaque (non-significant i.e. with luminal narrowing, significant i.e. with luminal narrowing). The significance of a stenosis was determined by visual estimation of the maximal grade of luminal narrowing caused by the plaque. Note that the plaque voxels were not segmented, but only the part of the artery containing the plaque was identified (Fig. 1b). The expert also annotated a number of segments of arteries without plaque and stenosis in different patients. We refer to the annotated and analyzed parts of the arteries as segments. However, note that these are not the anatomically defined coronary artery segments.
As patient management and treatment strategies depend on diagnosis at the segment-, artery- and patient-level [6, 7], we additionally evaluate the ability of the proposed method to detect stenosis and to classify its anatomical significance on an artery- and patient-level. Therefore, segment-level labels provided by the expert were translated to the artery- and patient-level as follows. Arteries were labeled according to the most severe stenosis significance among their segments. If no stenosis of any significance was found in any of the annotated segments, the artery was considered non-stenotic. Likewise, patients were labeled according to the most severe stenosis significance among their arteries, or considered as having no stenosis.
The dataset of 163 patients contained 1259 manually labeled arterial segments in 534 arteries. The manually annotated segments included 37 non-calcified, 161 mixed and 317 calcified plaques that caused non-significant stenosis. Additionally, there were segments with 29 non-calcified, 91 mixed and 41 calcified plaques that caused significant stenosis. Moreover, 583 segments without plaque and stenosis were annotated. Of the annotated segments, 528 were in the left anterior descending artery (LAD) or one of its branches, 305 were in the left circumflex artery (LCX) or one of its branches, and 426 were in the right coronary artery (RCA) or one of its branches.
To assess the interobserver agreement, a second trained observer, blinded to the reference standard, annotated the same set of arteries following the same guidelines.
To detect and characterize the type of coronary artery plaque, as well as to detect and determine the anatomical significance of coronary artery stenosis, an RCNN is designed. An illustration of the proposed workflow is shown in Fig. 2. Recently, RCNNs have been successfully used for video recognition and description (e.g. [26, 27, 28]), object recognition (e.g. ), speech modeling , and in medical image analysis (e.g. [31, 32, 33, 34]). RCNNs connect a CNN with an RNN in series to analyze a sequential input. The CNN extracts image features for each element of the sequence independently (e.g. frame in a video clip, word in a sentence, cardiac phase in cardiac cycle), and these extracted features are then fed to the RNN that analyzes the relevant sequential dependencies in the whole sequence.
In this work, the reference was defined following clinical practice where parts of the arteries are classified with respect to plaque and stenosis. Hence, only arterial segments containing plaque were identified, instead of e.g. annotating all cross-sections of the arterial lumen or all voxels in the arterial wall. Given that the appearance of the plaque along the whole segment is important for characterization of its type and for determination of stenosis presence and significance, the artery along the entire segment needs to be analyzed. Instead of extracting a single, possibly large, volume covering the segment at hand, we represent the segment as a sequence of small volumes along its centerline. This enables us to employ a relatively shallow CNN to extract image features from smaller volumes independently. A shallow CNN that analyzes smaller volumes may have fewer parameters and is therefore less prone to overfitting. To aggregate and analyze the features extracted from all small volumes along the segment, we employ an RNN. For the whole analyzed sequence, the network 1) detects and characterizes coronary artery plaque i.e. classifies the segment as either containing no plaque, containing non-calcified, mixed or calcified plaque, and 2) detects and determines the anatomical significance of coronary artery stenosis, i.e. classifies the segment as either containing no stenosis, or containing non-significant or significant stenosis.
An illustration of the proposed RCNN is shown in Fig. 3. The input of the network is a sequence with a maximum length of 25 cubes of
voxels with stride of 5 voxels extracted from the MPR along the coronary artery centerline. The maximal length was chosen based on the longest annotated plaque in the training set. The size of the cube was chosen so that it contains the whole arterial lumen and the vicinity of the artery that may be needed in case of positive remodeling. Each cube is analyzed by a 3D CNN. The CNN consists of three convolutional layers with kernels of elements, with 32, 64, 128 filters, respectively. Each convolutional layer is followed by a max-pooling layer and batch normalization 
. The features extracted by the CNN are fed to the RNN. The RNN consists of 2 layers of 64 Gated Recurrent Units (GRUs)37]
are used in both CNN and RNN layers as activation functions, except for the output layer of the RNN. To perform both classification tasks simultaneously, the output of the last layer of the RNN is fed into two separate multi-class softmax classifiers. The first classifier has four output units for detection of plaque and characterization of its type (no-plaque, non-calcified, mixed, calcified). The second classifier has three output units for detection of stenosis and determination of its anatomical significance (no-stenosis, non-significant stenosis, significant stenosis). The RCNN has a total number of 340,295 parameters.
Performance of the network was evaluated on segment-, artery- and patient-levels. For segment-level evaluation, only the predicted labels along the centerlines that fall within the manually annotated segment boundaries were considered. For artery-level evaluation, all predicted labels along the complete artery centerline were taken into account. For patient-level evaluation, all predicted labels along the complete centerlines of all arteries of a patient were taken into account.
An automatically labeled segment is considered to be a true positive in the classification of plaque type or stenosis significance when it has an overlap of at least 1 mm with a manually annotated segment sharing its label. On the contrary, a segment is considered to be true positive in the detection of plaque absence (no plaque) or stenosis absence (no stenosis), only when no point along the segment has any plaque or stenosis, respectively.
As most patients have multiple arterial plaques of different types, the evaluation of plaque detection and characterization was performed on a segment-level only. The average accuracy of the prediction over all segments and labels, i.e. the average percentage of correctly labeled segments, was computed. To assess the overall performance, the unweighted average of F1 score was computed. This was done for each label separately, and then computing the unweighted mean across all labels, averaged over all segments. Given the multiple categories of the plaque labels, unweighted Cohen’s metric was used to measure the reliability between the predicted plaque labels and the reference standard.
The evaluation of stenosis detection and characterization was performed on segment-, artery- and patient-levels. Automatically determined stenosis significance for an artery or a patient is considered true positive if any of the automatically detected labels along the artery centerline or patients arteries match the reference label of that artery or patient, respectively. On the contrary, an artery or patient is considered to be true positive in the detection of stenosis absence (no stenosis) only when no stenosis was detected at any point along the artery or in any of the patient’s arteries, respectively. As for plaque evaluation, the average accuracy and the unweighted average F1 score were computed to assess the overall agreement for predicting the stenosis labels. Given the grading of the stenosis, Cohen’s linearly weighted metric was used to measure the reliability between the predicted stenosis labels and the reference standard.
Interobserver reliability for both plaque and stenosis analyses was assessed using the same metrics by comparing the annotations of the second observer to the reference standard.
V Experiments and results
From the available dataset consisting of 163 patients, CCTA scans of 81 (50%) and 17 (10%) patients were randomly chosen for training and validation, respectively. The scans of the remaining 65 (40%) patients were used for testing the method. All CNN and RNN hyperparameters were determined in preliminary experiments using the training and validation scans only.
Prior to training, several data augmentation techniques were utilized to increase the training set size. First, to make the network invariant to rotations around the artery centerline, random rotations between 0 and 360 degrees around the coronary artery centerline were applied to all cubes of a sequence. Second, to make the network invariant to slight inaccuracies in manual annotations of the points defining the segment, a sequence of a segment was varied by randomly choosing centers of cubes with a stride of 5 voxels with a uniform random shift between voxels along the MPR centerline. Third, to make the network robust to possible inaccuracies in the extraction of the coronary artery centerline, the center of each cube was randomly shifted around its origin by up to 2 voxels, in any direction.
The network was trained with mini-batches containing only the manually annotated segments. In the dataset, the distribution of plaque types and stenosis grades is unbalanced (Section II-B
). To avoid a potential bias towards the most common type of plaque and stenosis in the dataset (i.e. calcified plaque with non-significant stenosis), a stratified random data sampling was performed during the training. Each training iteration included two distinct mini-batches. One mini-batch contained segments balanced with respect to their plaque classes regardless of the stenosis significance. A second mini-batch contained segments balanced with respect to the stenosis classes regardless of the plaque type. For each segment, a sequence of cubes, spanning the entire length of the segment, was extracted from the MPR volume along the artery centerline. The categorical cross-entropy was employed as loss function of each softmax classifier and the L2 regularization was used withfor all layers. The loss of the network was defined as the average of the two individual losses (Eq. 1).
are one-hot encoded vectors for the labels of plaque () and stenosis (), respectively, and and
are the softmax output probabilities for the plaque and stenosis, respectively.is a trainable weight in the network (), where is the total number of trainable parameters in the network.
During training, mini-batches of 36 sequences (i.e. segments) were used to minimize the loss function with Adam optimizer  with constant learning rate , and a random dropout  of 50% was applied in each recurrent layer to prevent overfitting. Training was performed for 50,000 iterations (Fig. 6(a)). To evaluate the effect of data augmentation during training, an identical network was trained without any augmentation and the accuracies during training with and without data augmentation are illustrated in Fig. 6(b). In these figures, the stability and the convergence of the training process with data augmentation are shown, while training without data augmentation shows signs of overfitting on the training set where accuracies on the validation set decrease.
Unlike in the training phase, where the defined start- and end-points of a segment were used, during testing, segments were not defined, and labels were predicted along the whole analyzed artery. Therefore, all points along the coronary centerline were classified and labeled by the network. This was performed by feeding the network a fixed-length sequence centered around each centerline point. The fixed-length sequence consisted of 5 cubes with a stride of 5 voxels extracted along the coronary artery centerline. These parameters were optimized in preliminary experiments on the training set. The output probabilities for plaque and stenosis were then assigned to the center of the evaluated sequence. Thereafter, a label for plaque and a label for stenosis was defined for each centerline point by the class having the highest probability in each task separately.
V-a Plaque detection and characterization
To assess the performance of the proposed RCNN for plaque detection and characterization (no plaque, non-calcified, mixed, calcified), we first evaluated the performance for each annotated segment. Table I
lists the confusion matrix showing the results. Next, we evaluated the performance of the proposed method for detection of plaque (plaque of any type vs. no plaque), and compared it with plaque detection and characterization at the segment level. These results are summarized in TableII. To allow comparison of the automatic analysis with that of an expert, Table II additionally provides results achieved by the second observer. In brief, for detection of plaque (plaque of any type vs. no plaque), the proposed method achieved a segment-level accuracy of 0.85 and the second observer reached an accuracy of 0.90. For detection and characterization of plaque (no plaque, non-calcified, mixed, calcified), the proposed method and the second observer achieved segment-level accuracies of 0.77 and 0.80 with unweighted of 0.61 and 0.67, respectively. Finally, to evaluate whether the performance of the method depends on the analyzed artery, Table III lists the performance obtained for the detection and characterization of plaque (no plaque, non-calcified, mixed, calcified) at the segment-level in the different coronary arteries (LAD / LCX / RCA). Plaque was most accurately detected in LCX (0.81) and the least in RCA (0.72). Examples of automatically predicted classification probabilities in different arteries are shown in Fig. 10. An example of an artery with predicted labels for plaque characterization is shown in Fig. 11(b).
|Plaque Analysis||Stenosis Analysis|
V-B Stenosis detection and characterization
To assess the performance of the proposed RCNN for detection and determination of the anatomical significance of stenosis, we analyzed the results achieved on the segment-, artery- and patient-levels. The obtained confusion matrices are calculated and shown in Table IV. To get an insight into the ability of the method to classify stenosis, Table II lists the performance obtained for detection and determination of the anatomical significance of the stenosis (no stenosis, non significant stenosis, significant stenosis) at the segment-, artery- and patient-level. The automatic method achieved accuracy of 0.80, 0.76 and 0,75, and the linearly weighted of 0.68, 0.66 and 0.67 at the segment-, artery- and patient-level, respectively. For the second observer the accuracies were 0.83, 0.80 and 0.75, and the linearly weighted were 0.70, 0.72 and 0.70, respectively. Given the importance of the detection of the anatomically significant stenosis, the performance of the automatic method as well as the performance of the second observer for the this task (significant stenosis vs. no stenosis or non-significant stenosis) were evaluated. The automatic method achieved accuracy of 0.94, 0.93 and 0.85 at the segment-, artery- and patient-level, respectively. For the second observer these were 0.91, 0.92 and 0.83, respectively. Table II also details these results. Finally, to compare the performance in the different coronary arteries, the performance achieved for the detection and determination of the anatomical significance of the stenosis (no stenosis, non-significant stenosis, significant stenosis) at the segment-level in the three main coronary arteries (LAD / LCX / RCA) was evaluated. The analysis showed that similar accuracies were achieved in all three arteries (0.81 for LAD, 0.80 for LCX and 0.79 for RCA). Detailed results are listed in Table III. Examples of automatically predicted classification probabilities in different arteries are shown in Fig. 10. An example of an artery with predicted labels for stenosis characterization is shown in Fig. 11(c).
V-C Impact of the RCNN architecture
To establish the value of the recurrent nature of the proposed network, an additional experiment was performed in which a network with an identical CNN architecture was utilized, while the RNN was replaced by fully connected (FC) layers (Fig. 12). To analyze different sequence lengths and to aggregate the features extracted by the CNN into one vector, a global max pooling layer was employed after the CNN. This layer was subsequently connected to two FC layers instead of the GRUs. To match the total number of trainable parameters in both architectures, the number of units in each of FC layers was raised from 64 to 192. In total, the network had 341,191 parameters (vs. 340,295 parameters in the RCNN network). To allow a comparison with the proposed RCNN network, this network was trained, validated and tested using the same sets of training, validation and test images. The obtained results are listed in Table V (second row).
V-D Single vs. multi-task classification
Given that plaque and stenoses analyses are related, the classification could be posed as a single multi-class task with seven unique output classes (no plaque, calcified, mixed or non-calcified plaque with non-significant stenosis, calcified, mixed or non-calcified plaque with significant stenosis).
To evaluate the performance of single task classification, two additional networks were examined. The first network utilized an identical RCNN architecture with a single softmax classifier with 7 output units. The second used a CNN-only architecture (as in Section V-C) also with a single softmax classifier with 7 output units. To allow a comparison with the proposed network, these networks were trained, validated and tested using the same sets of training, validation and test images. The obtained results are listed in Table V (first and third rows).
|Plaque Analysis||Stenosis Analysis|
V-E Comparison with previous work
Most published methods reported segment-level sensitivity and positive predictive value (PPV) for the detection of the anatomically significant stenosis . Shahzad et al.  reported a segment-level sensitivity of 0.50 and a PPV of 0.27 for the detection of the anatomically significant stenosis, while Wang et al.  reported a sensitivity of 0.28 and a PPV of 0.23. The proposed network achieved a sensitivity of 0.61 and a PPV of 0.65 for detecting significant stenosis. Nonetheless, comparison with these methods is not trivial as these methods required the artery lumen to be first segmented and then the stenosis was detected. Additionally, different studies reported performance using a different evaluation procedure and different sets of patients than the current work. Therefore, a direct comparison of the results should be used only as an indication.
Moreover, most methods for the automated detection of the calcified coronary artery plaque perform quantification of the plaque . Such methods usually perform voxel-level analysis which our method does not offer. Methods for detection of non-calcified plaque [14, 15] perform manual or semi-automatic quantification that requires substantial manual interaction by experts. Therefore, a direct comparison with such methods is not feasible.
Vi Discussion and conclusion
A method for automatic detection and characterization of the coronary artery plaque type, as well as detection and characterization of the anatomical significance of the coronary artery stenosis was presented. The method employs an RCNN that analyzes an MPR view of a coronary artery extracted from a CCTA scan using the coronary artery centerline. The RCNN utilizes a 3D CNN that computes image features from 3D volumes extracted along the coronary artery centerline. Subsequently, an RNN analyzes the computed image features to perform both classification tasks. Unlike most previous methods that detect and characterize coronary artery plaque and stenosis relying on the coronary artery lumen segmentation , the proposed method requires only the coronary artery centerline as an input along with the CCTA image.
The presented results reveal that detection and characterization of coronary artery plaque can be performed accurately, but with moderate reliability (Table I and Table II). For both plaque detection and plaque characterization, the second observer only achieved slightly better performance than the proposed method (Table II) indicating the complexity of the task. Nevertheless, the method was accurate in discriminating segments with plaque from those without any plaque. This is clinically important as absence of plaque does not lead to treatment. Moreover, further analysis of plaque characterization revealed that differentiation of the mixed plaque from the calcified and non-calcified plaque remains challenging. This is not surprising given that the mixed plaque contains both calcified and non-calcified components and that the distinction between mixed and calcified plaque, as well as between mixed and non-calcified plaque, is not clearly defined as illustrated in Fig. 10. To address this, the automatic method could perform detection of calcified and non-calcified components only, and the obtained results could be merged into calcified, non-calcified and mixed plaque based on their spatial distribution. In addition, it would be interesting to segment plaque on a voxel-level, but obtaining voxel-wise reference is extremely labor intensive and requires a very experienced expert. Furthermore, analysis of the results per coronary artery (Table III) reveals that plaque characterization achieves a slightly lower performance in segments located in the RCA than in LAD or LCX. This might be caused by the more prominent cardiac motion artifacts in the RCA compared to the other two arteries .
Anatomically significant stenosis could potentially lead to myocardial ischemia (i.e. functionally significant stenosis), and clinical guidelines suggest that different grades of stenosis in the coronary artery should be managed differently . Our experiments demonstrate that the proposed method is able to detect and determine the anatomical significance of coronary artery stenosis accurately with excellent reliability (Table II and Table IV). For detection and characterization of the anatomical significance of a stenosis, the proposed method achieved a performance approaching the level of the second observer (Table II). Moreover, we investigated whether the method is able to identify patients with anatomically significant stenosis. This is especially important as patients having such stenosis are usually referred to further functional testing and to invasive coronary angiography to measure fractional flow reserve (FFR). FFR determines the functional significance of the stenoses, and hence, establishes the patient’s treatment strategy. Our results reveal that patients with anatomically significant stenosis on CCTA are detected with high accuracy (Table II). However, one patient without stenosis was identified as having a significant stenosis (Table IV). An examination of the CCTA image of this patient revealed that although no artifacts were present, a coronary artery bifurcation was mistakenly detected as significant stenosis in one of the patients’ arteries. Moreover, in this work, only two distinct degrees of stenosis were differentiated; below and above 50% of luminal narrowing. Future work may investigate automatic classification of additional clinically relevant stenotic grades, e.g. 25% or 70%, or automatic estimation of stenosis degree, i.e. percentage of luminal narrowing. Nevertheless, for both tasks, a larger training set of patients with manual annotations would be required.
The contribution of the recurrent nature of the proposed network was evaluated. The results show clear advantage of the proposed recurrent architecture over the network containing no recurrent units (Table V). This is in agreement with our assumption that a sequential analysis of several small volumes along the coronary artery is needed to aggregate the knowledge of the entire analyzed region rather than just locally (e.g. a single volume). A similar concept was reported by Ng et al. , where incorporating information across video frames enabled better video classification. Possibly, given a sufficiently large and diverse data set, a deeper CNN-only (e.g. 3D U-Net ), analyzing a large single volume along the artery, could be employed to perform the presented analyses. However, obtaining such a large data set remains highly challenging, and employing deep networks, typically used for analysis of natural images (e.g. ResNet ), is likely not beneficial due to scarcity of the manually labeled training data.
Unlike most methods for coronary artery plaque and stenosis classification that depend on the extracted artery centerline followed by arterial lumen segmentation , the proposed method relies only on the extracted artery centerline. Arterial lumen segmentation is far from trivial task, which occasionally requires substantial manual interaction, especially in diseased population with heavily calcified arteries. We have here prevented potential error propagation by omitting this step. To extract coronary artery centerlines, we have employed our previously designed method for artery centerline extraction . However, any other manual, semi-automatic or automatic method could be employed instead. Although employing the extracted centerlines and the subsequent analysis of the MPR images simplify plaque and stenosis classification, small errors in centerline extraction might lead to errors and therefore negatively affect the overall performance. To mitigate impact of such errors, we trained the RCNN with augmented centerlines, simulating small inaccuracies (Fig.6b). Future work might investigate direct classification of plaque and stenosis from acquired CCTA images, omitting the intermediate centerline extraction.
In this work, we have treated plaque and stenosis characterization as two different tasks, that were performed jointly. Although this halved the time of inference (1.8 seconds per artery on average), it has a limitation. Given that the parameters of the two softmax classifiers in the network differ, a physiologically impossible scenario, where a plaque is not detected while a stenosis is detected, can occur. Although in our experiments this was the case only in less than 1.5% of the cases, future work could address this either by modifying the network architecture preventing such scenario, or by applying a high penalty for such cases in the loss function of the network. A single task network would prevent such a scenario. Nevertheless, experiments, comparing the proposed multi-task approach against networks with single softmax classifier each, demonstrate superior performance of the multi-task approach (Table V, first and third rows vs. second and fourth rows). The limited number of training samples may have prevented the single-task networks from generalizing well even with a relatively small network. Moreover, although the architecture of the RCNN was determined in preliminary experiments using the training set, a systematic extensive grid-search or other hyper-parameter optimization methods were not performed. Addressing this might further improve the results.
In the manual annotations defining the reference standard, a single label was assigned to a whole segment of the coronary artery containing plaque. Separating these segments to their local components might lead to different labels. Consequently, the identified start- and end-points of the automatically detected plaque and stenosis are not in full agreement with the reference annotations (see Fig. 11 and Fig. 10). Furthermore, coronary artery bifurcations were not manually annotated and the network was not trained to detect these as a separate class. As a sudden reduction in the coronary artery lumen diameter occurs distal to bifurcations , a stenosis might be mistakenly detected (see example illustrated in Fig. 10). Future work might address these limitations by modifying the reference standard so that each voxel in the arterial wall or each cross-section of the arterial lumen is annotated and the coronary artery bifurcations are indicated. Besides bifurcations, imaging artefacts caused by routinely used step-and-shoot protocol could affect performance of the method. Nonetheless, a qualitative evaluation of the results revealed that arterial segments containing such image artifacts were not incorrectly detected as having plaque or stenosis. Note that these regions were not included in the manual annotations either. Finally, the current study employed clinically obtained CCTA scans from a single vendor. Modern scanners applying new techniques (e.g. high-pitch spiral, 320-detector row) may improve image quality and potentially enable further increase in the performance of the here proposed method. Moreover, further studies are needed to evaluate the proposed method using a larger set of scans from different vendors and medical centers.
To conclude, this study presented an algorithm, based on a recurrent convolutional neural network, for automatic detection and characterization of coronary artery plaque, as well as detection and characterization of the anatomical significance of coronary artery stenosis. To the best of our knowledge, we are the first to propose an automatic method for both plaque and stenosis characterizations. This may enable automated triage of patients to those without coronary plaque, and those with coronary plaque and stenosis in need for further cardiovascular investigation.
This study was financially supported by the project FSCAD, funded by the Netherlands Organization for Health Research and Development (ZonMw) in the framework of the research programme IMDI (Innovative Medical Devices Initiative); project 104003009.
-  D. Mozaffarian, E. J. Benjamin, A. S. Go, D. K. Arnett, M. J. Blaha, M. Cushman, S. R. Das, S. de Ferranti, J.-P. Després, H. J. Fullerton et al., “Heart disease and stroke statistics - 2016 update,” Circulation, vol. 133, no. 4, pp. e38–e360, 2016.
-  G. Raff, A. Abidov, S. Achenbach, D. Berman, L. Boxt, M. Budoff, V. Cheng, T. DeFrance, J. Hellinger, R. Karlsberg et al., “SCCT guidelines for the interpretation and reporting of coronary computed tomographic angiography,” Journal of Cardiovascular Computed Tomography, vol. 3, no. 2, pp. 122–136, 2009.
-  J. M. Wolterink, T. Leiner, B. D. De Vos, J.-L. Coatrieux, B. M. Kelm, S. Kondo, R. A. Salgado, R. Shahzad, H. Shu, M. Snoeren et al., “An evaluation of automatic coronary artery calcium scoring methods with cardiac CT using the orCaScore framework,” Medical Physics, vol. 43, no. 5, pp. 2361–2373, 2016.
-  R. Virmani, A. P. Burke, A. Farb, and F. D. Kolodgie, “Pathology of the vulnerable plaque,” Journal of the American College of Cardiology, vol. 47, no. 8 Supplement, pp. C13–C18, 2006.
-  S. Achenbach, “Can CT detect the vulnerable coronary plaque?” The International Journal of Cardiovascular Imaging (formerly Cardiac Imaging), vol. 24, no. 3, pp. 311–312, 2008.
-  A. Cassar, D. R. Holmes Jr, C. S. Rihal, and B. J. Gersh, “Chronic coronary artery disease: diagnosis and management,” in Mayo Clinic Proceedings, vol. 84, no. 12. Elsevier, 2009, pp. 1130–1146.
-  R. C. Cury, S. Abbara, S. Achenbach, A. Agatston, D. S. Berman, M. J. Budoff, K. E. Dill, J. E. Jacobs, C. D. Maroules, G. D. Rubin et al., “CAD-RADSTM coronary artery disease–reporting and data system. an expert consensus document of the society of cardiovascular computed tomography (SCCT), the american college of radiology (ACR) and the north american society for cardiovascular imaging (NASCI). endorsed by the american college of cardiology,” Journal of Cardiovascular Computed Tomography, vol. 10, no. 4, pp. 269–281, 2016.
-  M. J. Budoff, D. Dowe, J. G. Jollis, M. Gitter, J. Sutherland, E. Halamert, M. Scherer, R. Bellinger, A. Martin, R. Benton et al., “Diagnostic performance of 64-multidetector row coronary computed tomographic angiography for evaluation of coronary artery stenosis in individuals without known coronary artery disease: results from the prospective multicenter ACCURACY trial,” Journal of the American College of Cardiology, vol. 52, no. 21, pp. 1724–1732, 2008.
-  H. Kirişli, M. Schaap, C. Metz, A. Dharampal, W. B. Meijboom, S.-L. Papadopoulou, A. Dedic, K. Nieman, M. De Graaf, M. Meijs et al., “Standardized evaluation framework for evaluating coronary artery stenosis detection, stenosis quantification and lumen segmentation algorithms in computed tomography angiography,” Medical Image Analysis, vol. 17, no. 8, pp. 859–876, 2013.
-  A. Arbab-Zadeh and J. Hoe, “Quantification of coronary arterial stenoses by multidetector CT angiography in comparison with conventional angiography: methods, caveats, and implications,” JACC: Cardiovascular Imaging, vol. 4, no. 2, pp. 191–202, 2011.
-  J. M. Wolterink, T. Leiner, B. D. de Vos, R. W. van Hamersvelt, M. A. Viergever, and I. Išgum, “Automatic coronary artery calcium scoring in cardiac CT angiography using paired convolutional neural networks,” Medical Image Analysis, vol. 34, pp. 123–136, 2016.
-  N. Lessmann, B. van Ginneken, M. Zreik, P. A. de Jong, B. D. de Vos, M. A. Viergever, and I. Išgum, “Automatic calcium scoring in low-dose chest CT using deep neural networks with dilated convolutions,” IEEE Transactions on Medical Imaging, vol. 37, pp. 615–625, 2018.
-  R. Shadmi, V. Mazo, O. Bregman-Amitai, and E. Elnekave, “Fully-convolutional deep-learning based system for coronary calcium score prediction from non-contrast chest CT,” in IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 24–28.
-  T. Schepis, M. Marwan, T. Pflederer, M. Seltmann, D. Ropers, W. G. Daniel, and S. Achenbach, “Quantification of noncalcified coronary atherosclerotic plaques with dual source computed tomography: comparison to intravascular ultrasound,” Heart, pp. hrt–2009, 2009.
-  D. Dey, T. Schepis, M. Marwan, P. J. Slomka, D. S. Berman, and S. Achenbach, “Automated three-dimensional quantification of noncalcified coronary plaque from coronary CT angiography: comparison with intravascular US,” Radiology, vol. 257, no. 2, pp. 516–522, 2010.
-  S. Mittal, Y. Zheng, B. Georgescu, F. Vega-Higuera, S. K. Zhou, P. Meer, and D. Comaniciu, “Fast automatic detection of calcified coronary lesions in 3D cardiac CT images,” in International Workshop on Machine Learning in Medical Imaging. Springer, 2010, pp. 1–9.
-  M. A. Zuluaga, I. E. Magnin, M. H. Hoyos, E. J. D. Leyton, F. Lozano, and M. Orkisz, “Automatic detection of abnormal vascular cross-sections based on density level detection and support vector machines,” International Journal of Computer Assisted Radiology and Surgery, vol. 6, no. 2, pp. 163–174, 2011.
-  S. Sankaran, M. Schaap, S. C. Hunley, J. K. Min, C. A. Taylor, and L. Grady, “HALE: Healthy area of lumen estimation for vessel stenosis quantification,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2016, pp. 380–387.
-  E. J. Halpern and D. J. Halpern, “Diagnosis of coronary stenosis with CT angiography: comparison of automated computer diagnosis with expert readings,” Academic Radiology, vol. 18, no. 3, pp. 324–333, 2011.
-  Y. Xu, G. Liang, G. Hu, Y. Yang, J. Geng, and P. K. Saha, “Quantification of coronary arterial stenoses in CTA using fuzzy distance transform,” Computerized Medical Imaging and Graphics, vol. 36, no. 1, pp. 11–24, 2012.
-  R. Shahzad, T. van Walsum, H. Kirisli, H. Tang, C. Metz, M. Schaap, L. van Vliet, and W. Niessen, “Automatic stenoses detection, quantification and lumen segmentation of the coronary arteries using a two point centerline extraction scheme,” in MICCAI 2012 workshop proceedings, 2012.
-  C. Wang, R. Moreno, and Ö. Smedby, “Vessel segmentation using implicit model-guided level sets,” in MICCAI Workshop” 3D Cardiovascular Imaging: a MICCAI segmentation Challenge”, Nice France, 1st of October 2012., 2012.
-  A. Broersen, P. Kitslaar, M. Frenay, and J. Dijkstra, “Frenchcoast: fast, robust extraction for the nice challenge on coronary artery segmentation of the tree,” in Proc. of MICCAI Workshop” 3D Cardiovascular Imaging: a MICCAI Segmentation Challenge, 2012.
-  Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
-  J. M. Wolterink, R. W. van Hamersvelt, M. A. Viergever, T. Leiner, and I. Išgum, “Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier,” Medical image analysis, 2018.
-  J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in , 2015, pp. 2625–2634.
-  A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3128–3137.
-  J. Y.-H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, “Beyond short snippets: Deep networks for video classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2015, pp. 4694–4702.
-  M. Liang and X. Hu, “Recurrent convolutional neural network for object recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3367–3375.
-  W. De Mulder, S. Bethard, and M.-F. Moens, “A survey on the application of recurrent neural networks to statistical language modeling,” Computer Speech & Language, vol. 30, no. 1, pp. 61–98, 2015.
-  R. P. Poudel, P. Lamata, and G. Montana, “Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation,” in Reconstruction, Segmentation, and Analysis of Medical Images. Springer, 2016, pp. 83–94.
-  G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical Image Analysis, vol. 42, no. Supplement C, pp. 60 – 88, 2017.
-  S. Andermatt, S. Pezold, and P. Cattin, “Multi-dimensional gated recurrent units for the segmentation of biomedical 3D-data,” in Deep Learning and Data Labeling for Medical Applications. Springer, 2016, pp. 142–151.
-  W. Xue, G. Brahm, S. Pandey, S. Leung, and S. Li, “Full left ventricle quantification via deep multitask relationships learning,” Medical Image Analysis, vol. 43, pp. 54–65, 2018.
-  S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proceedings of The 32nd International Conference on Machine Learning, 2015, pp. 448–456.
-  K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014.
X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,”
International Conference on Artificial Intelligence and Statistics, 2011, pp. 315–323.
-  D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
-  N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
-  C. Hong, C. R. Becker, A. Huber, U. J. Schoepf, B. Ohnesorge, A. Knez, R. Bruning, and M. F. Reiser, “ECG-gated reconstructed multi–detector row CT coronary angiography: effect of varying trigger delay on image quality,” Radiology, vol. 220, no. 3, pp. 712–717, 2001.
-  Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2016, pp. 424–432.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition. corr, vol. abs/1512.03385,” 2015.
-  L. Antiga and D. A. Steinman, “Robust and objective decomposition and mapping of bifurcating vessels,” IEEE Transactions on Medical Imaging, vol. 23, no. 6, pp. 704–713, 2004.