Traumatic brain injury (TBI) is a major contributor to injury-related deaths. In emergency departments, head computed tomography (CT) scans are routinely performed on patients under evaluation for suspected TBI. Existing works have shown that a computer vision system that rapidly and reliably detects emergency TBI findings, such as acute intracranial bleeding, on head CT scans can significantly reduce the time to diagnosis and potentially reduce death and long-term disability[1, 7]
. Deep learning techniques have been successful recently in detecting intracranial hemorrhages, e.g. 3D classification[1, 7] supervised by text reports, 2D classification , instance segmentation . However, to our knowledge, no semantic segmentation approach has shown performance competitive with human experts.
We propose to solve the detection and segmentation problem jointly as a semantic segmentation task. Segmentation offers many advantages over classification, including better interpretability, and quantifiable metrics for disease prognosis [2, 4]. Unlike , we view hemorrhage as “stuff” (e.g. water) rather than “things” (e.g. car) due to its fluid nature. As the clinical need is to know whether a scan (i.e. whole head) is positive, and where the positive pixels are, semantic segmentation is the simplest way to achieve it.
Among existing pixel-wise labeling techniques, fully convolutional networks  (FCN) are successful and widely adopted for such tasks in computer vision  and the medical imaging community [11, 6]. Most computer vision practitioners use whole images as inputs for their FCNs following . This is in contrast to how patch-based FCN training has been successful in applications such as retinopathy , MRI , and X-ray/CT imaging [8, 10]. Despite the wide adoption, there exists no systematic study on why patches improve FCN in many cases.
We propose PatchFCN and show that it outperforms standard FCN in localizing hemorrhages. Since no public dataset is available, one important challenge we face is to acquire pixelwise labeled data. Unlike the approaches that learn from text reports [1, 7], we collect a dataset of scans annotated pixelwise for the presence of hemorrhage by expert radiologists. Using x smaller data, PatchFCN significantly outperforms weakly supervised methods [1, 7] on classification tasks. Compared to the state-of-the-art segmentation method , our segmentation and classification results are competitive while using x less training data and a simpler system.
We analyze the following factors to better understand the performance gains of PatchFCN: 1) batch diversity, 2) amount of context, and 3) sliding window inference. We find that PatchFCN outperforms FCN by finding an optimal trade-off between batch diversity and the amount of context. In addition, sliding window inference helps to bridge the gap of train/test time and consistently improve performance. We hope these findings would benefit other segmentation tasks where patch-based training is effective.
The goals for hemorrhage detection are to find out: 1) whether a stack contains hemorrhage, and 2) where the hemorrhage is within the stack. In practice this may be used by the radiologists/neurosurgeons to assess the risk level of the patient and triage the patient to immediate surgical evacuation, monitoring in the intensive care unit (ICU), or routine monitoring on the hospital ward. Inspired by existing works [6, 11, 8, 10, 3], we propose to solve both tasks with PatchFCN as follows (see Fig.2):
2.0.1 Patch-based Training:
We train an FCN on random small patches cropped from the whole images centered on foreground. The model learns to predict the binary pixel label within the patches. For head CT data, the intuition of patch-based training comes from how radiologists make decisions – the morphology of contrast region is often a crucial cue for deciding whether it represents pathologies. Similarly, PatchFCN causes the network to make its decision based on the local image information without relying on excessive context. In addition, small patches allow larger batch size and hence higher batch diversity to stabilize network training. As most convolutional networks have built-in batch normalization e.g., PatchFCN leverages it by finding a good trade-off between large minibatch and adequate context for the task.
2.0.2 Patch-based Inference:
At test time, we evaluate the images in a sliding window fashion, as opposed to the typical fully convolutional inference. Sliding window at test time avoids any domain shift which occurs when training on small patches and evaluating fully convolutionally on the whole image. This is because the paddings present in convolution layers make a patch in the context of a whole image not the same as the patch by itself. Let the input image be of sizeand the patch size , then the total number of windows is given by , where is an adjustable parameter for the window overlap. As multiple predictions are made for each pixel, we simply average their scores. The frame-level score is obtained by averaging the pixel scores within the frame. To get stack-level scores from pixel scores, we first take -norm over the frame to obtain a stack-frame score. The stack score is defined as the maximum stack-frame score within a stack. is treated as a hyper-parameter and tuned on the trainval set.
2.0.3 Data Collection:
Our dataset consists of clinical head CT scans performed over 7 years from 2010 to 2017 on 4 different 64-detector-row CT scanners (GE Healthcare, Siemens) at our affiliated hospitals. We use the word “stack” for each patient’s head CT scan, and the word “frame” for each individual slice of the stack. The scans were anonymized by removing patient-related meta-data, skull, scalp, and face. Board-certified senior neuroradiologists who specialize in TBI identified all areas of hemorrhage in our dataset. Our data contains the typical spectrum of technical limitations seen in clinical practice (e.g. motion artifact, “streak” artifact near the skull base or in the presence of metal), and also contains all of the subtypes of acute intracranial hemorrhage, including epidural hematoma, subdural hematoma, subarachnoid hemorrhage, hemorrhagic contusion, and intracerebral hemorrhage (see Fig. 3 for examples). We randomly split the data into a trainval/test set of / stacks for development and internal validation. The hyper-parameters of PatchFCN are tuned within the trainval set.
2.0.4 Implementation Details:
We choose a DRN-38 backbone because it performs competitively among many network designs . Regarding the inputs, we clip the dynamic range of raw data at and Hounsfield unit (HU), and then rescale the intensity to lie within . Image size is . In both training and test time, we use a patch size of unless stated otherwise. We utilize the z-axis context by fusing the adjacent frames with the center frame at the input (3 channels in total). The optimization is done by SGD with momentum following 
setup. We train the network from scratch without using ImageNet pretraining, as we do not observe any gains using ImageNet. We re-weight the positive class loss byto balance the dominant negative class loss. The learning rate starts at and decreases by a factor of after and of the complete training iterations. At test time, we select to ensure good overlap between adjacent sliding windows. To compute stack-level score, we select in the norm. All parameters were found by cross validation on the trainval set.
3.1 Stack-level Benchmark with Human Experts
The first order task of hemorrhage detection is to determine whether a stack contains hemorrhage. We conduct internal as well as external validation for PatchFCN on stack-level as shown in Figure 3. The human expert is a neuroradiologist certified by the American Board of Radiology with 15 years of attending experience. The expert is instructed to examine each scan with the same level of care as a clinical scan. We allow the expert to take as much as time as needed. The expert can modify their reads on scans before submitting final answers on the whole data set. The groundtruths are determined by at least one neuroradiologist with more than 10 years of neuroradiology attending experience.
3.1.1 Internal (Retrospective) Validation:
We report the ROC curve of PatchFCN on the test set and compare it with a human expert (15-year attending) in a retrospective setting where the test data was collected before the model development. Our single model AUC of is competitive against the state-of-the-art (single model)  and (ensemble models) , while using much less training data. Our human expert has very low false positive rate at recall, better than the of PatchFCN. Using both trainval and test data, our 4-fold cross validation AUC is .
3.1.2 External (Prospective) Validation:
We collected a prospective test set of scans after the model was developed. No further hyper-parameter adjustment was allowed in order to prevent overfitting to the test set. To minimize selection bias, we randomly select from all head CT scans performed from November to December 2018 using the Radiology Information System (RIS) SQL database in our hospital. The positive rate is , which approximates the observed positive rates in emergency departments of many U.S. hospitals. Our ensemble model () achieves an AUC of , which is competitive against the state-of-the-art  and . PatchFCN approaches but does not exceed the human expert. Our best operating point is .
3.2 Pixel-level Evaluation
Apart from stack-level evaluation, we evaluate PatchFCN at pixel level because clinicians also want to know the location and volume of the bleeds for disease prognosis. Figure 1 visualizes the outputs of PatchFCN in comparison with the groundtruths. Results are shown on randomly selected positive frames in the retrospective test set.
On the retrospective test set, our model achieves pixelwise Dice score, Jaccard index, and average precision of, , and . In comparison,  reports Dice scores of to for a few types of hemorrhages they study. Our groundtruths are annotated pixelwise by senior neuroradiologists who specialize in TBI and include many subtle findings that could be easily missed by inexperienced radiologists. Using both trainval and test data, our 4-fold cross validation Dice score is .
3.3 PatchFCN vs. FCN
Table 1 shows that PatchFCN consistently improves over standard FCN for pixel and frame by a healthy margin for a wide range of patch sizes. We report average precision (AP), Dice score and Jaccard index at pixel level with a threshold of 0.5. Note how PatchFCN is robust to patch size and maintains the performance even at a patch size of 80. We have tried even smaller sizes and observed a significant performance drop due to difficult optimization. To compare across different patch sizes, we choose the batch size to control the number of input pixels per batch to be the same, and we choose the number of epochs such that the number of gradient steps are the same. We also ensure that all performances are saturated and training longer does not improve further.
3.4 What Makes PatchFCN effective?
Given the effectiveness of PatchFCN, we want to delve deeper to understand what makes patches so effective. We identify a few differences from standard FCN and study them by control experiments. For the following experiments, we define the batch size , which is the product of , the number of images per batch, and , the number of patches per image. The batch size is defined this way because we sample patches from each of the image samples. PatchFCN has , , and , where is the crop size, whereas the standard FCN has . We perform these analyses on the test split because it is larger and yields more stable performance. In this section, we control the number of input pixels and number of iterations the same way as in Section 3.3, unless otherwise stated.
3.4.1 Batch Diversity:
One possible advantage of PatchFCN is that we can fit a larger batch size and thus include more diverse data within any given GPU memory. To study the contribution of batch diversity, we control the batch size and decrease the number of images we sample patches from. Since , this means we sample more patches per image. As decreases, we expect batch diversity to decrease as well. The default PatchFCN has and , which has the greatest diversity for any given
. By fixing the other hyperparameters, we can safely say the only difference here is the batch diversity. Note that we control the number of steps to be the same, so we decrease the number of epochs linearly with.
Table 2 shows that decreased batch diversity results in lower pixel and frame-level performance. The breaking point is at , where the performance drops significantly from . We speculate that this is due to the use of batch normalization in residual networks. This experiment demonstrates the importance of batch diversity for PatchFCN.
3.4.2 How Much Context Does PatchFCN Need?
A trade-off of using patches is that we restrict the amount of context available to the network during training. Intuitively, one would think that more context is better. However, with limited amount of data, it is possible that less context could serve as an effective regularizer by forcing the prediction to rely on local information. To understand how much we lose/gain by having less context, we compare PatchFCN using different patch sizes while fixing the batch size and the number of steps (number of input pixels not the same here).
Table 3 shows that the improvement of context plateaus at patch size . Compared to , is significantly better. However, increasing the patch size beyond 240 does not offer any more gain. We speculate that the improvement comes from the context regularization of patches, which helps in case of limited data. Overall, controlling context with patches is effective and allows the use of a larger and more diverse batch as in Table 2.
To qualitatively study what cues PatchFCN uses, we backpropagate the gradients from each hemorrhage region to the image space (see Fig.4). The gradient responses primarily come from the pixels not confidently predicted and correspond to the cues used for hemorrhage prediction. Fig. 4 shows that FCN captures long range dependencies that can easily overfit to limited data, while PatchFCN focuses on the local morphology and may generalize better.
3.4.3 Patch-based Sliding Window Inference:
At inference time, standard FCN applies on the whole image at once . We hypothesize that this is sub-optimal for PatchFCN because the model is only trained on patches but has to take whole images at test time. That is why the default PatchFCN adopts sliding window inference to minimize the domain shift by letting PatchFCN evaluate patch by patch at test time. In Table 4, we show that sliding window inference consistently improves over fully convolutional inference for all patch sizes. Note that the gap is largest for the smallest crop size of , and decreases as patch size increases.
|80||144||3600||69.4 (-6.1)||53.1 (-7.6)||74.9 (-3.6)||85.5 (-2.3)|
|120||64||1600||75.0 (-0.9)||60.0 (-1.2)||75.6 (-2.5)||88.7 (-0.6)|
|240||16||400||75.9 (-0.7)||61.2 (-0.8)||76.4 (-2.1)||89.8 (-0.1)|
We propose PatchFCN – a simple yet effective framework for intracranial hemorrhage detection. PatchFCN approaches the performance of an expert neuroradiologist as well as performs competitively with the state-of-the-art at stack level. In addition, it localizes many subtypes of hemorrhages well and has strong pixel level performance. Analyses show that PatchFCN outperforms FCN by finding a good trade-off between batch diversity and the amount of context. Our work shows the capability of PatchFCN for intracranial hemorrhage detection and potentially for other medical segmentation tasks.
-  (2018) Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. npj Digital Medicine 1 (1), pp. 9. Cited by: §1, §1.
Hybrid 3d/2d convolutional neural network for hemorrhage evaluation on head ct. American Journal of Neuroradiology 39 (9), pp. 1609–1616. Cited by: §1, §1, §1, §3.1.1, §3.1.2, §3.2.
-  (2018) Harmonizing diffusion mri data across magnetic field strengths. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 116–124. Cited by: §2.
-  (2019) An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nature Biomedical Engineering 3 (3), pp. 173. Cited by: §1, §1, §3.1.1, §3.1.2.
-  (2015) Fully convolutional networks for semantic segmentation. In CVPR, Cited by: §1, §3.4.3.
-  (2018) Autofocus layer for semantic segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 603–611. Cited by: §1, §2.
-  (2018) Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 24 (9), pp. 1337–1341. Cited by: §1, §1.
-  (2017) A multi-atlas approach to region of interest detection for medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 168–176. Cited by: §1, §2.
-  (2017) Dilated residual networks. In CVPR, Cited by: §2.0.1, §2.0.4, §3.4.1.
-  (2018) Convolutional neural network based metal artifact reduction in x-ray computed tomography. IEEE transactions on medical imaging 37 (6), pp. 1370–1381. Cited by: §1, §2.
-  (2018) Deep supervision with additional labels for retinal vessel segmentation task. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 83–91. Cited by: §1, §2.