Large scale public chest X-ray datasets usually have had only the posteroanterior (PA) view available (e.g. the ChestX-ray14 (Wang et al., 2017)
dataset). This has allowed the development of several convolutional neural networks (CNN) based approaches that are built to use only the PA view for automatic prediction(Rajpurkar et al., 2017; Yao et al., 2017; Rajpurkar et al., 2018; Li et al., 2018; Cohen et al., 2019). Other views, such as the lateral (L) view, are not commonly acquired as they are difficult to read without specific training (Feigin, 2010) and only considered useful for specific diagnoses. The lateral view, in particular, is now usually replaced by a CT scan which is only ordered if the PA view is insufficient to diagnose. This practice delays prognosis as the patient would typically need to schedule another appointment. It also increases risk of exposure to larger doses of radiation used in performing CT scans.
There are specific cases in which the lateral view provides information for diagnosis that isn’t clear or visible on the PA view (Shiraishi et al., 2007; Feigin, 2010; Ittyachen et al., 2017). For example, up to 15% of the lung can be obscured by cardiovascular structures and the diaphragm (Raoof et al., 2012). It is unclear if the information is completely missing from the PA view, or if it is present but in a way that makes it too difficult for a human to read. In the first case, the lateral view becomes relevant for making a prediction but a sufficiently advanced model could make it redundant in the second.
This question was previously challenging to answer due to the lack of public large scale datasets of paired PA and L views. But the previous year has seen the release of sizeable de-identified chest X-ray datasets containing multiple views, namely CheXpert Irvin et al. (2019) & MIMIC-CXR Johnson et al. (2019) from the United States and PadChest Bustos et al. (2019) from Spain. This provides us with an opportunity to explore the usefulness of the lateral view. In this work, we analyse the efficacy of multi-view models on the PadChest dataset. Preliminary work by Bertrand et al. (2019) suggests there is value in predicting from a lateral image (with a single view model) for certain radiological labels. We choose PadChest for our analysis due to the variety of radiological labels: it has a total of 194 distinct labels compared to 14 in the other two. This enables a much more fine-grained analysis of how the lateral view can contribute for chest X-ray prediction. We investigate the following questions —
Is there a benefit of also including the lateral view in a prediction model? If so, in what cases specifically?
Is there a trade-off between training on PA views of a large amount of patients and training on paired PA & L views of a smaller amount of patients?
The structure of the paper is as follows. We provide a brief overview of multi-view CNNs in radiology in Section 2 and a description of PadChest and the preprocessing we used in Section 3. We present the models we used in Section 4. The experiments and results are shown in Section 5, and finally we address these questions with our findings in Section 6.
2 Related work
Using multiple views can help in increasing detection performance in radiology. Setio et al. (2016) achieved an increase of around 54% and 70% in baseline metrics when the number of views was increased to 3 and 9 respectively from 1 in a pulmonary nodule detection task. Shachor et al. (2019) found that their ‘Mixture of Views’ models achieved higher metrics than single view methods for breast cancer classification and brain MRI segmentation tasks. Multi-view networks also increased performance in tasks like mammogram classification (Geras et al., 2017; Carneiro et al., 2017; Trent Kyono et al., 2019; Nasir Khan et al., 2019), emphysema classification (Bermejo-Peláez et al., 2018), lung segmentation (El-Regaily et al., 2019), fracture detection (Kitamura et al., 2019) and lesion detection (Li et al., 2019).
These works provide a trend of a gain in performance when multiple views are used, which motivates our work. The use of the lateral view in deep learning on chest X-rays has been limited. Rajkomar et al. (2017) tried to predict whether an X-ray was a frontal view or lateral on a dataset from California. Rubin et al. (2018) is one of the initial works that assessed whether combining the frontal and lateral views would help the network. They found that combining the PA and L views led to an increase of 3% in the average AUC over all labels and improved the performance for 12 of the 14 labels in MIMIC-CXR.
There has not been any assessment of multi-view models on the PadChest dataset, to the best of our knowledge. While Yao et al. (2019) performed a preliminary benchmark on a simple binary classification (normal/abnormal), it used only the PA view. Bertrand et al. (2019) observed that the lateral views did contain useful information for some prediction tasks on PadChest which is the primary motivation for our analysis on this dataset.
3 Data and preprocessing
We use the PadChest (Bustos et al., 2019) dataset in our analysis. It is comprised of 160,000 chest X-rays and reports gathered from a Spanish hospital spanning over 67,000 patients with multiple visits and views available. The images have been annotated with various types of radiological findings and differential diagnoses.
We extract the set of patients who have a paired PA and lateral view available forming a total of 30,699 patients. We use only the first study from every patient and discard any additional ones. We resize the images to pixels, utilizing a center crop if the aspect ratio is uneven, and scale the pixel values to for the training. Each visit can have any number of labels from the total of 194. Since the PadChest dataset defines a hierarchy of labels, we mapped the labels to their respective top level one, in order to maximize the number of images for each label. From those top level labels, we retain only those that occur in at least a 50 patients. Some of them are of low clinical interest, such as “electrical device”, however they provide a sanity check on the results of the models.
All models are built with four dense blocks following the same configuration as DenseNet-121 Huang et al. (2017); Rajpurkar et al. (2017). Along with one type of single view model, we test four different methods of combining both views.
DenseNet: This is a standard DenseNet-121 with its fully connected layer modified for a multi-label output. It is used as a single view model and serves as the baseline. We use two DenseNets, each trained and tested on the PA and L views respectively.
Stacked: The architecture is essentially the same as a standard DenseNet-121 but the lateral view is stacked as a second channel of the PA image. It is the simplest way to combine the views before giving them to a model. This model is not designed to work if only one view is available.
DualNet (Rubin et al., 2018): The PA and L views are processed by two separate CNNs composed of four DenseNet blocks each. Their output feature maps are then passed through a global average pooling layer, concatenated and given to a fully connected layer that maps them to the output labels. Similar to Stacked, this model also relies on both views to be available to make a prediction.
HeMIS (Havaei et al., 2016)
: The HeMIS architecture involves propagating each input view/modality through its separate set of layers, combining them by calculating pixelwise statistics and propagating further for classification. We create a HeMIS-style architecture where the PA and L views are processed separately by two separate CNNs ‘branches’ composed of the first three DenseNet blocks each. The mean and variance of the output feature maps of both CNNs are computed pixelwise which are then concatenated and given to another dense block. The output of this block is passed through a global average pooling layer and given to a fully connected layer that maps them to the output classes. This model works even if only one view is available, with variance of the feature maps substituted with a tensor of zeroes of the same shape.
: We modify DualNet by adding two separate fully connected layers to the PA and L branches respectively that map it to the same output labels and train all three losses jointly. Other than concatenating the pooled vectors of the branches, we also add an option to average them. Inspired byLee et al. (2015)
(we classify the output of each branch rather than layer) andTrent Kyono et al. (2019), this could be considered as a DualNet regularized by auxiliary view-specific losses. This model is designed to be robust to missing views; if a view is not present the prediction is calculated off the available view’s fully connected layer.
We train all models for 40 epochs with Adam and a batch size of 8. Other hyperparameters differ for the models. In addition, we use curriculum learning with HeMIS and AuxLoss denoted by the ‘-CL’ suffix hereon. A description of our training details including hyperparameters and how we implement curriculum learning is given in AppendixB.
We evaluate the multi-view models using the AUC achieved in three configurations of the test set. The first is the general case where paired views are available for all patients. The other two are the cases when only a single view is available: either only the PA view is present or the L view. For models that require both views to be available (Stacked and DualNet), we use a zero matrix in place of the missing view. With this testing regime, we want to see how the multi-view models perform with only one view relative to the respective single view model. We perform five runs of every model with a different random split every time and report the results, averaged over all runs, in tab:results.
It is apparent that there is very little difference between the models and most of the models’ confidence intervals overlap when both views are available. We perform t-tests between pairs of all models and find that only two pairs have differences that are statistically significant. AuxLoss-CL achieves a significantly better performance than DualNet () and Stacked () but not the rest. A very interesting observation is that the Stacked model performs as well as the rest, despite the lack of a spatial correspondence between the channels of the input.
The multi-view models diverge in performance when given only a single view. In the case of testing on only PA images of patients in the test set, DualNet emerges as a clear winner. Its confidence interval seems very close to that of Stacked, AuxLoss and AuxLoss-CL but a t-test proves the difference is statistically significant (, , respectively). The AUC of the single view DenseNet-PA lies between DualNet and Stacked and we find that the two multi-view models are not significantly better or worse respectively. Curriculum learning also seems to not have a discernible effect, in fact HeMIS with curriculum learning is much worse than a vanilla HeMIS. There is a large change in the performance when evaluating on only lateral images. While the AuxLoss models experience a 5-7% decrease in AUC relative to paired images, the other models deteriorate by more than 20%. The single view model is better than the multi-view in this case.
Test AUC achieved by the different models, averaged over five runs with standard deviation also reported. DenseNet is trained on a single view denoted by the suffix, rest are trained on both views. The symbol in the superscript indicates the difference between a pair of models in the same column is statistically significant.
5.2 How does the lateral view help in prediction?
As evidenced in Section 5.1, there is a major difference between the multi-view models when it comes to the view(s) they have to use for making a prediction. We wanted to examine how the AUC is affected when the proportion of patients having a paired L view in the test set is varied. We do this in fig:lprop by iteratively removing the lateral view for 1% of the patients and testing the models, incrementing the % removed until 100%.
The most conspicuous observation is that the DualNet does not seem to use the lateral view for prediction at all. HeMIS is also unusual where adding paired views actually hurts performance until the proportion crosses a threshold. This could be caused by the usage of variance in combining the views: when HeMIS is tested on only PA images, the variance of the feature maps was substituted with a zero tensor causing the model to believe there was zero variance. As we add patients with paired lateral views, smaller proportions of paired views led to high variance causing the dip in the curve in fig:lprop. Past a certain proportion, the variance of the feature maps started approximating the population variance which led to increasing performance.
The performance of the multi-view models in the three testing regimes (tab:results) suggests that the AuxLoss model combined with curriculum learning uses the lateral view most productively. We use AuxLoss-CL to find the effect of the lateral view on individual labels in the dataset: we perform a comparison between the label-wise AUC of AuxLoss-CL and DenseNet-PA averaged over all runs. If the difference between the average AUC for a given label is lesser than the standard deviation of the better model, we annotate that label as indifferent to the model used. We find that 32 of the 64 labels see an improvement in AUC when both the views are used jointly, visualized in fig:betterlateral
5.3 Benefits of increasing the training set size
We form a second dataset by adding the remaining patients that have a PA view available; we call this the extended dataset. This adds 27,576 PA images from new patients. We then train the DenseNet-PA and AuxLoss-CL models on this extended dataset, and evaluate them on the main test set. Those results are reported in Table 2.
The DenseNet-PA model gains one percent of AUC from doubling the dataset size. This marginal increase in performance from a major increase in dataset size is consistent with what is observed in other computer vision tasks(Sun et al., 2017). We examine how the AUC of individual labels changes between the DenseNet-PA trained on the main and extended datasets and find that, similar to AuxLoss-CL, 32 labels see an increase with 22 of them overlapping with the AuxLoss-CL. The top three increases are in the labels ‘azygos lobe’ (), ‘tracheal shift’ () and ‘diaphragmatic eventration’ (). A plot detailing all the improvements in the style of fig:betterlateral is included in Appendix A.
On the other hand AuxLoss-CL becomes worse at using both views and is indistinguishable in performance from the extended DenseNet-PA. This is likely be due to the dominant presence of PA images, and careful over-sampling of joint and lateral images during the curriculum learning might fix that problem.
|Train Data (AUC on test reported)|
|DenseNet-PA||PA||0.793 0.007||0.813 0.005|
|AuxLoss-CL||PA||0.788 0.005||0.812 0.006|
|AuxLoss-CL||Both||0.809 0.003||0.772 0.018|
To return to the questions posed in the introduction, it appears a well-tuned PA-only model is competitive with a well-tuned joint model. However, for the specific labels shown in Figure 3, the lateral view allows a statistically significant improvement over using just the PA view. We proposed a joint model using auxiliary tasks that gives strong results with any subset of views.
While all multi-view models offer similar performance when given both views, they do not perform as well when given only one view. Testing these models on different proportions of paired L images brings very intriguing observations to surface. First, it indicates that models other than AuxLoss were relying too heavily on the PA view to make a prediction, especially DualNet which completely ignores the L view. Second, it might explain how the Stacked model performs well on the first two testing regimes: it just learns to use less information from the second channel of the input image to favour predicting from the first channel only. Third, adding curriculum learning to AuxLoss mitigates this reliance on the PA view but does not have much effect on HeMIS.
For the second question, we found that doubling the number of PA images in the training set only gives a marginal increase in performance. A joint model trained on only half the data gives the same performance. If it is more costly to bring in a new patient than to acquire another view for an already present patient, then our results suggest acquiring both views for a smaller number of patients rather than one view on twice that number.
We also find that although the different approaches achieve similar performance with both views, the training is less sensitive to hyperparameters for the AuxLoss and DualNet models shown in Figure 4. With this result, we would conclude that these models would be easier to train as the range of optimal hyperparameters would be relatively wider.
This study has two limitations. First, while we have shown that the lateral view is useful for some labels, we did not sort those by clinical relevance. This can be seen notably by the presence of “electrical device” or “catheter”. It might be that most of those labels are extremely rare, or typically diagnosed through other means than a chest x-ray. Second, this study is based entirely on the PadChest dataset. The population differs from other available datasets, at least by ethnicity, but likely also by clinical practice. In other words, we expect both covariate shift and concept shift (Moreno-Torres et al., 2012). This implies that while the lateral view will still be useful for many labels, those labels might not be the same as the ones we found here.
We thank Paul Bertin for his code and Mathieu Germain for discussions. We thank AcademicTorrents.com for making data available for our research. This work is partially funded by a grant from the Institut de valorisation des donnees (IVADO). This work utilized the supercomputing facilities managed by Mila, NSERC, Compute Canada, and Calcul Quebec. We also thank NVIDIA for donating a DGX-1 computer used in this work.
- Bermejo-Peláez et al. (2018) D Bermejo-Peláez, R S J Estepar, and M J Ledesma-Carbayo. Emphysema classification using a multi-view convolutional network. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). ieeexplore.ieee.org, 2018. doi: 10.1109/ISBI.2018.8363629.
- Bertrand et al. (2019) Hadrien Bertrand, Mohammad Hashir, and Joseph Paul Cohen. Do Lateral Views Help Automated Chest X-ray Predictions? In Medical Imaging with Deep Learning Abstract, 2019.
- Bustos et al. (2019) Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. PadChest: A large chest x-ray image dataset with multi-label annotated reports, 2019.
- Carneiro et al. (2017) Gustavo Carneiro, Jacinto Nascimento, and Andrew P Bradley. Automated Analysis of Unregistered Multi-View Mammograms With Deep Learning. IEEE transactions on medical imaging, 2017. doi: 10.1109/TMI.2017.2751523.
- Cohen et al. (2019) Joseph Paul Cohen, Paul Bertin, and Vincent Frappier. Chester: A Web Delivered Locally Computed Chest X-Ray Disease Prediction System, 2019.
- El-Regaily et al. (2019) Salsabil Amin El-Regaily, Mohammed Abdel Megeed Salem, Mohamed Hassan Abdel Aziz, and Mohamed Ismail Roushdy. Multi-view Convolutional Neural Network for lung nodule false positive reduction. Expert systems with applications, 2019. doi: 10.1016/j.eswa.2019.113017.
- Feigin (2010) David S Feigin. Lateral Chest Radiograph. A Systematic Approach. Academic Radiology, 2010. doi: 10.1016/j.acra.2010.07.004.
- Geras et al. (2017) Krzysztof J. Geras, Stacey Wolfson, Yiqiu Shen, Nan Wu, S. Gene Kim, Eric Kim, Laura Heacock, Ujas Parikh, Linda Moy, and Kyunghyun Cho. High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks, 2017.
- Havaei et al. (2016) Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, and Yoshua Bengio. HeMIS: Hetero-modal image segmentation. In Medical Image Computing and Computer Assisted Intervention, volume 9901 LNCS, 2016. doi: 10.1007/978-3-319-46723-8–_˝54.
Huang et al. (2017)
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.
Densely Connected Convolutional Networks.
Computer Vision and Pattern Recognition, 2017.
- Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, and Others. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Thirty-Third AAAI Conference on Artificial Intelligence. aaai.org, 2019.
- Ittyachen et al. (2017) Abraham M. Ittyachen, Anuroopa Vijayan, and Megha Isac. The forgotten view: Chest x-ray - lateral view. Respiratory Medicine Case Reports, 2017. doi: https://doi.org/10.1016/j.rmcr.2017.09.009.
- Johnson et al. (2019) AEW Johnson, TJ Pollard, S Berkowitz, NR Greenbaum, MP Lungren, C-Y Deng, RG Mark, and S Horng. Mimic-cxr: A large publicly available database of labeled chest radiographs, 2019.
- Kitamura et al. (2019) Gene Kitamura, Chul Y Chung, and Barry E Moore, 2nd. Ankle Fracture Detection Utilizing a Convolutional Neural Network Ensemble Implemented with a Small Sample, De Novo Training, and Multiview Incorporation. Journal of digital imaging, 2019. doi: 10.1007/s10278-018-0167-7.
- Lee et al. (2015) Chen Yu Lee, Saining Xie, Patrick W. Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In International Conference on Artificial Intelligence and Statistics (AISTATS). Microtome Publishing, 2015.
- Li et al. (2018) Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and Li Fei-Fei. Thoracic disease identification and localization with limited supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
- Li et al. (2019) Zihao Li, Shu Zhang, Junge Zhang, Kaiqi Huang, Yizhou Wang, and Yizhou Yu. MVP-Net: Multi-view FPN with Position-aware Attention for Deep Universal Lesion Detection, 2019.
- Moreno-Torres et al. (2012) Jose G. Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 2012. doi: 10.1016/j.patcog.2011.06.019.
- Nasir Khan et al. (2019) H Nasir Khan, A R Shahid, B Raza, A H Dar, and H Alquhayz. Multi-View Feature Fusion Based Four Views Model for Mammogram Classification Using Convolutional Neural Network. IEEE Access, 2019. doi: 10.1109/ACCESS.2019.2953318.
- Rajkomar et al. (2017) Alvin Rajkomar, Sneha Lingam, Andrew G Taylor, Michael Blum, and John Mongan. High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks. Journal of digital imaging, 2017. doi: 10.1007/s10278-016-9914-9.
- Rajpurkar et al. (2017) Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P Lungren, and Andrew Y Ng. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning, 2017.
- Rajpurkar et al. (2018) Pranav Rajpurkar, Jeremy Irvin, Robyn L. Ball, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P. Langlotz, Bhavik N. Patel, Kristen W. Yeom, Katie Shpanskaya, Francis G. Blankenberg, Jayne Seekins, Timothy J. Amrhein, David A. Mong, Safwan S. Halabi, Evan J. Zucker, Andrew Y. Ng, and Matthew P. Lungren. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLOS Medicine, 2018. doi: 10.1371/journal.pmed.1002686.
- Raoof et al. (2012) Suhail Raoof, David Feigin, Arthur Sung, Sabiha Raoof, Lavanya Irugulpati, and Edward C Rosenow III. Interpretation of plain chest roentgenogram. Chest, 2012. doi: 10.1378/chest.10-1302.
- Rubin et al. (2018) Jonathan Rubin, Deepan Sanghavi, Claire Zhao, Kathy Lee, Ashequl Qadir, and Minnan Xu-Wilson. Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks, 2018.
- Setio et al. (2016) Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Geert Litjens, Paul Gerke, Colin Jacobs, Sarah J van Riel, Mathilde Marie Winkler Wille, Matiullah Naqibullah, Clara I Sanchez, and Bram van Ginneken. Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks. IEEE transactions on medical imaging, 2016. doi: 10.1109/TMI.2016.2536809.
- Shachor et al. (2019) Yaniv Shachor, Hayit Greenspan, and Jacob Goldberger. A mixture of views network with applications to multi-view medical imaging. Neurocomputing, 2019. doi: 10.1016/j.neucom.2019.09.027.
- Shiraishi et al. (2007) Junji Shiraishi, Feng Li, and Kunio Doi. Computer-aided diagnosis for improved detection of lung nodules by use of posterior-anterior and lateral chest radiographs. Academic radiology, 2007. doi: 10.1016/j.acra.2006.09.057.
- Sun et al. (2017) Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Harikrishna Mulam. Revisiting unreasonable effectiveness of data in deep learning era, 2017. doi: 10.1109/ICCV.2017.97.
- Trent Kyono et al. (2019) Trent Kyono, Fiona J. Gilbert, and Mihaela van der Schaar. Multi-view Multi-task Learning for Improving Autonomous Mammogram Diagnosis. In Machine Learning for Healthcare, 2019.
- Wang et al. (2017) Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. In Computer Vision and Pattern Recognition, 2017. doi: 10.1109/CVPR.2017.369.
- Yao et al. (2017) Li Yao, Eric Poblenz, Dmitry Dagunts, Ben Covington, Devon Bernard, and Kevin Lyman. Learning to diagnose from scratch by exploiting dependencies among labels. arXiv preprint arXiv:1710.10501, 2017.
- Yao et al. (2019) Li Yao, Jordan Prosky, Ben Covington, and Kevin Lyman. A Strong Baseline for Domain Adaptation and Generalization in Medical Imaging, 2019.
Appendix A Improvement in labels with extended dataset and paired views
|DenseNet-PA (Extended)||AuxLoss-CL (Main)|
|air fluid level||0.051||0.021|
|costophrenic angle blunting||0.026||0.035|
|artificial heart valve||0.024||0.034|
Appendix B Hyperparameters
b.1 Stability of training
For each model, we performed a random search over the learning rates, the amount of dropout, and for CL models the probability of dropping a view for each batch. 40 combinations of those hyperparameters were tried per model.
In Figure 4, we show the distributions of the AUC on the validation set over the combinations, per model. We observe strong variations in the shape of those distributions. HeMIS, for example, is very sensitive to the choice of hyperparameters, whereas AuxLoss is much more concentrated.
b.2 Training details
For all models, we use the Adam optimizer and a batch size of 8 and train up to 40 epochs. We also compute class weights (clamped at 5) to balance the loss as the labels are highly imbalanced. All models use early stopping based on the AUC achieved on the validation set.
The initial learning rate differs for every model but it is decayed by a factor of 10 halfway through training for all. These model-specific hyperparameters were found through an extensive random search and are given in tab:hyperparams.
We use dropout with a different probability for each model, given in tab:hyperparams
We utilize a 60-20-20 split in the dataset for train, validation and test sets. We also use data augmentation and
Loss weights for AuxLoss
AuxLoss uses a weighted sum of the three losses (averaged over all labels) with weights of 1.0, 0.3 and 0.3 for the joint, PA and L losses.
On HeMIS, we randomly drop one of the views with probability 0.25 each. For AuxLoss, we randomly select one of the PA or L loss with probability of 0.2 for each and use that to update that view’s branch instead of the entire model. The weighted sum of the three losses is used to update the entire model only 60% of the time. This is done to make the model rely less on having both views available all the time.