Deep Neural Networks (DNN) have shown promising results in various medical applications, but highly depend on the amount and the diversity of training data. In the context of medical imaging, this is particularly challenging since the required training data may not be available in a single institution due to the low incidence rate of some pathologies and limited numbers of patients. At the same time, it is often infeasible to collect and share patient data in a centralised data lake due to medical data privacy regulations.
One recent method that tackles this problem is Federated Learning (FL) [6, 8]: it allows collaborative and decentralised training of DNNs without sharing the patient data. Each node trains its own local model and, periodically, submits it to a parameter server. The server accumulates and aggregates the individual contributions to yield a global model, which is then shared with all nodes. It should be noted that the training data remains private to each node and is never shared during the learning process. Only the model’s trainable weights or updates are shared, thus keeping patient data private. Consequently, FL succinctly sidesteps many of the data security challenges by leaving the data where they are and enables multi-institutional collaboration.
Although FL can provide a high level of security in terms of privacy, it is still vulnerable to misuse such as reconstructions of the training examples by model inversion. One effective countermeasure is to inject noise to each node’s training process, distort the updates and limit the granularity of information shared among them [1, 9]
. However, existing privacy-preserving research only focuses on general machine learning benchmarks such as MNIST, and uses vanilla stochastic gradient descent algorithms.
In this work, we implement and evaluate practical federated learning systems for brain tumour segmentation. Throughout a series of experiments on the BraTS 2018 data, we demonstrate the feasibility of privacy-preserving techniques. Our primary contributions are: (1) implement and evaluate, to the best of our knowledge, the first privacy-preserving federated learning system for medical image analysis; (2) compare and contrast various aspects of federated averaging algorithms for handling momentum-based optimisation and imbalanced training nodes; (3) empirically study the sparse vector technique for a strong differential privacy guarantee.
We study FL systems based on a client-server architecture (illustrated in Fig. 1 (left)) implementing the federated averaging algorithm . In this configuration, a centralised server maintains a global DNN model and coordinates clients’ local stochastic gradient descent (SGD) updates. This section presents the client-side model training procedure, the server-side model aggregation procedure, and the privacy-preserving module deployed on the client-side.
2.1 Client-side model training
We assume each federated client has a fixed local dataset and reasonable computational resources to run mini-batch SGD updates. The clients also share the same DNN structure and loss functions. The proposed training procedure is listed in Algorithm1. At federated round , the local model is initialised by reading global model parameters from the server, and is updated to by running multiple iterations of SGD. After a fixed number of iterations , the model difference is shared with the aggregation server.
DNNs for medical image are often trained with a momentum-based SGD. Introducing the momentum terms takes the previous SGD steps into account when computing the current one. It can help accelerate the training and reduce oscillation. We explore the choices of design for handling these terms in FL. In the proposed Algorithm 1 (exemplified with Adam optimiser ), we re-initialise each client’s momentums at the beginning of each federated round (denoted as m. restart). Since local model parameters are initialised from the global ones, which aggregated information from other clients, the restarting operation effectively clears the clients’ local states that could interfere the training process. This is empirically compared with (a) clients maintaining a set of local momentum variables without sharing; denoted as baseline m. (b) treating the momentum variables as a part of the model, i.e., the variables are updated locally and aggregated by the server (denoted as m. aggregation). Although m. aggregation is theoretically plausible , it requires the momentums to be released to the server. This increases both communication overheads and data security risks.
2.2 Client-side privacy-preserving module
The client-side is designed to have full control over which data to share and local training data never leave the client’s site. Still, model inversion attacks such as  can potentially extract sensitive patient data from the update or the model during federated training. We adopt a selective parameter update  and the sparse vector technique (SVT)  to provide strong protection against indirect data leakage.
Selective parameter sharing The full model at the end of a client-side training process might have over-fitted and memorised local training examples. Sharing this model poses risks of revealing the training data. Selective parameter sharing methods limit the amount of information that a client shares. This is achieved by (1) only uploading a fraction of : component of will be shared iif is greater than a threshold ; (2) further replacing by clipping the values to a fixed range . Here denotes the absolute value of ; is chosen by computing the percentile of ;
is independent of specific training data and can be chosen via a small publicly available validation set before training. Gradient clipping is also applied, which is a widely-used method, acting as a model regulariser to prevent over-fitting.
Differential privacy module The selective parameter sharing can be further improved by having a strong differential privacy guarantee using SVT. The procedure of selecting and sharing distorted components of is described in Algorithm 2. Intuitively, instead of simply thresholding and sharing its components , every sharing is controlled by the Laplacian mechanism. This is implemented by first comparing a clipped and noisy version of with a noisy threshold (Line 8, Algorithm 2), and then only sharing a noisy answer , if the thresholding condition is satisfied. Here
denotes a random variable sampled from the Laplace distribution parameterised by; denotes clipping of to be in the range of ; denotes the sensitivity of the federated gradient which is bounded by in this case . The selection procedure is repeated until fraction of is released. This procedure satisfies -differential privacy .
2.3 Server-side model aggregation
The server distributes a global model and receives synchronised updates from all clients at each federated round (Algorithm 3). Different clients may have different numbers of local iterations at round , thus the contributions from the clients could be SGD updates at different training speeds. It is important to require an from the clients, and weight the contributions when aggregating them (Line 8, Algorithm 3). In the case of partial model sharing, utilising the sparse property of to reduce the communication overheads is left for future work.
This section describes the experimental setup, including the common hyper-parameters used for each FL system.
Data preparation The BraTS 2018 dataset  contains multi-parametric pre-operative MRI scans of 285 subjects with brain tumours. Each subject was scanned with four modalities, i.e. (1) T1-weighted, (2) T1-weighted with contrast enhancement, (3) T2-weighted, and (4) T2 fluid-attenuated inversion recovery (T2-FLAIR). Each subject was associated with voxel-level annotations of “whole tumour”, “tumour core”, and “enhancing tumour”. For details of the imaging and annotation protocols, we refer the readers to Bakas et al. . The dataset was previously used for benchmarking machine learning algorithms and is publicly available. We use it to evaluate the FL algorithms on the multi-modal and multi-class segmentation task. For the client-side local training, we adapted the state-of-the-art training pipeline originally designed for data-centralised training  and implemented as a part of the NVIDIA Clara Train SDK111https://devblogs.nvidia.com/annotate-adapt-model-medical-imaging-clara-train-sdk/.
To test the generalisation ability across the subjects, we randomly split the dataset into a model training set ( subjects) and a held-out test set ( subjects). The scans were collected from thirteen institutions with different equipment and imaging protocols, and thus heterogeneous image feature distributions. To make our federated setup realistic, we further stratified the training set into thirteen disjoint subsets, according to where the image data were originated and assigned each to a federated client. The setup is challenging for FL algorithms, because (1) each client only processes data from a single institution, which potentially suffers from more severe domain-shift and over-fitting issues compared with a data-centralised training; (2) it reflects the highly imbalanced nature of the dataset (shown in Fig. 1).
Federated model setup The evaluation of the FL procedures is perpendicular to the choice of convolutional network architectures. Without loss of generality, we chose the segmentation backbone of 
as the underlying federated model and used the same set of local training hyperparameters for all experiments: the input image window size of the network wasvoxels, and spatial dropout ratio of the first convolutional layer was . Similarly to , we minimised a soft Dice loss using Adam  with a learning rate of , batch size of , of , of , and weight decay coefficient of . For all federated training, we set the number of federated rounds to
with two local epochs per federated round. A local epoch is defined as every client “sees” its local training examples exactly once. At the beginning of each epoch, data were shuffled locally for each client. For a comparison of model convergences, we also train a data-centralised baseline forepochs.
In terms of computational costs, the segmentation model has about parameters; a training iteration with an NVIDIA Tesla V100 GPU took .
Evaluation metrics We measure the segmentation performance of the models on the held-out test set using mean-class Dice score averaged over the three types of tumour regions and all testing subjects. For the FL systems, we report the performance of the global model shared among the federated clients.
Privacy-preserving setup The selective parameter updates module has two system parameters: fraction of the model and the gradient clipping value . We report model performance by varying both. For differential privacy, we fixed to , the sensitivity to , and to according to . The model performance by varying , , and are reported in the next section.
Federated vs. data-centralised training
The FL systems are compared with the data-centralised training in Fig. 2 (left). The proposed FL procedure can achieve a comparable segmentation performance without sharing clients’ data. In terms of training time, the data-centralised model converged at about training epochs, FL training at about . In our experiments, an epoch of data-centralised training () with an NVIDIA Tesla V100 GPU takes per epoch. The FL training time was determined by the slowest client (), which takes plus small overheads for client-server communication.
Momentum restarting and weighted averaging Fig. 2 (left) also compares variants of the FL procedure. For the treatment of momentum variables, restarting them at each federated round outperforms all the other variants. This suggests (1) each client maintaining an independent set of momentum variables slows down the convergence of the federated model; (2) averaging the momentum variables across clients improved the convergence speed over baseline m., but still gave a worse global model than the data-centralised model. On the server-side, weighted averaging of the model parameters outperforms the simple model averaging (i.e. ). This suggests that the weighted version can handle imbalanced numbers of iterations across the clients.
Partial model sharing Fig. 2 (right) compares partial model sharing by varying the fraction of the model to share and the gradient clipping values. The figure suggests that sharing larger proportions of models can achieve better performance. Partial model sharing does not affect the model convergence speed and the performance decrease can be almost negligible when only 40% of the full model is shared among the clients. Clipping of the gradient can, sometimes, improve the model performance. However, the value needs to be carefully tuned.
Differential privacy module The model performances by varying differential privacy (DP) parameters are shown in Fig. 3. As expected, there is a trade-off between DP protection and model performance. Sharing 10% model showed better performance than sharing 40% under the same DP setup. This is due to the fact that the overall privacy costs are jointly defined by the amount of noise added and the number of parameters shared during training. By fixing the per-parameter DP costs, sharing fewer variables has less overall DP costs and thus better model performance.
We propose a federated learning system for brain tumour segmentation. We studied various practical aspects of the federated model sharing with an emphasis on preserving patient data privacy. While a strong differential privacy guarantee is provided, the privacy cost allocation is conservative. In the future, we will explore differentially private SGD (e.g. ) for medical image analysis tasks.
We thank Rong Ou at NVIDIA for the helpful discussions.
The research was supported by the Wellcome/EPSRC Centre for Medical Engineering (WT203148/Z/16/Z), the Wellcome Flagship Programme (WT213038/Z/18/Z), the UKRI funded London Medical Imaging and AI centre for Value-based Healthcare, and the NIHR Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
-  (2016) Deep Learning with Differential Privacy. SIGSAC Conference on Computer and Communications Security, pp. 308–318. Cited by: §1, §5.
-  (2018) Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629. Cited by: §3.
-  (2017) Deep models under the GAN: information leakage from collaborative deep learning. In SIGSAC Conference on Computer and Communications Security, pp. 603–618. Cited by: §2.2.
-  (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §2.1, §3.
-  (2017) Understanding the sparse vector technique for differential privacy. Proceedings of the VLDB Endowment 10 (6), pp. 637–648. Cited by: §2.2, §2.2, §3.
-  (2017) Communication efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273–1282. Cited by: §1, §2.
3D MRI brain tumor segmentation using autoencoder regularization. In MICCAI Brainlesion Workshop, pp. 311–320. Cited by: §3, §3.
-  (2018) Multi-institutional deep learning modeling without sharing patient data: a feasibility study on brain tumor segmentation. In MICCAI Brainlesion Workshop, pp. 92–104. Cited by: §1.
-  (2015) Privacy-Preserving Deep Learning. In SIGSAC Conference on Computer and Communications Security, pp. 1310–1321. Cited by: §1, §2.2, §2.2.
-  (2017) Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, Cited by: §1.
-  (2019) On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In ICML, Cited by: §2.1.