Predicting the binding between viral peptides and human proteins from the adaptive immune system using machine learning may serve as a valuable tool to increase the speed of vaccine development in the ongoing SARS-CoV-2-pandemic as well as in future health crises. Accelerated vaccine development supported by computational biology tools may become especially relevant against the background of an evolutionary arms race between viral escape variants and vaccine adaptation until herd immunity can finally be reached.
Major histocompatibility complex (MHC) molecules encoded by the human leukocyte antigen (HLA) gene complex, play a crucial role in the adaptive immune system (Klein and Sato, 2000; Wieczorek et al., 2017). They induce an immune response by presenting antigen fragments on the cell-surface to immune effector cells (Wieczorek et al., 2017; Vyas et al., 2008) , and therefore take part in gaining acquired immunity through vaccination. E.g., novel RNA-based vaccines against SARS-CoV-2 enter human cells and elicit the expression of viral spike proteins. They are broken down by the proteasome into antigen peptides which bind to MHC proteins with varying binding affinity. Bound antigen peptides (protein-derived epitopes) are presented by MHC on the cell surface and tie to T-cells that trigger an immune response leading to acquired immunity (Sahin et al., 2014).
MHC is highly polymorphic such that humans express individual combinations of MHC alleles that bind differently tight to a given peptide, which can affect the potency of an evoked immune response (Winchester, 2008). Moreover, there are different MHC classes. MHC class I molecules are found on almost every nucleated body cell and on platelets at varying densities. They continuously present fragments of proteins produced in the cell – self or non-self antigens (e.g., viruses) – to CD8 T cells (Groothuis et al., 2005; Shastri et al., 2005). MHC class II occurs mainly in professional antigen presenting cells of the immune system (e.g., B-lymphocytes) where they present fragments of extracellular ingested pathogens to CD4 T cells (Vyas et al., 2008).
At present, several (e.g., mRNA-based) COVID-19-vaccines make use of the amino acid sequence of the SARS-CoV-2 spike protein, which constitutes about 1/8 of the viral proteome (Prachar et al., 2020). Viral escape variants of the spike protein that would degrade into peptides that bind less tight to MHC can be expected to become more prevalent due to evolutionary pressure as a result of widespread vaccine campaigns (Weisblum et al., 2020). In this case, it might be necessary to leverage selected parts of the remaining 7/8 of the viral proteome for novel vaccine candidates (Prachar et al., 2020; Grifoni et al., 2020). Identifying and increasing the number of immunodominant B- and T-cell epitopes (while excluding those that may even cause adverse effects) is a potential strategy in vaccination design to generate protective immunogenicity (Dong et al., 2020). Multi-epitope vaccines against SARS-CoV-2 might be able to achieve a more precise immune response and to limit the risk of allergic reactions (see Kar et al., 2020). While full experimental characterization of all potential peptides of several virus variants is slow or might not be feasible at all, prioritization by MHC-peptide binding stability prediction may substantially accelerate the development of a more effective vaccine (Prachar et al., 2020; Grifoni et al., 2020). This approach may also enable the creation of epitope vaccines targeted against several virus strains at the same time.
A wide range of binding affinity prediction methods based on machine learning has been developed with potential application to vaccine development as well as to personalized cancer immunotherapy. These methods are summarized in a recent comparative review (Zhao and Sher, 2018); see also Prachar et al. (2020)
for a comparison with particular focus on SARS-CoV-2. At this point, it is worth stressing that many of the established methods rely on complicated training procedures with intricate model selection procedures and/or rely on heuristics to identify, e.g., binding regions.
In this work, we evaluate the performance of a novel algorithm for peptide-MHC binding affinity/stability prediction on a recently released dataset with binding stability measurements between SARS-CoV-2 peptides and ten alleles of MHC class I and one allele of MHC class II (Prachar et al., 2020). The algorithm is based on recurrent neural networks and was recently proposed as USMPep (Vielhaben et al., 2020). The publication of the dataset by Prachar et al. (2020) also contains a benchmark comparison of about twenty state-of-the-art prediction algorithms (published before 2 March 2020) on these new binding stability measurements. With this contribution, we provide an update for this benchmark by adding the results of an extended version of USMPep (which was published on 2 July 2020, i.e. after the ‘reporting date’ of Prachar et al. (2020)).
2 Materials and Methods
Datasets & Targets
Objective of our work is to predict the binding stability between SARS-CoV-2 peptides and MHC based on the amino acid sequences of the peptides. For this purpose, we train and finally evaluate recurrent neural networks on three different types of lab measurements, involving a peptide of known amino acid sequence and a given MHC allele. Therefore, we distinguish three qualitatively different kinds of targets: During training, we encounter binding affinity (BA) for peptides and mass-spectrometry-eluted (MS) ligands. Whereas the former represents a continuous target (leaving aside qualitative binding affinity labels as provided by O'Donnell et al. (2018)), the latter only yield positive (i.e. binding) samples, which are typically combined with artificial negative samples in order to be able to train a classifier on this data. Finally, during test time, we aim to predict binding stability (BS), which is also a continuous target, but quantifies the stability of the binding and is hence a more specific measure than binding affinity (Harndahl et al., 2012; Jørgensen et al., 2014). Due to a lack of appropriate training data, we use BA as a proxy target for BS.
We use a MHC class I BA dataset provided by O'Donnell et al. (2018) and a MS dataset compiled by Jurtz et al. (2017) for model training. For MHC class II alleles, we train our models on a BA dataset from Jensen et al. (2018) and a MS dataset from Reynisson et al. (2020). All datasets are based on data retrieved from the Immune Epitope Database (Vita et al., 2018). MS datasets additionally include artificial decoys. We evaluate our tools on the aforementioned BS dataset provided by Prachar et al. (2020), where the stability measurements are normalized to an allele-specific reference peptide.
We consider the most predominantly used metrics in the field (Zhao and Sher, 2018; Prachar et al., 2020), namely Spearman’s and the area under the receiver operating curve (AUCROC) upon framing the task as a classification task using a threshold value of stability. In order to compare the overall performance, we follow Vielhaben et al. (2020) and consider summary metrics, in this case the median due to the small number of alleles under consideration, across alleles.
We build our approach on USMPep, a recently proposed, conceptually simple yet very powerful method (Vielhaben et al., 2020)
, which is based on a recurrent neural network, in this case with a single-layer long short-term memory (LSTM) architecture. We focus on single-allele models, and precondition the model weights based on an (up to the classification head) identical architecture pretrained with an autoregressive language model objective(Vielhaben et al., 2020), which generally lead to slight but consistent improvements compared to training from scratch. We consider ensembles of ten individual models for improved stability.
The quantitative and qualitative subsets of the available data let us consider two training objectives: On the log-transformed BA data, we train a regression model (USMPep_BA), as in Vielhaben et al. (2020)
using a modified mean squared error loss function that allows to include also qualitative BA measurements(O'Donnell et al., 2018). To leverage the additional, complementary data available through qualitative MS measurements, we train separate classification models using the epitopes identified via MS as well as the artificial negative samples provided in the original MS data using a crossentropy loss. Finally, we consider combined BA+MS ensembles (USMPep_BAMS) by averaging log-transformed BA and MS predictions, for the first time in the MHC binding prediction literature, to the best of our knowledge. The source code for training and evaluating our models is available at https://github.com/nstrodt/USMPep.
Figure 1 and Table 1 compare the performance of USMPep to other state-of-the-art methods. We show the performance on single alleles based on AUCROC in Figure 1. Both USMPep-variants improve the current state-of-the-art for allele A*01:01, the allele with the overall best performance. For B*40:01, USMPep_BA raises the current state-of-the-art to a new level. USMPep_BA is the only tool in the benchmark that is trained on BA data and achieves the highest AUCROC on more than one allele. While USMPep is one of the few tools that provide predictions for the only MHC class II allele in the test set (DRB1*04:01), its performance on this allele is weaker in comparison to the few other available tools.
Turning to the overall predictive performance in terms of median AUCROC and Spearman’s as shown in Table 1, USMPep_BAMS turns out to be the overall best-performing method among all tools in terms of Spearman’s and show the fourth best performance in terms of AUCROC. In particular, the ensembling of five regressors trained on BA data and five classifiers trained on MS data considerably improves the overall performance compared to an ensemble of ten regressors on BA-data alone (USMPep_BA). It is noteworthy that the USMPep_BAMS-ensemble profits from the diversity of the different predictors, e.g. a regressor trained on both BA and MS data along the lines of (O'Donnell et al., 2018) in an ensemble with a classifier trained on MS data yields a weaker performance (Spearman’s 0.5260 and AUCROC 0.8097). In summary, these results establish both the original USMPep_BA but in particular the newly proposed USMPep_BAMS as strong predictors for MHC binding stability compared to other state-of-the-art tools.
between predicted binding probability and actual BS across alleles. Besides, the median AUCROC across alleles was evaluated. The results of the state-of-the-art-methods were extracted from Figure 2 ofPrachar et al. (2020). Because numerous tools do not provide predictions for alleles C*01:02, C*07:01 and DRB1*04:01, these were excluded for the median of Spearman’s and AUCROC. AUCROC was only evaluated on alleles with more than ten stable binders, which further excludes two remaining HLA-C alleles. The five highest scores are marked in bold for both metrics and are underlined for the best-performing methods.
4 Summary and Discussion
We evaluate a novel MHC binding prediction tool on recently published BS measurements involving SARS-CoV-2 peptides. The USMPep algorithm is characterized by a conceptually simple architecture and training procedure, can process peptides of arbitrary length and does not rely on further heuristics. In order to exploit more training data, we adapt and extend the algorithm to consider not only quantitative BA, but also qualitative MS measurements. We find a very high overall performance of USMPep in comparison to other state-of the-art methods, and USMPep even outperforms all existing methods on selected single alleles. The method can potentially extend the computational immunology toolkit, and help to accelerate vaccine development, and to prevent future epidemics.
Several limits of the work should be considered. Training a model (on BA and MS measurements as proxy) in order to predict the binding of a given peptide to a certain MHC allele can only serve as first step. It neither necessarily implies BS (as pointed out by Prachar et al., 2020) nor immunogenicity, nor efficacy, nor safety of a potential (e.g., RNA-based) epitope vaccine derived from the amino acid sequence of the peptide. Moreover, additional BS measurements covering a wider range of MHC alleles appear necessary to realise the full potential of this and other prediction tools; in particular in order to warrant that the global population can profit to the same degree in a fair manner, since MHC allele expression may vary with sex and ethnicity (Schneider-Hohendorf et al., 2018; Quiñones-Parra et al., 2014).
- A systematic review of SARS-CoV-2 vaccine candidates. Signal transduction and targeted therapy 5 (1), pp. 1–14. External Links: Cited by: §1.
- A sequence homology and bioinformatic approach can predict candidate targets for immune responses to SARS-CoV-2. Cell host & microbe 27 (4), pp. 671–680. External Links: Cited by: §1.
- MHC class I alleles and their exploration of the antigen-processing machinery. Immunological Reviews 207 (1), pp. 60–76. External Links: Cited by: §1.
- Peptide-MHC class I stability is a better predictor than peptide affinity of CTL immunogenicity. European journal of immunology 42 (6), pp. 1405–1416. External Links: Cited by: §2.
- Improved methods for predicting peptide binding affinity to MHC class II molecules. Immunology 154 (3), pp. 394–406. External Links: Cited by: §2.
- NetMHCstab–predicting stability of peptide–MHC-I complexes; impacts for cytotoxic T lymphocyte epitope discovery. Immunology 141 (1), pp. 18–26. External Links: Cited by: §2.
- NetMHCpan-4.0: improved peptide-MHC class I interaction predictions integrating eluted ligand and peptide binding affinity data. Journal of immunology (Baltimore, Md. : 1950) 199 (9), pp. 3360–3368. External Links: Cited by: §2.
- A candidate multi-epitope vaccine against SARS-CoV-2. Scientific reports 10 (1), pp. 1–24. External Links: Cited by: §1.
- The HLA system. New England Journal of Medicine 343 (10), pp. 702–709. External Links: Cited by: §1.
MHCflurry: open-source class I MHC binding affinity prediction. Cell Systems 7 (1), pp. 129–132.e4. External Links: Cited by: §2, §2, §2, §3.
- Identification and validation of 174 COVID-19 vaccine candidate epitopes reveals low performance of common epitope prediction tools. Scientific Reports 10 (1). External Links: Cited by: §1, §1, §1, §2, §2, Figure 1, Table 1, §4.
- Preexisting CD8+ T-cell immunity to the H7N9 influenza A virus varies across ethnicities. Proceedings of the National Academy of Sciences 111 (3), pp. 1049–1054. External Links: Cited by: §4.
- NetMHCpan-4.1 and NetMHCIIpan-4.0: improved predictions of MHC antigen presentation by concurrent motif deconvolution and integration of MS MHC eluted ligand data. Nucleic Acids Research 48 (W1), pp. W449–W454. External Links: Cited by: §2.
- mRNA-based therapeutics — developing a new class of drugs. Nature Reviews Drug Discovery 13 (10), pp. 759–780. External Links: Cited by: §1.
- Sex bias in MHC I-associated shaping of the adaptive immune system. Proceedings of the National Academy of Sciences 115 (9), pp. 2168–2173. External Links: Cited by: §4.
- All the peptides that fit: the beginning, the middle, and the end of the MHC class I antigen-processing pathway. Immunological Reviews 207 (1), pp. 31–41. External Links: Cited by: §1.
- USMPep: universal sequence models for major histocompatibility complex binding affinity prediction. BMC Bioinformatics 21 (1). External Links: Cited by: §1, §2, §2, §2.
- The Immune Epitope Database (IEDB): 2018 update. Nucleic Acids Research 47 (D1), pp. D339–D343. External Links: Cited by: §2.
- The known unknowns of antigen processing and presentation. Nature Reviews Immunology 8 (8), pp. 607–618. External Links: Cited by: §1, §1.
- Escape from neutralizing antibodies by SARS-CoV-2 spike protein variants. eLife 9, pp. e61312. External Links: Cited by: §1.
- Major histocompatibility complex (MHC) class I and MHC class II proteins: conformational plasticity in antigen presentation. Frontiers in Immunology 8, pp. 292. External Links: Cited by: §1.
- 5 - the major histocompatibility complex. In Clinical Immunology, R. R. Rich, T. A. Fleisher, W. T. Shearer, H. W. Schroeder, A. J. Frew, and C. M. Weyand (Eds.), pp. 79–90. External Links: Cited by: §1.
- Systematically benchmarking peptide-MHC binding predictors: from synthetic to naturally processed epitopes. PLOS Computational Biology 14 (11), pp. e1006457. External Links: Cited by: §1, §2.