1 Introduction
Deep neural networks have been successfully applied to many domains. However, these blackbox models are generally difficult to analyze and their behavior is not guaranteed. Moreover, it has been shown that the predictions of deep networks become unreliable and unstable when tested in unseen situations, e.g., in the presence of small adversarial perturbations to the input (szegedy2013intriguing; goodfellow2014explaining; lin2019art). Therefore, neural network verification has become an important tool for analyzing and understanding the behavior of neural networks, with applications in safetycritical applications (katz2017reluplex; julian2019verifying; lin2019art), model explanation (shih2018symbolic) and robustness analysis (tjeng2017evaluating; wang2018formal; gehr2018ai2; wong2018provable; singh2018fast; weng2018towards; zhang2018efficient).
Formally, a neural network verification algorithm aims to provably characterize the prediction of a network within some input space. For example, given a way classification model , where stands for the predicted score of class
, we can verify some linear specification (defined by a vector
) as below:(1) 
where is a predefined input space. In the robustness verification problem, is defined as some small ball around the original example , and setting up
enables us to verify whether the logit output of class
is always greater than another class for any input within . This is a nonconvex optimization problem which makes computing the exact solution challenging, and thus several algorithms are recently proposed to find the lower bounds of Eq. (1) in order to efficiently obtain a safety guarantee (gehr2018ai2; weng2018towards; zhang2018efficient; singh2019abstract). Moreover, extensions of these algorithms can be used for verifying properties beyond robustness, such as rotation or shift invariant (singh2019abstract), conservation of energy (qin2019verification) and model correctness (yang2019correctness).However, most of existing verification methods focus on relatively simple neural network architectures, such as feedforward and recurrent neural networks, while they cannot handle complex structures. In this paper, we develop the first robustness verification algorithm for Transformers
(vaswani2017attention)with selfattention layers. Transformers have been widely used in natural language processing
(devlin2018bert; yang2019xlnet; liu2019roberta) and many other domains (parmar2018image; kang2018self; li2019visualbert; su2019vl; li2019semi). For frames under perturbation in the input sequence, we aim to compute a lower bound such that when these frames are perturbed within balls centered at the original frames respectively and with a radius of , the model prediction is certified to be unchanged. To compute such bounds efficiently, we adopt the linearrelaxation framework (weng2018towards; zhang2018efficient)– we recursively propagate and compute linear lower and upper bounds for each neuron w.r.t the input within the perturbation space
.We resolve several particular challenges in verifying Transformers. First
, Transformers with selfattention layers have a complicated architecture. Unlike simpler networks, they cannot be written as multiple layers of affine transformations or elementwise activation functions. Therefore, we need to propagate linear bounds differently for selfattention layers.
Second, dot products, softmax, and weighted summation in selfattention layers involve multiplication or division of two variables both under perturbation, namely crossnonlinearity, which is not present in feedforward networks. ko2019popqorn proposed a gradient descent based approach to find linear bounds, however it is inefficient and poses a computational challenge for Transformer verification since selfattention is the core of Transformers. In contrast, we derive closedform linear bounds that can be computed in complexity. Third, in the computation of selfattention, output neurons in each position depend on all input neurons from different positions (namely crossposition dependency), unlike the case in recurrent neural networks where outputs depend on only the hidden features from the previous position and the current input. Previous works (zhang2018efficient; weng2018towards; ko2019popqorn) have to track all such dependency and thus is costly in time and memory. To tackle this, we introduce an efficient bound propagating process in a forward manner specially for selfattention layers, enabling the tighter backward bounding process for other layers to utilize bounds computed by the forward process. In this way, we avoid crossposition dependency in the backward process which is relatively slower but produces tighter bounds. Combined with the forward process, the complexity of the backward process is reduced by for input length , while the computed bounds remain comparably tight. Our contributions are summarized below:
We propose an effective and efficient algorithm for verifying the robustness of Transformers with selfattention layers. To our best knowledge, this is the first method for verifying Transformers.

We resolve key challenges in verifying Transformers, including crossnonlinearity and crossposition dependency. Our bounds are significantly tighter than those by adapting Interval Bound Propagation (IBP) (mirman2018differentiable; gowal2018effectiveness).

We quantitatively and qualitatively show that the certified bounds computed by our algorithm consistently reflect the importance of input words in sentiment analysis, which justifies that these bounds are meaningful in practice and they shed light on interpreting Transformers.
2 Related Work
Robustness Verification for Neural Networks.
Given an input and a small region , the goal of robustness verification is to verify whether the prediction of the neural network is unchanged within this region. This problem can be mathematically formulated as Eq. (1). If Eq. (1) can be solved optimally, then we can derive the minimum adversarial perturbation of by conducting binary search on . Equivalently, we obtain the maximum such that any perturbation within cannot change the predicted label.
Several works focus on solving Eq. (1
) exactly and optimally, using mixed integer linear programming (MILP)
(tjeng2017evaluating; dutta2018output), branch and bound (BaB) (bunel2018unified), and satisfiability modulo theory (SMT) (ehlers2017formal; katz2017reluplex). Unfortunately, due to the nonconvexity of model , solving Eq. (1) is NPhard even for a simple ReLU network
(katz2017reluplex). Therefore, we can only expect to compute a lower bound of Eq. (1) efficiently by using relaxations. Many algorithms can be seen as using convex relaxations for nonlinear activation functions (salman2019convex), including using duality (wong2018provable; dvijotham2018dual), abstract domains (gehr2018ai2; singh2018fast; mirman2018differentiable; singh2019abstract), layerbylayer reachability analysis (wang2018efficient; weng2018towards; zhang2018efficient; gowal2018effectiveness) and semidefinite relaxations (raghunathan2018semidefinite; dvijothamefficient2019). Additionally, robustness verification can rely on analysis on local Lipschitz constants (hein2017formal; zhang2019recurjac). However, existing methods are mostly limited to verifying networks with relatively simple architectures, such as feedforward networks and RNNs (wang2018verification; akintunde2019verification; ko2019popqorn), while none of them are able to handle Transformers.Transformers and SelfAttentive Models.
Transformers (vaswani2017attention) based on selfattention mechanism, further with pretraining on largescale corpora, such as BERT (devlin2018bert), XLNet (yang2019xlnet), RoBERTa (liu2019roberta)
, achieved stateoftheart performance on many NLP tasks. Selfattentive models are also useful beyond NLP, including VisualBERT on vision and language applications
(li2019visualbert; su2019vl), image transformer for image generation (parmar2018image), acoustic models for speech recognition, sequential recommendation (kang2018self) and graph embedding (li2019semi).The robustness of NLP models has been studied, especially many methods have been proposed to generate adversarial examples (papernot2016crafting; jia2017adversarial; zhao2017generating; alzantot2018generating; cheng2018seq2sick; ebrahimi2018hotflip; shi2019adversarial). In particular, hsieh2019robustness showed that Transformers are more robust than LSTMs. However, there is not much work on robustness verification for NLP models. ko2019popqorn verified RNN/LSTM. jia2019certified; huang2019achieving used Interval Bound Propagation (IBP) for certified robustness training of CNN and LSTM. In this paper, we propose the first verification method for Transformers.
3 Methodology
We aim to verify the robustness of a Transformer whose input is a sequence of frames . We take binary text classification as a running example, where is a word embedding and the model outputs a score for each class (). Nevertheless, our method for verifying Transformers is general and can also be applied in other applications.
For a clean input sequence
correctly classified by the model, let
be the set of perturbed positions, where is the number of perturbed positions. Thus the perturbed input belongs to . Assuming that is the gold class, the goal of robustness verification is to computeIf , the output score of the correct class is always larger than the incorrect one for any input within . As mentioned previously, computing the exact values of is NPhard, and thus our goal is to efficiently compute a lower bound .
3.1 Base Framework
We obtain by computing the bounds of each neuron when is perturbed within (
can be regarded as a final neuron). A Transformer layer can be decomposed into a number of sublayers, where each sublayer contains neurons after some operations. These operations can be categorized into three categories: 1) linear transformations, 2) unary nonlinear functions, and 3) operations in selfattention. Each sublayer contains
positions in the sequence and each position contains a group of neurons. We assume that the Transformer we verify has sublayers in total, and the value of the th neuron at the th position in the th sublayer is , where is a vector for the specified sublayer and position. Specially, taking . We aim to compute a global lower bound and a global upper bound of for .We compute bounds from the first sublayer to the last sublayer. For neurons in the th layer, we aim to represent their bounds as linear functions of neurons in a previous layer, the th layer:
(2) 
where and are parameters of linear lower and upper bounds respectively. Using linear bounds enables us to efficiently compute bounds with a reasonable tightness. We initially have and . Thereby the righthandside of Eq. (2) equals to when . Generally, we use a backward process to propagate the bounds to previous sublayers, by substituting with linear functions of previous neurons. It can be recursively conducted until the input layer . Since is constant, we can regard the bounds as linear functions of the perturbed embeddings , and take the global bounds for :
(3) 
(4) 
where with . These steps resemble to CROWN (zhang2018efficient) which is proposed to verify feedforward networks. We further support verifying selfattentive Transformers which are more complex than feedforward networks. Moreover, unlike CROWN that conducts a fully backward process, we combine the backward process with a forward process (see Sec. 3.3) to reduce the computational complexity of verifying Transformers.
3.2 Linear Transformations and Unary Nonlinear Functions
Linear transformations and unary nonlinear functions are basic operations in neural networks. We show how bounds Eq. (2) at the th sublayer are propagated to the th layer.
Linear Transformations
If the th sublayer is connected with the th sublayer with a linear transformation where are parameters of the linear transformation, we propagate the bounds to the th layer by substituting :
where “” means that the equations hold for both lower bounds and upper bounds respectively.
Unary Nonlinear Functions
If the th layer is obtained from the th layer with an unary nonlinear function , to propagate linear bounds over the nonlinear function, we first bound with two linear functions of :
where are parameters such that the inequation holds true for all within its bounds computed previously. Such linear relaxations can be done for different functions, respectively. We provide detailed bounds for functions involved in Transformers in Appendix B.
We then back propagate the bounds:
where and mean to retain positive and negative elements in vector respectively and set other elements to 0.
3.3 SelfAttention Mechanism
Selfattention layers are the most challenging parts for verifying Transformers. We assume that is the input to a selfattention layer. We describe our method for computing bounds for one attention head, and bounds for different heads of the multihead attention in Transformers can be easily concatenated. is first linearly projected to queries , keys , and values with different linear projections, and their bounds can be obtained as described in Sec. 3.2. We also keep their linear bounds that are linear functions of the perturbed embeddings. For convenience, let , where indicates vector concatenation, and thereby we represent the linear bounds as linear functions of :
where and mean that the inequation holds true for queries, keys and values respectively. We then bound the output of the selfattention layer starting from , , .
Bounds of Multiplications and Divisions
We bound multiplications and divisions in the selfattention mechanism with linear functions. We aim to bound bivariate function or with two linear functions and , where are bounds of obtained previously. For , we derive optimal parameters: , , , , . We provide a proof in Appendix C. However, directly bounding is tricky; fortunately, we can bound it indirectly by first bounding a unary function and then bounding the multiplication .
A Forward Process
For the selfattention mechanism, instead of using the backward process like CROWN (zhang2018efficient), we compute bounds with a forward process which we will show later that it can reduce the computational complexity. Attention scores are computed from and : where is the dimension of and . For each multiplication , it is bounded by:
We then obtain the bounds of :
In this way, linear bounds of and are forward propagated to
. Attention scores are normalized into attention probabilities with a softmax, i.e.
, where is a normalized attention probability. is an unary nonlinear function and can be bounded by . So we forward propagate bounds of to bound with , where:By summing up bounds of each , linear bounds can be further propagated to . With bounds of and ready, we forward propagate the bounds to with a division similarly to bounding . The output of the selfattention is obtained with a summation of weighted by attention probability : , which can be regarded as a dot product of and , where whose bounds can be obtained from those of with a transposing. Therefore, bounds of and can be forward propagated to similarly to bounding . In this way, we obtain the output bounds of the selfattention:
(5) 
Recall that is a concatenation of . We can split into vectors with equal dimensions, , such that Eq. (5) becomes
(6) 
Backward Process to SelfAttention Layers
When computing bounds for a later sublayer, the th sublayer, using the backward process, we directly propagate the bounds at the the closest previous selfattention layer assumed to be the th layer, to the input layer, and we skip other previous sublayers. The bounds propagated to the th layer are as Eq. (2). We substitute with linear bounds in Eq. (6):
We take global bounds as Eq. (3) and Eq. (4) to obtain the bounds of the th layer.
Advantageous of Combining the Backward Process with a Forward Process
Introducing a forward process can significantly reduce the complexity of verifying Transformers. With the backward process only, we need to compute and , where the major cost is on and there are such matrices to compute. The factor is from the dependency between all pairs of positions in the input and output respectively, which makes the algorithm inefficient especially when the input sequence is long. In contrast, the forward process represents the bounds as linear functions of the perturbed positions only instead of all positions by computing and . Imperceptible adversarial examples may not have many perturbed positions (gao2018black; ko2019popqorn), and thus we may assume that the number of perturbed positions, , is small. The major cost is on while there are only such matrices and the sizes of and are relatively comparable for a small . We combine the backward process and the forward process. The number of matrices in the forward process is , and for the backward process, since we do not propagate bounds over selfattention layers and there is no crossposition dependency in other sublayers, we only compute such that , and thus the number of matrices is reduced to . Therefore, the total number of matrices and we compute is and is times smaller than when only the backward process is used. Moreover, the backward process makes bounds tighter compared to solely the forward one, as we explain in Appendix D.
4 Experiments
To demonstrate the effectiveness of our algorithm, we compute certified bounds for several sentiment classification models and perform an ablation study to show the advantage of combining the backward and forward processes. We also demonstrate the meaningfulness of our certified bounds with an application on identifying important words.
4.1 Datasets and Models
We use two datasets: Yelp (zhang2015character) and SST (socher2013recursive). Yelp consists of 560,000/38,000 examples in the training/test set and SST consists of 67,349/872/1,821 examples in the training/development/test set. Each example is a sentence or a sentence segment (for the training data of SST only) labeled with a binary sentiment polarity.
We verify the robustness of Transformers trained from scratch. For the main experiments, we consider layer models (
), with 4 attention heads, hidden sizes of 256 and 512 for selfattention and feedforward layers respectively, and we use ReLU activations for feedforward layers. We remove the variance related terms in layer normalization, making Transformers verification bounds tighter while the clean accuracies remain comparable (see Appendix
E for discussions). Although our method can be in principal applied to Transformers with any number of layers, we do not use largescale pretrained models such as BERT because they are too challenging to be tightly verified with the current technologies.4.2 Certified Bounds
Dataset  Acc.  Upper  Lower (IBP)  Lower (Ours)  Ours vs Upper  

Min  Avg  Min  Avg  Min  Avg  Min  Avg  
Yelp  1  91.5  9.085  13.917  1.4E4  3.1E4  1.423  1.809  16%  13%  
0.695  1.005  1.4E4  3.1E4  0.384  0.483  55%  48%  
0.117  0.155  1.4E4  3.1E4  0.034  0.043  29%  27%  
2  91.5  10.228  15.452  1.4E7  2.2E7  0.389  0.512  4%  3%  
0.773  1.103  1.4E7  2.2E7  0.116  0.149  15%  14%  
0.122  0.161  1.4E7  2.2E7  0.010  0.013  9%  8%  
3  91.6  11.137  15.041  4.3E10  7.1E10  0.152  0.284  1%  2%  
0.826  1.090  4.3E10  7.1E10  0.042  0.072  5%  7%  
0.136  0.187  4.3E10  7.1E10  0.004  0.006  3%  3%  
SST  1  83.2  7.418  8.849  2.4E4  2.7E4  2.503  2.689  34%  30%  
0.560  0.658  2.4E4  2.7E4  0.418  0.454  75%  69%  
0.091  0.111  2.4E4  2.7E4  0.033  0.036  36%  32%  
2  83.5  6.781  8.367  3.6E7  3.8E7  1.919  1.969  28%  24%  
0.520  0.628  3.6E7  3.8E7  0.305  0.315  59%  50%  
0.085  0.105  3.6E7  3.8E7  0.024  0.024  28%  23%  
3  83.9  6.475  7.877  5.7E10  6.7E10  1.007  1.031  16%  13%  
0.497  0.590  5.7E10  6.7E10  0.169  0.173  34%  29%  
0.084  0.101  5.7E10  6.7E10  0.013  0.014  16%  13% 
Yelp  SST  
Lower (IBP)  Lower (Ours)  Lower (IBP)  Lower (Ours)  
Min  Avg  Min  Avg  Min  Avg  Min  Avg  
1  6.5E5  1.2E4  0.242  0.290  1.1E4  1.1E4  0.212  0.229 
2  6.2E8  8.6E8  0.060  0.078  1.5E7  1.5E7  0.145  0.149 
3  2.8E10  4.4E10  0.023  0.035  3.3E10  4.5E10  0.081  0.083 
We compute certified lower bounds for different models on different datasets. We include 1position perturbation constrained by norms and 2position perturbation constrained by norm. We compare our lower bounds with those computed by the Interval Bound Propagation (IBP) (gowal2018effectiveness) baseline. For 1position perturbation, we also compare with upper bounds computed by enumerating all the words in the vocabulary and finding the word closest to the original one such that the word substitution alters the predicted label. This method has an exponential complexity with respect to the vocabulary size and can hardly be extended to perturbations on 2 or more positions; thus we do not include upper bounds for 2position perturbation. For each example, we enumerate possible options of perturbed positions (there are options), and we integrate results from different options by taking the minimum or average respectively. We report the average results on 10 correctly classified random test examples with sentence lengths no more than 32 for 1position perturbation and 16 for 2position perturbation. Table 1 and Table 2 present the results for 1position and 2position perturbation respectively. Our certified lower bounds are significantly larger and thus tighter than those by IBP. For 1position perturbation, the lower bounds are consistently smaller than the upper bounds, and the gap between the upper bounds and our lower bounds are reasonable compared with that in previous work on verification of feedforward networks, e.g., in (weng2018towards; zhang2018efficient) the upper bounds are in the order of 10 times larger than lower bounds. This demonstrates that our proposed method can compute robustness bounds for Transformers in a similar quality to the bounds of simpler neural networks.
4.3 Effectiveness of Combining the Backward Process with a Forward Process
Dataset  Acc.  FullyForward  FullyBackward  Backward & Forward  

Min  Avg  Time  Min  Avg  Time  Min  Avg  Time  
Yelp  91.3  2.122  2.173  12.6  3.485  3.737  141.4  3.479  3.729  24.0  
0.576  0.599  12.4  0.867  0.947  140.4  0.866  0.946  26.0  
0.081  0.084  12.6  0.123  0.136  143.9  0.123  0.136  26.4  
SST  83.3  1.545  1.592  13.7  1.891  1.961  177.6  1.891  1.961  26.5  
0.352  0.366  12.6  0.419  0.439  178.8  0.419  0.439  24.3  
0.048  0.050  14.6  0.058  0.061  181.3  0.058  0.061  24.3 
In the following, we show the effectiveness of combining the backward process with a forward process. We compare our proposed method (Backward & Forward) with two variations: 1) FullyForward propagates bounds in a forward manner for all sublayers besides selfattention layers; 2) FullyBackward computes bounds for all sublayers including selfattention layers using the backward bound propagation and without the forward process. We compare the tightness of bounds and computation time of the three methods. We use smaller models with the hidden sizes reduced by 75%, and we use 1position perturbation only, to accommodate FullyBackward with large computational cost. Experiments are conducted on an NVIDIA TITAN X GPU. Table 3 presents the results. Bounds by FullyForward are significantly looser while those by FullyBackward and Backward & Forward are comparable. Meanwhile, the computation time of Backward & Forward is significantly shorter than that of FullyBackward. This demonstrates that our method of combining the backward and forward processes can compute comparably tight bounds much more efficiently.
4.4 Identifying Words Important to Prediction
The certified lower bounds can reflect how sensitive a model is to the perturbation of each input word. Intuitively, if a word is more important to the prediction, the model is more sensitive to its perturbation. Therefore, the certified lower bounds can be used to identify important words. In the following, we conduct an experiment to verify whether important words can be identified by our certified lower bounds. We use a 1layer Transformer classifier under 1position perturbation constrained by
norm. We compare our method with two baselines that also estimate local vulnerability: 1)
Upper uses upper bounds; 2) Gradient identifies the word whose embedding has the largest norm of gradients as the most important and vice versa.Quantitative Analysis on SST
SST contains sentiment labels for all phrases on parse trees, where the labels range from very negative (0) to very positive (4), and 2 for neutral. For each word, assuming its label is , we take , i.e., the distance to the neutral label, as the importance score, since less neutral words tend to be more important for the sentiment polarity of the sentence. We evaluate on 100 random test input sentences and compute the average importance scores of the most or least important words identified from the examples. In Table 4, compared to the baselines (“Upper” and “Grad”), the average importance score of the most important words identified by our lower bounds are the largest, while the least important words identified by our method have the smallest average score. This demonstrates that our method identifies the most and least important words more accurately compared to baseline methods.
Qualitative Analysis on Yelp
We further analyze the results on a larger dataset, Yelp. Since Yelp does not provide perword sentiment labels, importance scores cannot be computed as on SST. Thus, we demonstrate a qualitative analysis. We use 10 random test examples and collect the words identified as the most and least important word in each example. In Table 4, most words identified as the most important by certified lower bounds are exactly the words reflecting sentiment polarities (boldfaced words), while those identified as the least important words are mostly stopwords. Baseline methods mistakenly identify more words containing no sentiment polarity as the most important. This again demonstrates that our certified lower bounds identify word importance better than baselines and our bounds provide meaningful interpretations in practice. While gradients evaluate the sensitivity of each input word, this evaluation only holds true within a very small neighborhood (where the classifier can be approximated by a firstorder Taylor expansion) around the input sentence. Our certified method gives valid lower bounds that hold true within a large neighborhood specified by a perturbation set , and thus it provides more accurate results.
5 Conclusion
We propose the first robustness verification method for Transformers, and tackle key challenges in verifying Transformers, including crossnonlinearity and crossposition dependency. Our method computes certified lower bounds that are significantly tighter than those by IBP. Quantitative and qualitative analyses further show that our bounds are meaningful and can reflect the importance of different words in sentiment analysis.
Acknowledgement
This work is jointly supported by Tsinghua Scholarship for Undergraduate Overseas Studies, Tsinghua University Initiative Scientific Research Program, Tsinghua University DCST Student Academic Training Program, and NSF IIS1719097 and IIS1927554.
References
Appendix A Illustration of Different Bounding Processes
Figure 1 illustrates a comparison of the FullyForward, FullyBackward and Backward & Forward processes, for a 2layer Transformer as an example. For FullyForward, there are only forward processes connecting adjacent layers and blocks. For FullyBackward, there are only backward processes, and each layer needs a backward bound propagation to all the previous layers. For our Backward & Forward algorithm, we use backward processes for the feedforward parts and forward processes for selfattention layers, and for layers after selfattention layers, they no longer need backward bound propagation to layers prior to selfattention layers. In this way, we resolve the crossposition dependency in verifying Transformers while still keeping bounds comparably tight as those by using fully backward processes. Empirical comparison of the three frameworks are presented in Sec. 4.3.
Appendix B Linear Bounds of Unary Nonlinear Functions
We show in Sec. 3.2 that linear bounds can be propagated over unary nonlinear functions as long as the unary nonlinear functions can be bounded with linear functions. Such bounds are determined for each neuron respectively, according to the bounds of the input for the function. Specifically, for a unary nonlinear function , with the bounds of obtained previously as , we aim to derive a linear lower bound and a linear upper bound , such that
where parameters are dependent on and designed for different functions respectively. We introduce how the parameters are determined for different unary nonlinear functions involved in Transformers such that the linear bounds are valid and as tight as possible. Bounds of ReLU and tanh has been discussed by zhang2018efficient, and we further derive bounds of , , , . and are only used when the layer normalization is not modified for experiments to study the impact of our modification. For the following description, we define the endpoints of the function to be bounded within range as and . We describe how the lines corresponding to the linear bounds of different functions can be determined, and thereby parameters can be determined accordingly.
ReLU
For ReLU activation, . ReLU is inherently linear on segments and respectively, so we make the linear bounds exactly for or ; and for , we take the line passing the two endpoints as the upper bound; and we take when and when as the lower bound, to minimize the gap between the lower bound and the original function.
Tanh
For activation, . is concave for , and thus we take the line passing the two endpoints as the lower bound and take a tangent line passing as the upper bound. For , is convex, and thus we take the line passing the two endpoints as the upper bound and take a tangent line passing as the lower bound. For , we take a tangent line passing the right endpoint and as the lower bound, and take a tangent line passing the left endpoint and as the upper bound. and