Log In Sign Up

Learning to Encode Position for Transformer with Continuous Dynamical Model

We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are less sensitive to position. The main reason is that position information among input units is not inherently encoded, i.e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input. However, this solution has clear limitations: the sinusoidal encoding is not flexible enough as it is manually designed and does not contain any learnable parameters, whereas the position embedding restricts the maximum length of input sequences. It is thus desirable to design a new position layer that contains learnable parameters to adjust to different datasets and different architectures. At the same time, we would also like the encodings to extrapolate in accordance with the variable length of inputs. In our proposed solution, we borrow from the recent Neural ODE approach, which may be viewed as a versatile continuous version of a ResNet. This model is capable of modeling many kinds of dynamical systems. We model the evolution of encoded results along position index by such a dynamical system, thereby overcoming the above limitations of existing methods. We evaluate our new position layers on a variety of neural machine translation and language understanding tasks, the experimental results show consistent improvements over the baselines.


Improve Transformer Models with Better Relative Position Embeddings

Transformer architectures rely on explicit position encodings in order t...

Demystifying the Better Performance of Position Encoding Variants for Transformer

Transformers are state of the art models in NLP that map a given input s...

Dynamic Position Encoding for Transformers

Recurrent models have been dominating the field of neural machine transl...

Rethinking Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...

Word Order Matters when you Increase Masking

Word order, an essential property of natural languages, is injected in T...

Rethinking the Positional Encoding in Language Pre-training

How to explicitly encode positional information into neural networks is ...

Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

In contrast to fully connected networks, Convolutional Neural Networks (...

1 Introduction

Transformer based models [20, 4, 22, 14, 7, 15]

have become one of the most effective approaches to model sequence data of variable lengths. Transformers have shown wide applicability to many natural language processing (NLP) tasks such as language modeling 

[14], neural machine translation (NMT) [20], and language understanding [4]

. Unlike traditional recurrent-based models (e.g., RNN or LSTM), Transformer utilizes a non-recurrent but self-attentive neural architecture to model the dependency among elements at different positions in the sequence, which leads to better parallelization using modern hardware and alleviates the vanishing/exploding gradient problem in traditional recurrent models.

[23] prove that the design of self-attentive architecture leads to a family of permutation equivalence functions. Thus, for applications where the ordering of the elements matters, how to properly encode position information is crucial for Transformer based models. There have been many attempts to encode position information for the Transformer. In the original Transformer paper [20], a family of pre-defined sinusoidal functions was adapted to construct a set of embeddings for each position. These fixed position embeddings are then added to the word embeddings of the input sequence accordingly. To further construct these position embeddings in a more data-driven way, many recent Transformer variants such as [4, 9] include these embeddings as learnable model parameters in the training stage. This data-driven approach comes at the cost of the limitation of a fixed maximum length of input sequence and the computational/memory overhead of additional parameters, where is usually set to 512 in many applications, and is the dimension of the embeddings. [18] propose a relative position representation to reduce the number of parameters to by dropping the interactions between tokens with a distance greater than . In addition to just the input layer, [3] and [7] suggest that the injection of position information to every layer leads to even better performance for the Transformer.

An ideal position encoding approach should satisfy the following three properties:

  1. [nosep,leftmargin=1em,labelwidth=*,align=left]

  2. Inductive: the ability to handle sequences longer than any sequence seen in the training time.

  3. Data-Driven: the position encoding should be learnable from the data.

  4. Parameter Efficient: number of trainable parameters introduced by the encoding should be limited to avoid increased model size, which could hurt generalization.

In Table 1, we summarize some of the existing position encoding approaches in terms of these three properties.

In this paper, we propose a new method to encode position with minimum cost. The main idea is to model position encoding as a continuous dynamical system, so we only need to learn the system dynamics instead of learning the embeddings for each position independently. By doing so, our method enjoys the best of both worlds – we bring back the inductive bias, and the encoding method is freely trainable while being parameter efficient. To enable training of this dynamical system with backpropagation, we adopt the recent progress in continuous neural network 

[1], officially called Neural ODE. In some generative modeling literature, it is also called the free-form flow model [5], so we call our model FLOw-bAsed TransformER (FLOATER). We highlight our contributions as follows:

  • [nosep,leftmargin=1em,labelwidth=*,align=left]

  • We propose FLOATER, a new position encoder for Transformer, which models the position information via a continuous dynamical model in a data-driven and parameter-efficient manner.

  • Due to the use of a continuous dynamic model, FLOATER can handle sequences of any length. This property makes inference more flexible.

  • With careful design, our position encoder is compatible with the original Transformer; i.e., the original Transformer can be regarded as a special case of our proposed position encoding approach. As a result, we are not only able to train a Transformer model with FLOATER from scratch but also plug FLOATER into most existing pre-trained Transformer models such as BERT, RoBERTa, etc.

  • We demonstrate that FLOATER consistent improvements over baseline models across a variety of NLP tasks ranging from machine translations, language understanding, and question answering.

width=0.7center Methods Inductive Data-Driven Parameter Efficient Sinusoidal [20] Embedding [4] Relative [18] This paper

Table 1: Comparing position representation methods

2 Background and Related Work

2.1 Importance of Position Encoding for Transformer

We use a simplified self-attentive sequence encoder to illustrate the importance of position encoding in the Transformer. Without position encoding, the Transformer architecture can be viewed as a stack of blocks containing a self-attentive and a feed-forward layer

. By dropping the residual connections and layer normalization, the architecture of a simplified Transformer encoder can be represented as follows.


where , is the length of the sequence and is the dimension of the word embedding. and are the self-attentive and feed-forward layer in the -th block , respectively.

Each row of can be regarded as a weighted sum of the value matrix , with the weights determined by similarity scores between the key matrix and query matrix as follows:


and are the weight and bias parameters introduced in the self-attentive function . The output of the feed-forward function used in the Transformer is also a matrix with rows. In particular, the -th row is obtained as follows.


where and

are the weights and biases of linear transforms, and

is the activation function. It is not hard to see from (

3) and (4) that both and are permutation equivalent. Thus, we can conclude that the entire function defined in (1) is also permutation equivalent, i.e., for any permutation matrix . This permutation equivalence property restricts the Transformer without position information from modeling sequences where the ordering of elements matters.

Figure 1: The architecture of our model (FLOATER). The main differences between FLOATER and the original Transformer model are: 1) the position representation is integrated into each block in the hierarchy (there are blocks in total); and 2) there is a dynamical model (see (8

)) that generates position encoding vectors for each block. The dynamics are solved with a black-box ODE solver detailed in the supplementary material.

2.2 Position Encoding in Transformer

As mentioned in Section 1, there are many attempts to inject position information in self-attentive components. Most of them can be described in the following form:


where is a position encoding function.

[20] propose to keep and inject position information only at the input block with a family of pre-defined sinusoidal functions: , where ] is a position embedding matrix with the -th row corresponding to the -th position in the input sequence. In particular, the -th dimension of the -th row is defined as follows.


where . [3] and [7] observe better performance by further injecting the position information at each block, i.e., as follows:


Note that for the above two approaches, position encoding functions are fixed for all the applications. Although no additional parameters are introduced in the model, both approaches are inductive and can handle input sequences of variable length.

Many successful variants of pre-trained Transformer models, such as BERT [4] and RoBERTa [9], include the entire embedding matrix in as training parameters. As the number of training parameters needs to be fixed, the maximum length of a sequence, , is required to be determined before the training. Although it lacks the inductive property, this data-driven approach is found to be effective for many NLP tasks. Note that, unlike the fixed sinusoidal position encoding, there is no attempt to inject a learnable position embedding matrix at each block for Transformer due to a large number of additional parameters ().

3 FLOATER: Our Proposed Position Encoder

We introduce our method in three steps. In the first step, we only look at one Transformer block, and describe how to learn the position representation driven by a dynamical system; in the second step, we show how to save parameters if we add position signals to every layer; lastly, we slightly change the architecture to save trainable parameters further and make FLOATER “compatible” with the original Transformer [20]. The compatibility means our model is a strict superset of the vanilla Transformer so that it can be initialized from the Transformer.

3.1 Position Encoding with Dynamical Systems

Position representations in Transformer models are a sequence of vectors to be added to the sequence of the input representations {. Existing position encoding approaches either apply a fixed sinusoidal function to obtain , or include them as uncorrelated learnable parameters. Both of them fail to capture the dependency or dynamics among these position representations . In this paper, we propose to use a dynamical system to model these position representations; that is, there is a “latent force” denoted by that drives the changes from to . To encourage smoothness, we consider as the continuous version of the discrete sequence . In particular, our proposed continuous dynamical system is characterized as follows:


together with an initial vector , where is a neural network parameterized by and takes the previous state . Notice that the domain of is . The position sequence can be obtained by taking on a series of points : . One simple strategy is to set so that the points are equidistant, where

is a hyperparameter (e.g.,

). With this strategy, we are implicitly assuming the position signals evolve steadily as we go through each token in a sentence. In general, can be any monotonically increasing series, which allows us to extend our work to more applications where the elements in the sequence are not always observed with the same interval. More discussions about the applicability for this general setting is included in the Supplementary material. For the NLP applications discussed in this paper, we choose .

Eq. (8) is equivalent to an ODE problem , which is guaranteed to have a unique solution under mild conditions [19]. We follow the efficient approach by [1] to calculate the gradients of with respect to the overall training loss, which allows us to include this parameterized dynamical position encoder into the end-to-end training of Transformer models. More details can be found in the Supplementary material.

Our dynamical system (8) is quite flexible to admit the standard sinusoidal position encoding (6) as a special case:


This indicates that for simple sinusoidal encoding, there exists a dynamical system which is also sinusoidal function.

3.2 Parameter Sharing among Blocks

As mentioned in Section 2, injecting position information to each block for Transformer leads to better performance [3, 7] in some language understanding tasks. Our proposed position encoder FLOATER (8) can also be injected into each block. The idea is illustrated in Figure 1. Typically there are blocks in sequence-to-sequence Transformer and or blocks in BERT. We add a superscript to denote dynamics at -th block:

As we can imagine, having different dynamical models for each block can introduce too many parameters and cause significant training overhead. Instead, we address this issue by sharing parameters across all the blocks, namely


Note that (10) does not imply that all the are the same, as we will assign different initial values for each block, that is for .

Transformer-Base Transformer-Large
En-De En-Fr En-De En-Fr
Position encoders at all blocks
FLOATER 28.6 41.6 29.2 42.7
Pre-defined Sinusoidal Position Encoder 28.2 40.6 28.4 42.0
Fixed-length Position Embedding 26.9 40.9 28.3 42.0
Position encoder only at input block
FLOATER 28.3 41.1 29.1 42.4
Pre-defined Sinusoidal Position Encoder 27.9 40.4 28.4 41.8
Fixed-length Position Embedding 27.8 40.9 28.5 42.4
Table 2: Experimental results of various position encoders on the machine translation task.

3.3 Compatibility and Warm-start Training

In this section, we change the way to add position encoding so that our FLOATER can be directly initialized from Transformer. As an example, we use the standard Transformer model, which has a fixed sinusoidal encoding at the input block and no position encoding at deeper levels. Note that this technique can be extended to other variants of Transformers with different position encoding methods, such as embedding matrix. We first examine the standard Transformer model, the query matrix at block- is


where and are parameters in  (3); is the sinusoidal encoding; is the -th row of . Here we add a tilde sign to indicate the sinusoidal vectors. Formulas for and have a very similar form and are omitted for brevity.

Now we consider the case of FLOATER, where new position encodings are added


It is easy to see that the changing the position embedding from to

is equivalent to adding a position-aware bias vector

into each self-attentive layers . As a result, we can instead apply (8) to model the dynamics of . In particular, we have the following dynamical system:


After that, we set . We can see that if and , then . This implies (12) degenerates to (11). Note that (13) has the same form as (8), except that we are now modeling the bias terms in (3). We will apply the same technique to and .

To summarize, our model has a tight connection to the original Transformer: if we set all dynamical models to zero, which means , then our FLOATER model will be equivalent to the original Transformer with the sinusoidal encoding. The same trick also works for Transformer with position embedding such as BERT [4].

We strive to make our model compatible with the original Transformer due to the following reasons. First of all, the original Transformer is faster to train as it does not contain any recurrent computation; this is in contrast to our dynamical model (8), where the next position depends on the previous one

. By leveraging the compatibility of model architecture, we can directly initialize FLOATER model from a pre-trained Transformer model checkpoint and then fine-tune for the downstream task for a few more epochs. By doing so, we enjoy all the benefits of our FLOATER model but still maintain an acceptable training budget. Likewise, for models such as BERT or Transformer-XL, we already have well-organized checkpoints out of the box for downstream tasks. These models are costly to train from scratch, and since our goal is to examine whether our proposed position representation method can improve over the original one, we decided to copy the weights layer by layer for attention as well as FFN layers, and randomly initialize the dynamical model


4 Experimental Results

In this section, we perform experiments to see if FLOATER can improve over the existing position encoding approaches for a given Transformer model on various NLP tasks. Thus, all the metrics reported in this paper are computed from a single (not ensemble) Transformer model over each evaluation NLP task. Albeit lower than top scores on the leaderboard, these metrics are able to reveal more clear signal to judge the effectiveness of the proposed position encoder.

All our codes to perform experiments in this paper are based on the Transformer implementations in the fairseq [11] package. Implementation details can be found in the Supplementary material. Our experimental codes will be made publicly available.

Model Single Sentence Similarity and Paraphrase Natural Language Inference
Base model
RoBERTa 63.6 94.8 88.2 91.9 91.2 87.6 92.8 78.7
FLOATER 63.4 95.1 89.0 91.7 91.5 87.7 93.1 80.5
Large model
RoBERTa 68.0 96.4 90.9 92.2 92.4 90.2 94.7 86.6
FLOATER 69.0 96.7 91.4 92.2 92.5 90.4 94.8 87.0
Table 3: Experimental results on GLUE benchmark
Model Accuracy Middle High
Single model on test, large model
RoBERTa 82.8 86.5 81.3
FLOATER 83.3 87.1 81.7
Table 4: Experiment results on RACE benchmark. “Middle” means middle school level English exams, “High” means high school exams. Other details can be found in [6].

4.1 Neural Machine Translation

Neural Machine Translation (NMT) is the first application that demonstrates the superiority of a sequence-to-sequence Transformer model over conventional recurrent sequence models. We include the following three additive position encoders: .

  • [nosep,leftmargin=1em,labelwidth=*,align=left]

  • Data-driven FLOATER: is generated by our proposed continuous dynamical models with data-driven parameters described in (8).

  • Pre-defined sinusoidal position encoder: is constructed by a pre-defined function described in , which is proposed by [20] and extended by [3].

  • Length-fixed position embedding: is included as learnable training parameters. This is first introduced by [20] and adopted in many variants of Transformer [4, 9].

To better demonstrate the parameter efficiency brought by FLOATER, for each above encoder, we also include two experimental settings: position encoder at all blocks or only at the input block (i.e., ).

In Table 2, we present the BLEU scores on WMT14 Ee-De and En-Fr datasets with both Transformer-base and Transformer-large models described in [20]. Among all the data/model combinations, our proposed FLOATER at all blocks outperforms two other position encoders.

On the other hand, we also observe that adding position encoders at all blocks yields better performance than only at the input block. While there is an exception in the fixed-length position embedding approach. We suspect that this phenomenon is due to over-fitting cased by learnable parameters introduced by this approach. In contrast, our proposed FLOATER is parameter efficient (more discussions in Section 4.3), so the performance can be improved by injecting the position encoder at all the blocks of Transformer without much additional overhead.

4.2 Language Understanding and Question Answering

Model SQuAD 1.1 SQuAD 2.0
Single models on dev, w/o data augmentation
RoBERTa 88.9 94.6 86.5 89.4
FLOATER 88.9 94.6 86.6 89.5
Table 5: Experiment results on SQuAD benchmark. All results are obtained from RoBERTa-large model.

Pretrained Transformer models such as BERT and RoBERTa have become the key to achieving the state-of-the-art performance for various language understanding and question answering tasks. In this section, we want to evaluate the effectiveness of the proposed FLOATER on these tasks. In particular, we focus on three language understanding benchmark sets, GLUE [21], RACE [6] and SQuAD [17]. As mentioned in Section 3.3, FLOATER is carefully designed to be compatible with the existing Transformer models. Thus, we can utilize pretrained Transformer models to warm-start a FLOATER model easily to be used to finetune on these NLP tasks. In this paper, we download the same pre-trained RoBERTa model from the official repository as our pretrained Transformer model for all NLP tasks discussed in this section. GLUE Benchmark. This benchmark is commonly used to evaluate the language understanding skills of NLP models. Experimental results in Table 3 show that our FLOATER model outperforms RoBERTa in most datasets, even though the only difference is the choice of positional encoding. RACE benchmark Similar to the GLUE benchmark, the RACE benchmark is another widely used test suit for language understanding. Compared with GLUE, each item in RACE contains a significantly longer context, which we believe requires more important to grasp the accurate position information. Like in GLUE benchmark, we finetune the model from the same pretrained RoBERTa checkpoint. We keep the hyperparameters, such as batch size and learning rate, to also be the same. Table 4 shows the experimental results. We again see consistent improvement of FLOATER across all subtasks.

Sentence length

BLEU score




Figure 2: Comparing BLEU scores of different encoding methods.
BLEU () #Parameters ()
FLOATER 28.57 526.3K
1-layer RNN + scalar 27.99 263.2K
2-layer RNN + scalar 28.16 526.3K
1-layer RNN + vector 27.99 1,050.0K
Table 6: Performance comparison on WMT14 En-De data and Transformer-base architecture. Both BLUE scores and the number of trainable parameters inside each position encoder are included.

SQuAD benchmark SQuAD benchmark [17, 16] is another challenging task to evaluate the question answering skills of NLP models. In this dataset, each item contains a lengthy paragraph containing facts and several questions related to the paragraph. The model needs to predict the range of characters that answer the questions. In SQuAD-v2, the problem becomes more challenging that the questions might be unanswerable by the context. We follow the same data processing script as BERT/RoBERTa for fair comparison; more details about the training process are described in the Supplementary material. The experiment results are presented in Table 5. As we can see, the FLOATER model beats the baseline RoBERTa model consistently across most datasets. The improvement is significant, considering that both models are finetuned from the same pretrained checkpoint.

4.3 More Discussions and Analysis

How inductive is FLOATER?

FLOATER is designed to be inductive by a data-driven dynamical model (8). To see how inductive FLOATER is when comparing to existing approaches, we design the following experiment. We first notice that in WMT14 En-De dataset, of the training sentences are shorter than tokens. Based on that, we make a new dataset called En-De short to long (or S2L for brevity): this dataset takes all the short sentences ( tokens) as the training split and all the long sentences ( tokens) as the testing split. We further divide the testing split to four bins according to the source length fallen in . BLEU scores are calculated in each bin, and the results are presented in Figure 2.

Our FLOATER model performs particularly well on long sentences, even though only short sentences are seen by the model during training. This empirical observation supports our conjecture that FLOATER model is inductive: the dynamics learned from shorter sequences can be appropriately generalized to longer sequences.

Is RNN a good alternative to model the dynamics?

Recurrent neural network (RNN) is commonly used to perform sequential modeling. RNN and our continuous dynamical model (8) indeed share some commonality. Computing the value at the -th step relies on the results at the -st step. Further, they all contain trainable parameters, allowing them to adapt to each particular task. Lastly, they can be extrapolated to any length as needed. To see if RNN works equally well, we model the sequence with RNN models:


where is the input to the RNN model at index . Recall in RNN language models, is the word embedding or hidden feature of the -th token. In our case, since we apply RNN to learn the encodings as opposed to hidden features, sensible inputs can be scalar value or vectorized value by sinusoidal encoding. We tried both choices on WMT14 En-De data and found that vectorized value generally works better, though not as good as our FLOATER model. Detailed results can be found in Table 6.

(a) Sinusoidal
(b) Position embedding
(d) RNN
Figure 3: Visualizing the four different position methods. All models are trained using the Transformer-base architecture and En-De dataset. For better visualization, dimension indices are permuted in Figure 2(b)-2(d).

What does each position encoding look like?

To better understand how different position encodings affect the sequence modeling, in Figure 3, we visualize the position embedding matrix obtained from four different position encoding approaches for the Transformer-base backbone on WMT14 En-De dataset. We can see that sinusoidal encoding (2(a)) is the most structural, while position embedding (2(b)) is quite chaotic. Our FLOATER model learns position representation completely from data, but still exhibits some regularities (2(c)). Finally, the RNN model (2(d)

) fails to extract sufficient positional information, probably due to the vanishing gradient problem. Another finding is that by looking at (

2(b)), we observe that the vectors are nearly constant among different large positions (near the bottom of Figure 2(b), we see patterns of vertical lines with the same color). This phenomenon is due to long sentences in the dataset being scarce, and so the positional information carried by lower indices cannot be extrapolated to higher indices. On the contrary, the dynamical model proposed in this paper enjoys the best of both worlds – it is adaptive to dataset distribution, and it is inductive to handle sequences with lengths longer than the training split.

4.4 Remarks on Training and Testing Efficiency

It is not surprising that during the training time, our flow-based method adds a non-negligible time and memory overhead; this is because solving the Neural ODE precisely involves times forward and backward propagations of the flow model. Even though we deliberately designed a small flow model (consisting of only two FFN and one nonlinearity layers), stacking them together still increases training time substantially. To make it possible to train big models, we use the following optimizations:

  • [nosep,leftmargin=1em,labelwidth=*,align=left]

  • Initialize with pretrained models that do not contain flow-based dynamics, as discussed in Section 3.3.

  • From (8), we know that if is close to zero, then the position information diminishes (derived in appendix). In this way, our model degenerates to the original Transformer. Inspired by this property, we can initialize the FLOATER with smaller weights. Combining with the previous trick, we obtain an informed initialization that incurs lower training loss at the beginning.

  • We observed that weights in are more stable and easy to train. Thus, we can separate the weights of from the remaining parts of the Transformer model. Concretely, we can 1) cache the positional bias vectors for some iterations without re-computing, 2) update the weights of flow models less frequently than other parts of the Transformer, and 3) update the flow models with a larger learning rate to accelerate convergence.

  • For the RoBERTa model, we adopt an even more straightforward strategy: we first download a pretrained RoBERTa model, plug in some flow-based encoding layers, and re-train the encoding layers on WikiText-103 dataset for one epoch. When finetuning on GLUE datasets, we can choose to freeze the encoding layers.

Combining those tricks, we successfully train our proposed models with only -% overhead compared to traditional models, and virtually no overhead when finetuning RoBERTa model on GLUE benchmarks. Moreover, there is no overhead during the inference stage if we store the pre-calculated positional bias vectors in the checkpoints.

5 Conclusions

In this paper, we have shown that learning position encoding with a dynamical model can be an advantageous approach to improve Transformer models. Our proposed position encoding approach is inductive, data-driven, and parameter efficient. We have also demonstrated the superiority of our proposed model over existing position encoding approaches on various natural language processing tasks such as neural machine translation, language understanding, and question answering tasks.


  • [1] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud (2018)

    Neural ordinary differential equations

    In Advances in Neural Information Processing Systems, pp. 6571–6583. Cited by: Appendix A, Appendix A, §1, §3.1.
  • [2] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud (2018) Neural ordinary differential equations. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), pp. 6571–6583. External Links: Link Cited by: 2nd item.
  • [3] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser (2018) Universal transformers. arXiv preprint arXiv:1807.03819. Cited by: §1, §2.2, §3.2, 2nd item.
  • [4] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: Table 1, §1, §1, §2.2, §3.3, 3rd item.
  • [5] W. Grathwohl, R. T. Chen, J. Betterncourt, I. Sutskever, and D. Duvenaud (2018) Ffjord: free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367. Cited by: §1.
  • [6] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy (2017) RACE: large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. Cited by: §4.2, Table 4.
  • [7] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut (2019)

    Albert: a lite bert for self-supervised learning of language representations

    arXiv preprint arXiv:1909.11942. Cited by: §1, §1, §2.2, §3.2.
  • [8] Y. Liu and M. Lapata (2019)

    Hierarchical transformers for multi-document summarization

    arXiv preprint arXiv:1905.13164. Cited by: 1st item.
  • [9] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §1, §2.2, 3rd item.
  • [10] S. Merity, C. Xiong, J. Bradbury, and R. Socher (2016) Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Cited by: §B.3.
  • [11] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019) Fairseq: a fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. Cited by: §B.2, §4.
  • [12] M. Ott, S. Edunov, D. Grangier, and M. Auli (2018) Scaling neural machine translation. arXiv preprint arXiv:1806.00187. Cited by: §B.2.
  • [13] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery (1992) Numerical recipes in c++. The art of scientific computing 2, pp. 1002. Cited by: §B.1.
  • [14] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. Cited by: §1.
  • [15] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2019)

    Exploring the limits of transfer learning with a unified text-to-text transformer

    arXiv preprint arXiv:1910.10683. Cited by: §1.
  • [16] P. Rajpurkar, R. Jia, and P. Liang (2018) Know what you don’t know: unanswerable questions for squad. arXiv preprint arXiv:1806.03822. Cited by: §4.2.
  • [17] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016) SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. Cited by: §4.2, §4.2.
  • [18] P. Shaw, J. Uszkoreit, and A. Vaswani (2018) Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464–468. Cited by: Table 1, §1.
  • [19] M. Tenenbaum and H. Pollard (1985) Ordinary differential equations: an elementary textbook for students of mathematics, engineering, and the sciences. Dover Books on Mathematics, Dover Publications. External Links: ISBN 9780486649405, LCCN lc85012983, Link Cited by: §3.1.
  • [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008. Cited by: §B.2, Table 1, §1, §1, §2.2, §3, 2nd item, 3rd item, §4.1.
  • [21] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. Cited by: §4.2.
  • [22] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le (2019) XLNet: generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. Cited by: §1.
  • [23] C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar (2019) Are transformers universal approximators of sequence-to-sequence functions?. arXiv preprint arXiv:1912.10077. Cited by: §1.
  • [24] X. Zhang, F. Wei, and M. Zhou (2019) HIBERT: document level pre-training of hierarchical bidirectional transformers for document summarization. CoRR abs/1905.06566. External Links: Link, 1905.06566 Cited by: 1st item.

Appendix A Training a Neural ODE model in Transformer

We discuss the details of training the dynamical model , recall in our FLOWER model, function joins in the computational graph implicitly by generating a sequence of position encoding vectors , conditioning on a freely initialized vector . The generation steps are computed iteratively as follows (suppose we choose the interval between two consecutive tokens to be )


Finally, the loss of this sequence is going to be a function of all position encoding results , which is further a function of model parameters . The question is how to calculate the gradient through backpropagation. This question is fully solved in Neural ODE method [1] with an efficient adjoint ODE solver. To illustrate the principle, we draw a diagram showing the forward and backward propagation in Figure 4.

Figure 4: Direction of forward and backward propagation. Here we consider a simplified version where only position encodings and are in the computational graph.

From [1], we know that the gradients can be computed by


where defined in is called the “adjoint state” of ODE, which can be computed by solving another ODE


Note that the computation of (17) only involves Jacobian-vector product so it can be efficiently calculated by automatic differentiation.

Appendix B Implementation details

b.1 Settings of ODE solver

To setup the ODE server, we need to first choose the numerical algorithms [13]. We have different setups for different datasets. For neural machine translation problems (WMT14 En-De and En-Fr), we use the more accurate Runge-Kutta scheme with discretization step to solve the adjoint equation (recall that we set the interval of two neighboring tokens to be globally). While for datasets with long sentences such as GLUE and RACE benchmarks, we found that solving the adjoint equation with high order scheme is too slow, in such case we adopt simple midpoint method with discretization step , and the gradients are calculated by automatic differentiation rather than adjoint method. The third party implementation of ODE solver can be found at

b.2 Training NMT tasks

We run the same preprocessing script provided by fairseq [11], which is also used in ScalingNMT [12]. With the standard training script, we first successfully reproduce all the results in Transformer paper [20]. Based on that we execute the following protocol to get our results:

  1. Train the original Transformer model for 30 epochs.

  2. Random initialize FLOWER model of same shape configuration.

  3. Copy tensors from the best performing checkpoint (validation set) to initialize FLOWER model. Initialize weights in the dynamical model with small values.

  4. Half the peak learning rate (e.g. in Transformer-base + En-De, the peak learning rate is changed from to ).

  5. With the warm-initialized FLOWER checkpoint, retrain on the same dataset for 10 epochs (En-De) or 1 epoch (En-Fr).

  6. Averaging last 5 checkpoints and compute BLEU score on test split.

b.3 Training language understanding tasks

For GLUE/SQuAD/RACE benchmarks, our experiments are all conducted upon RoBERTa, in which both base and large configurations are available. Due to resource constraint (and to show the compatibility to existing models), we initialize our FLOWER model with pretrained RoBERTa, which is similar to NMT task. However, the weights in dynamic function are not trained in large corpus, given that GLUE/SQuAD/RACE datasets are too small to train dynamics from scratch, we decided to pretrain alone in WikiText103 [10] data using masked language modeling loss. We have found that when we train alone, it only takes a few hours (2x Titan V100) and one epoch to convergence.

Once having the pretrained FLOWER model, we can run following downstream tasks and compare with RoBERTa under the same setting:

GLUE benchmark

consists of eight datasets and each have different hyperparameter settings. For hyperparameters such as learning rate, batch size, training iterations, warm-up iterations, etc., we use the same values recommended by official repository of RoBERTa111Available at:

SQuAD benchmark.

For this benchmark we wrote our own finetuning code because currently there is no official code available. During the implementation process, we mainly refer to the third-party repositories222 Mainly and We are not able to exactly match the official result reported in RoBERTa paper but quite close ( difference in F1). For our FLOWER model, we use the same hyperparameters as RoBERTa.

RACE benchmark.

This benchmark has the longest context and sequence length. We follow the official training script333 and reproduce the result. Similar to other benchmarks, we then repeat the training process using exactly the same training hyperparameters to make a fair comparison. In this benchmark we freeze the weights and only finetune the weights of RoBERTa.

Appendix C Cases suitable for non-equidistant discritization

Although our model allows continuous values of and in (8), limiting the scope to text modeling tasks, positions are discrete values as . Once the continuous version of position representation is obtained, we simply take the discritized as the actual values to feed into Transformer model, where is a hyperparameter (e.g. ). By choosing positions equidistantly, we are implicitly assuming the position signal evolves steadily as we go through each token in a sentence. More generally, the dynamics in (8) can deal with the case in which positions are not integers etc., but arbitrary monotone increasing series which may not be equidistant. In appendix, we exemplify this general situation with several widely deployed tasks; we regard this as a interesting future direction. This makes our model particularly suitable for following scenarios yet traditional position representation may not be good at:

  • [nosep,leftmargin=1em,labelwidth=*,align=left]

  • Hierarchical Transformer model [8, 24]. The model is a direct extension of hierarchical RNN and is often used in long document processing. It works by first running a word-level Transformer model on each sentence to extract the sentence embedding, and then applying a sentence-level Transformer scanning through each sentence embedding sequentially. We argue that when processing at the sentence level, it could be better to set the increment of position index proportional to the length of the -th sentence. This is because longer sentences tend to carry more information, so is likely to move farther from .

  • Transformer for time-series events. As measurement time is continuous, time-series data is another scenario when a continuous position makes more sense than a discrete counterpart. More importantly, to predict the future values by modeling historical values observed at irregular time grids, it is better to consider the length of time horizon between two consecutive measures. A successful previous work is the Latent ODE [2], except that they use RNN as the backbone, and they model the hidden states rather than position representations with Neural ODE (because RNN itself provides positional bias).

In this paper, we are not going to explore the more general cases discussed above. Instead, we decided to leave them as interesting future work.