A K-variate Time Series Is Worth K Words: Evolution of the Vanilla Transformer Architecture for Long-term Multivariate Time Series Forecasting

12/06/2022
by   Zanwei Zhou, et al.
0

Multivariate time series forecasting (MTSF) is a fundamental problem in numerous real-world applications. Recently, Transformer has become the de facto solution for MTSF, especially for the long-term cases. However, except for the one forward operation, the basic configurations in existing MTSF Transformer architectures were barely carefully verified. In this study, we point out that the current tokenization strategy in MTSF Transformer architectures ignores the token uniformity inductive bias of Transformers. Therefore, the vanilla MTSF transformer struggles to capture details in time series and presents inferior performance. Based on this observation, we make a series of evolution on the basic architecture of the vanilla MTSF transformer. We vary the flawed tokenization strategy, along with the decoder structure and embeddings. Surprisingly, the evolved simple transformer architecture is highly effective, which successfully avoids the over-smoothing phenomena in the vanilla MTSF transformer, achieves a more detailed and accurate prediction, and even substantially outperforms the state-of-the-art Transformers that are well-designed for MTSF.

READ FULL TEXT

page 1

page 4

page 5

research
02/03/2022

ETSformer: Exponential Smoothing Transformers for Time-series Forecasting

Transformers have been actively studied for time-series forecasting in r...
research
05/26/2022

Are Transformers Effective for Time Series Forecasting?

Recently, there has been a surge of Transformer-based solutions for the ...
research
09/08/2022

W-Transformers : A Wavelet-based Transformer Framework for Univariate Time Series Forecasting

Deep learning utilizing transformers has recently achieved a lot of succ...
research
08/30/2022

Persistence Initialization: A novel adaptation of the Transformer architecture for Time Series Forecasting

Time series forecasting is an important problem, with many real world ap...
research
06/14/2023

TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

Transformers have gained popularity in time series forecasting for their...
research
03/08/2020

Progressive Growing of Neural ODEs

Neural Ordinary Differential Equations (NODEs) have proven to be a power...
research
10/26/2020

Peak Detection On Data Independent Acquisition Mass Spectrometry Data With Semisupervised Convolutional Transformers

Liquid Chromatography coupled to Mass Spectrometry (LC-MS) based methods...

Please sign up or login with your details

Forgot password? Click here to reset