A Differential Attention Fusion Model Based on Transformer for Time Series Forecasting

02/23/2022
by   Benhan Li, et al.
0

Time series forecasting is widely used in the fields of equipment life cycle forecasting, weather forecasting, traffic flow forecasting, and other fields. Recently, some scholars have tried to apply Transformer to time series forecasting because of its powerful parallel training ability. However, the existing Transformer methods do not pay enough attention to the small time segments that play a decisive role in prediction, making it insensitive to small changes that affect the trend of time series, and it is difficult to effectively learn continuous time-dependent features. To solve this problem, we propose a differential attention fusion model based on Transformer, which designs the differential layer, neighbor attention, sliding fusion mechanism, and residual layer on the basis of classical Transformer architecture. Specifically, the differences of adjacent time points are extracted and focused by difference and neighbor attention. The sliding fusion mechanism fuses various features of each time point so that the data can participate in encoding and decoding without losing important information. The residual layer including convolution and LSTM further learns the dependence between time points and enables our model to carry out deeper training. A large number of experiments on three datasets show that the prediction results produced by our method are favorably comparable to the state-of-the-art.

READ FULL TEXT
research
06/11/2022

DRAformer: Differentially Reconstructed Attention Transformer for Time-Series Forecasting

Time-series forecasting plays an important role in many real-world scena...
research
07/19/2021

Long-term series forecasting with Query Selector – efficient model of sparse attention

Various modifications of TRANSFORMER were recently used to solve time-se...
research
07/28/2021

Long-term series forecasting with Query Selector

Various modifications of TRANSFORMER were recently used to solve time-se...
research
08/18/2021

Construction Cost Index Forecasting: A Multi-feature Fusion Approach

The construction cost index is an important indicator in the constructio...
research
06/29/2019

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Time series forecasting is an important problem across many domains, inc...
research
04/14/2022

Joint Forecasting of Panoptic Segmentations with Difference Attention

Forecasting of a representation is important for safe and effective auto...
research
02/07/2022

TACTiS: Transformer-Attentional Copulas for Time Series

The estimation of time-varying quantities is a fundamental component of ...

Please sign up or login with your details

Forgot password? Click here to reset