Transformers predicting the future. Applying attention in next-frame and time series forecasting

08/18/2021
by   Radostin Cholakov, et al.
0

Recurrent Neural Networks were, until recently, one of the best ways to capture the timely dependencies in sequences. However, with the introduction of the Transformer, it has been proven that an architecture with only attention-mechanisms without any RNN can improve on the results in various sequence processing tasks (e.g. NLP). Multiple studies since then have shown that similar approaches can be applied for images, point clouds, video, audio or time series forecasting. Furthermore, solutions such as the Perceiver or the Informer have been introduced to expand on the applicability of the Transformer. Our main objective is testing and evaluating the effectiveness of applying Transformer-like models on time series data, tackling susceptibility to anomalies, context awareness and space complexity by fine-tuning the hyperparameters, preprocessing the data, applying dimensionality reduction or convolutional encodings, etc. We are also looking at the problem of next-frame prediction and exploring ways to modify existing solutions in order to achieve higher performance and learn generalized knowledge.

READ FULL TEXT
research
05/26/2022

Are Transformers Effective for Time Series Forecasting?

Recently, there has been a surge of Transformer-based solutions for the ...
research
02/03/2022

ETSformer: Exponential Smoothing Transformers for Time-series Forecasting

Transformers have been actively studied for time-series forecasting in r...
research
09/20/2023

Transformers versus LSTMs for electronic trading

With the rapid development of artificial intelligence, long short term m...
research
06/08/2022

Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

The performance of time series forecasting has recently been greatly imp...
research
06/29/2019

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

Time series forecasting is an important problem across many domains, inc...
research
08/28/2021

A new rotating machinery fault diagnosis method based on the Time Series Transformer

Fault diagnosis of rotating machinery is an important engineering proble...
research
08/21/2018

Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks

Recently, researchers have started applying convolutional neural network...

Please sign up or login with your details

Forgot password? Click here to reset