DeepAI
Log In Sign Up

A Transformer-based Framework for Multivariate Time Series Representation Learning

10/06/2020
by   George Zerveas, et al.
24

In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series. Pre-trained models can be potentially used for downstream tasks such as regression and classification, forecasting and missing value imputation. By evaluating our models on several benchmark datasets for multivariate time series regression and classification, we show that not only does our modeling approach represent the most successful method employing unsupervised learning of multivariate time series presented to date, but also that it exceeds the current state-of-the-art performance of supervised methods; it does so even when the number of training samples is very limited, while offering computational efficiency. Finally, we demonstrate that unsupervised pre-training of our transformer models offers a substantial performance benefit over fully supervised learning, even without leveraging additional unlabeled data, i.e., by reusing the same data samples through the unsupervised objective.

READ FULL TEXT

page 1

page 2

page 3

page 4

10/04/2022

MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting

Large-scale self-supervised pre-training Transformer architecture have s...
12/02/2018

Improving Clinical Predictions through Unsupervised Time Series Representation Learning

In this work, we investigate unsupervised representation learning on med...
07/29/2021

Self-supervised Transformer for Multivariate Clinical Time-Series with Missing Values

Multivariate time-series (MVTS) data are frequently observed in critical...
03/28/2022

Enhancing Transformer Efficiency for Multivariate Time Series Classification

Most current multivariate time series (MTS) classification algorithms fo...
09/16/2022

DBT-DMAE: An Effective Multivariate Time Series Pre-Train Model under Missing Data

Multivariate time series(MTS) is a universal data type related to many p...
10/24/2016

Representation Learning with Deconvolution for Multivariate Time Series Classification and Visualization

We propose a new model based on the deconvolutional networks and SAX dis...

Code Repositories

mvts_transformer

Multivariate Time Series Transformer, public version


view repo

bert_timeseries

Multivariate time series representation learning (using bert-like model adapted for TS)


view repo

Transformers-for-In-Air-Handwriting-Recognition

None


view repo