SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

02/02/2023
by   Jiaxiang Dong, et al.
12

Time series analysis is widely used in extensive areas. Recently, to reduce labeling expenses and benefit various tasks, self-supervised pre-training has attracted immense interest. One mainstream paradigm is masked modeling, which successfully pre-trains deep models by learning to reconstruct the masked content based on the unmasked part. However, since the semantic information of time series is mainly contained in temporal variations, the standard way of randomly masking a portion of time points will ruin vital temporal variations of time series seriously, making the reconstruction task too difficult to guide representation learning. We thus present SimMTM, a Simple pre-training framework for Masked Time-series Modeling. By relating masked modeling to manifold learning, SimMTM proposes to recover masked time points by the weighted aggregation of multiple neighbors outside the manifold, which eases the reconstruction task by assembling ruined but complementary temporal variations from multiple masked series. SimMTM further learns to uncover the local structure of the manifold helpful for masked modeling. Experimentally, SimMTM achieves state-of-the-art fine-tuning performance in two canonical time series analysis tasks: forecasting and classification, covering both in- and cross-domain settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/24/2023

UniTS: A Universal Time Series Analysis Framework with Self-supervised Representation Learning

Machine learning has emerged as a powerful tool for time series analysis...
research
09/11/2023

Examining the Effect of Pre-training on Time Series Classification

Although the pre-training followed by fine-tuning paradigm is used exten...
research
05/20/2022

Cross Reconstruction Transformer for Self-Supervised Time Series Representation Learning

Unsupervised/self-supervised representation learning in time series is c...
research
10/05/2022

TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis

Time series analysis is of immense importance in extensive applications,...
research
10/06/2020

A Transformer-based Framework for Multivariate Time Series Representation Learning

In this work we propose for the first time a transformer-based framework...
research
02/18/2023

FrAug: Frequency Domain Augmentation for Time Series Forecasting

Data augmentation (DA) has become a de facto solution to expand training...
research
03/09/2023

Masked Image Modeling with Local Multi-Scale Reconstruction

Masked Image Modeling (MIM) achieves outstanding success in self-supervi...

Please sign up or login with your details

Forgot password? Click here to reset